E-Marketing in Minneapolis David Vinge, eMarketing Dashboard : Internationalization (I18n) - Character encodings for beginners

Tuesday, March 4, 2008

Internationalization (I18n) - Character encodings for beginners

What is a character encoding, and why should I care?

First, why should I care?
If you use anything other than the most basic letters and numbers of the English alphabet, people may not be able to read your text unless you say what character encoding you used.

For example, you may intend the text to look like this:

but it may actually display like this:

Not only does inadequate encoding information spoil the readability of displayed text, but it may mean that your data cannot be found by a search, or reliably processed in a number of other ways.

So what's a character encoding?
Words and sentences in text are created from characters. Examples of characters include the Latin letter á or the Chinese ideograph or the Devanagari character .
You may not be able to see some of the characters in this page because you don't have the necessary fonts. If you click on the place where you expected to see a character you will link to a graphic version. This page is encoded in UTF-8. Characters are grouped into a character set (also called a repertoire). This is then called a coded character set when each character is assigned a particular number, called a codepoint. These codepoints will be represented in the computer by one or more bytes. Basically, this means that all characters are stored in computers using a code, like the ciphers used in espionage. A character encoding is a key to unlock (ie. crack) the code. It is a set of mappings between the bytes representing numbers in the computer and characters in the coded character set. Without the key, the data looks like garbage.

The misleading term charset is often used to refer to what are in reality character encodings. You should be aware of this usage, but stick to using the term character encodings whenever you can. Unfortunately, there are many different character sets and character encodings, ie. many different ways of mapping between bytes, codepoints and characters. The following panel provides more detail for those who are interested.

For example, in the coded character set called ISO 8859-1 (also known as Latin1) the codepoint value for the letter é is 233. In ISO 8859-5, the same codepoint represents the Cyrillic character щ. These character sets contain fewer than 256 characters and map codepoints to byte values directly. So a codepoint with the value 233 is represented by a single byte with a value of 233. Note however that that byte may represent either é or щ, depending on the context.
Other coded character sets use a different approach. For example, with the
Unicode character set, you can represent both characters in the same set. In fact, Unicode contains, in a single set, most characters you are likely to ever need. While the value of 233 still represents the é, the Cyrillic character щ now has a codepoint value of 1097. This is too large a number to be represented by a single byte*. If you use the character encoding for Unicode text called UTF-8, щ will be represented by two bytes, but the codepoint value is not simply derived from the value of the two bytes spliced together - some more complicated decoding is needed. Other Unicode characters map to one, three or four bytes in the UTF-8 encoding.
UTF-8 is the most widely used way to represent Unicode text in web pages. But UTF-8 is only one of the possible ways of encoding Unicode characters. In other words, a single codepoint in the Unicode character set can actually be mapped to different byte sequences, depending on which encoding was used for the document. Unicode codepoints can be mapped to bytes using any one of the encodings called UTF-8, UTF-16 or UTF-32. The Devanagari character
, with codepoint 2325 (which is 915 in hexadecimal notation), will be represented by two bytes when using the UTF-16 encoding (09 15), three bytes with UTF-8 (E0 A4 95), or four bytes with UTF-32 (00 00 09 15).

There can be further complications beyond those described in the panel above (such as byte order and escape sequences), but the detail described there shows why it is important that the application you are working with knows which character encoding is appropriate for your data, and knows how to handle that encoding.
Most of the time, however, you will not need to know the details. You will just need to be sure that you consider the advice in the section How does this affect me? below.

How do fonts fit into this?
A font is a collection of glyphs (definitions of shapes) used to display characters.
Once your application has worked out what characters it is dealing with, it will then look in the font for glyphs in order to display or print those characters. (Of course, if the encoding information was wrong, it will be looking up glyphs for the wrong characters.)

A given font will usually cover a single character set, or in the case of a large character set like Unicode, just a subset of all the characters in the set. If your font doesn't have a glyph for a particular character, some applications will look for the missing character in other fonts on your system (which will mean that the glyph will look different from the surrounding text, like a ransom note). Otherwise you will typically see a square box, a question mark or some other character instead. For example:

How does this affect me?
As a content author or developer, you need to choose the best character encoding for your content or data. Unicode encodings are often a good choice here, since you can use a single encoding to handle pretty much any character you are likely to meet. This greatly simplifies things. Using Unicode throughout your system also removes the need to track and convert between various character encodings.

Content authors need to check what encoding their editor or scripts are saving text in, and how to save text in the encoding of your choice. Note, however, that just declaring a different encoding won't change the bytes; you need to save the text in that encoding too.
Content authors also need to find out how to declare the character encoding used for the document format they are working with. You may also need to check that your server is serving documents with the right HTTP declarations. Developers need to ensure that the various parts of the system can communicate with each other, understand which character encodings are being used, and support all the necessary encodings and characters.

No comments: