UTF-8 and Unicode FAQUTF-8 and Unicode FAQ for Unix/Linuxby Markus KuhnThis text is a very comprehensive one-stop information resourceon how you can use Unicode/UTF-8 on POSIX systems (Linux, Unix). Youwill find here both introductory information for every user, as well asdetailed references for the experienced developer.Unicode has started to replace ASCII, ISO 8859 and EUC at alllevels. It enables users to handle not only practically any script andlanguage used on this planet, it also supports a comprehensive set ofmathematical and technical symbols to simplify scientific informationexchange.With the UTF-8 encoding, Unicode can be used in a convenient andbackwards compatible way in environments that were designed entirelyaround ASCII, like Unix. UTF-8 is the way in which Unicode is usedunder Unix, Linux, and similar systems. It is now time to make surethat you are well familiar with it and that your software supportsUTF-8 smoothly.ContentsWhat are UCS and ISO 10646?What are combining characters?What are UCS implementation levels?Has UCS been adopted as a national standard?What is Unicode?So what is the difference between Unicode and ISO 10646?What is UTF-8?Who invented UTF-8?Where do I find nice UTF-8 example files?What different encodings are there?What programming languages support Unicode?How should Unicode be used under Linux?How do I have to modify my software?C support for Unicode and UTF-8How should the UTF-8 mode be activated?How do I get a UTF-8 version of xterm?How much of Unicode does xterm support?Where do I find ISO 10646-1 X11 fonts?What are the issues related to UTF-8 terminal emulators?What UTF-8 enabled applications are available?[UPDATED]What patches to improve UTF-8 support are available?Are there free libraries for dealing with Unicode available?What is the status of Unicode support for various X widget libraries?What packages with UTF-8 support are currently under development?How does UTF-8 support work under Solaris?Can I use UTF-8 on the Web?How are PostScript glyph names related to UCS codes?Are there any well-defined UCS subsets?What issues are there to consider when converting encodingsIs X11 ready for Unicode? What are useful Perl one-liners for working with UTF-8? How can I enter Unicode characters? [NEW]Are there any good mailing lists on these issues?Further referencesWhat are UCS and ISO 10646?The international standard ISO 10646 defines theUniversal Character Set (UCS). UCS is a superset of all othercharacter set standards. It guarantees round-tripcompatibility to other character sets. This means simply that noinformation is lost if you convert any text string to UCS and thenback to its original encoding.UCS contains the characters required to represent practically allknown languages. This includes not only the Latin, Greek, Cyrillic,Hebrew, Arabic, Armenian, and Georgian scripts, but also Chinese,Japanese and Korean Han ideographs as well as scripts such asHiragana, Katakana, Hangul, Devanagari, Bengali, Gurmukhi, Gujarati,Oriya, Tamil, Telugu, Kannada, Malayalam, Thai, Lao, Khmer, Bopomofo,Tibetan, Runic, Ethiopic, Canadian Syllabics, Cherokee, Mongolian,Ogham, Myanmar, Sinhala, Thaana, Yi, and others. For scripts not yetcovered, research on how to best encode them for computer usage isstill going on and they will be added eventually. This includes notonly historic scripts such as Cuneiform, Hieroglyphs and various Indo-European notations, but even someselected artistic scripts such as Tolkien’s Tengwar and Cirth. UCS also covers a large number of graphical,typographical, mathematical and scientific symbols, including thoseprovided by TeX, PostScript, APL, the International Phonetic Alphabet(IPA), MS-DOS, MS-Windows, Macintosh, OCR fonts, as well as many wordprocessing and publishing systems. The standard continues to bemaintained and updated. Ever more exotic and specialized symbols andcharacters will be added for many years to come.ISO 10646 originally defined a 31-bit character set. The subsets of216 characters where the elements differ (in a 32-bitinteger representation) only in the 16 least-significant bits arecalled the planes of UCS.The most commonly used characters, including all those found inmajor older encoding standards, have been placed into the first plane(0x0000 to 0xFFFD), which is called theBasic Multilingual Plane (BMP) or Plane 0. The characters thatwere later added outside the 16-bit BMP are mostly for specialistapplications such as historic scripts and scientific notation. Currentplans are that there will never be characters assigned outside the21-bit code space from 0x000000 to 0x10FFFF, which covers a bit overone million potential future characters. The ISO 10646-1 standard wasfirst published in 1993 and defines the architecture of the characterset and the content of the BMP. A second part ISO 10646-2 was added in2001 and defines characters encoded outside the BMP. In the 2003edition, the two parts were combined into a single ISO 10646 standard.New characters are still being added on a continuous basis, but theexisting characters will not be changed any more and are stable.UCS assigns to each character not only a code number but also anofficial name. A hexadecimal number that represents a UCS or Unicodevalue is commonly preceded by “U+” as in U+0041 for the character“Latin capital letter A”. The UCS characters U+0000 to U+007F areidentical to those in US-ASCII (ISO 646 IRV) and the range U+0000 toU+00FF is identical to ISO 8859-1 (Latin-1). The range U+E000 toU+F8FF and also larger ranges outside the BMP are reserved for privateuse. UCS also defines several methods for encoding a string ofcharacters as a sequence of bytes, such as UTF-8 and UTF-16.The full reference for the UCS standard isInternational Standard ISO/IEC 10646, Information technology— Universal Multiple-Octet Coded Character Set (UCS) . Thirdedition, International Organization for Standardization, Geneva, 2003.The standard can be orderedonline from ISO as a set of PDF files on CD-ROM for 112 CHF.[NEW] In September 2006, ISO releaseda free online PDF copy of ISO 10646:2003 on its FreelyAvailable Standards web page. The ZIPfile is 82 MB long.What are combining characters?Some code points in UCS have been assigned to combiningcharacters. These are similar to the non-spacing accent keys on atypewriter. A combining character is not a full character by itself.It is an accent or other diacritical mark that is added to theprevious character. This way, it is possible to place any accent onany character. The most important accented characters, like those usedin the orthographies of common languages, have codes of their own inUCS to ensure backwards compatibility with older character sets. Theyare known asprecomposed characters. Precomposed characters are available inUCS for backwards compatibility with older encodings that have nocombining characters, such as ISO 8859. The combining-charactermechanism allows one to add accents and other diacritical marks to anycharacter. This is especially important for scientific notations suchas mathematical formulae and the International Phonetic Alphabet,where any possible combination of a base character and one or severaldiacritical marks could be needed.Combining characters follow the character which they modify. Forexample, the German umlaut character Ä (“Latin capital letter A withdiaeresis”) can either be represented by the precomposed UCS codeU+00C4, or alternatively by the combination of a normal “Latin capitalletter A” followed by a “combining diaeresis”: U+0041 U+0308. Severalcombining characters can be applied when it is necessary to stackmultiple accents or add combining marks both above and below the basecharacter. The Thai script, for example, needs up to two combiningcharacters on a single base character.What are UCS implementation levels?Not all systems can be expected to support all the advancedmechanisms of UCS, such as combining characters. Therefore, ISO 10646specifies the following three implementation levels:Level 1Combining characters and Hangul Jamo characters are not supported.[Hangul Jamo are an alternative representation ofprecomposed modern Hangul syllables as a sequence of consonants andvowels. They are required to fully support the Korean script includingMiddle Korean.]Level 2Like level 1, however in some scripts, a fixed list ofcombining characters is now allowed (e.g., for Hebrew, Arabic,Devanagari, Bengali, Gurmukhi, Gujarati, Oriya, Tamil, Telugo, Kannada,Malayalam, Thai and Lao). These scripts cannot be representedadequately in UCS without support for at least certain combiningcharacters.Level 3All UCS characters are supported, such that, forexample, mathematicians can place a tilde or an arrow (or both) on anycharacter.Has UCS been adopted as a national standard?Yes, a number of countries have published national adoptions of ISO10646, sometimes after adding additional annexes with cross-referencesto older national standards, implementation guidelines, andspecifications of various national implementation subsets:China: GB 13000.1-93Japan: JISX 0221-1:2001Korea: KS X 1005-1:1995 (includes ISO 10646-1:1993 amendments 1-7)Vietnam: TCVN6909:2001(This “16-bit Coded Vietnamese Character Set” is asmall UCS subset and to be implemented for data interchange with andwithin government agencies as of 2002-07-01.)Iran: ISIRI6219:2002, Information Technology — Persian InformationInterchange and Display Mechanism, using Unicode. (This is not aversion or subset of ISO 10646, but a separate document that providesadditional national guidance and clarification on handling the Persianlanguage and the Arabic script in Unicode.)What is Unicode?In the late 1980s, there have been two independent attempts tocreate a single unified character set. One was the ISO 10646 projectof the International Organization forStandardization (ISO), the other was the Unicode Project organized by aconsortium of (initially mostly US) manufacturers of multi-lingualsoftware. Fortunately, the participants of both projects realized inaround 1991 that two different unified character sets is not exactlywhat the world needs. They joined their efforts and worked together oncreating a single code table. Both projects still exist and publishtheir respective standards independently, however the UnicodeConsortium and ISO/IEC JTC1/SC2 have agreed to keep the code tables ofthe Unicode and ISO 10646 standards compatible and they closelycoordinate any further extensions. Unicode 1.1 corresponded to ISO10646-1:1993, Unicode 3.0 corresponded to ISO 10646-1:2000, Unicode3.2 added ISO 10646-2:2001, and Unicode 4.0 corresponds to ISO10646:2003, and Unicode 5.0 corresponds to ISO 10646:2003 plus itsamendments 1–3. All Unicode versions since 2.0 are compatible, onlynew characters will be added, no existing characters will be removedor renamed in the future.The Unicode Standard can be ordered like any normal book, forinstance via amazon.comfor around 60 USD:The Unicode Consortium: TheUnicode Standard 5.0,Addison-Wesley, 2006,ISBN 0-321-48091-0.If you work frequently with text processing and character sets, youdefinitely should get a copy. Unicode 5.0 is also available online.So what is the difference between Unicode and ISO 10646?The UnicodeStandard published by the Unicode Consortium corresponds to ISO10646 at implementation level 3. All characters are at the samepositions and have the same names in both standards.The Unicode Standard defines in addition much more semanticsassociated with some of the characters and is in general a betterreference for implementors of high-quality typographic publishingsystems. Unicode specifies algorithms for rendering presentation formsof some scripts (say Arabic), handling of bi-directional texts thatmix for instance Latin and Hebrew, algorithms for sorting and stringcomparison, and much more.The ISO 10646 standard on the other hand is not much more than asimple character set table, comparable to the old ISO 8859 standards.It specifies some terminology related to the standard, defines someencoding alternatives, and it contains specifications of how to useUCS in connection with other established ISO standards such as ISO6429 and ISO 2022. There are other closely related ISO standards, forinstance ISO14651 on sorting UCS strings. A nice feature of the ISO 10646-1standard is that it provides CJK example glyphs in five differentstyle variants, while the Unicode standard shows the CJK ideographsonly in a Chinese variant.What is UTF-8?UCS and Unicode are first of all just code tables that assigninteger numbers to characters. There exist several alternatives forhow a sequence of such characters or their respective integer valuescan be represented as a sequence of bytes. The two most obviousencodings store Unicode text as sequences of either 2 or 4 bytessequences. The official terms for these encodings are UCS-2 and UCS-4,respectively. Unless otherwise specified, the most significant bytecomes first in these (Bigendian convention). An ASCII or Latin-1 filecan be transformed into a UCS-2 file by simply inserting a 0x00 bytein front of every ASCII byte. If we want to have a UCS-4 file, we haveto insert three 0x00 bytes instead before every ASCII byte.Using UCS-2 (or UCS-4) under Unix would lead to very severeproblems. Strings with these encodings can contain as parts of manywide characters bytes like “\0” or “/” which have a special meaning infilenames and other C library function parameters. In addition, themajority of UNIX tools expects ASCII files and cannot read 16-bit wordsas characters without major modifications. For these reasons,UCS-2 is not a suitable external encoding of Unicode infilenames, text files, environment variables, etc.The UTF-8 encoding defined in ISO 10646-1:2000 Annex D and also described in RFC 3629 as well assection 3.9 of the Unicode 4.0 standard does not have these problems.It is clearly the way to go for usingUnicode under Unix-style operating systems.UTF-8 has the following properties:UCS characters U+0000 to U+007F (ASCII) are encoded simply asbytes 0x00 to 0x7F (ASCII compatibility). This means that files andstrings which contain only 7-bit ASCII characters have the sameencoding under both ASCII and UTF-8.All UCS characters >U+007F are encoded as a sequence of severalbytes, each of which has the most significant bit set. Therefore, noASCII byte (0x00-0x7F) can appear as part of any other character.The first byte of a multibyte sequence that represents a non-ASCIIcharacter is always in the range 0xC0 to 0xFD and it indicates howmany bytes follow for this character. All further bytes in a multibytesequence are in the range 0x80 to 0xBF. This allows easyresynchronization and makes the encoding stateless and robust againstmissing bytes.All possible 231 UCS codes can be encoded.UTF-8 encoded characters may theoretically be up to six byteslong, however 16-bit BMP characters are only up to three bytes long.The sorting order of Bigendian UCS-4 byte strings is preserved.The bytes 0xFE and 0xFF are never used in the UTF-8 encoding.The following byte sequences are used to represent a character. Thesequence to be used depends on the Unicode number of the character:U-00000000 – U-0000007F:0xxxxxxxU-00000080 – U-000007FF:110xxxxx 10xxxxxxU-00000800 – U-0000FFFF:1110xxxx 10xxxxxx 10xxxxxxU-00010000 – U-001FFFFF:11110xxx 10xxxxxx 10xxxxxx 10xxxxxxU-00200000 – U-03FFFFFF:111110xx 10xxxxxx 10xxxxxx 10xxxxxx10xxxxxxU-04000000 – U-7FFFFFFF:1111110x 10xxxxxx 10xxxxxx 10xxxxxx10xxxxxx 10xxxxxxThe xxx bit positions are filled with the bits of thecharacter code number in binary representation. The rightmost xbit is the least-significant bit. Only the shortest possible multibytesequence which can represent the code number of the character can beused. Note that in multibyte sequences, the number of leading 1 bitsin the first byte is identical to the number of bytes in the entiresequence.Examples: The Unicode character U+00A9 = 10101001 (copyright sign) is encoded in UTF-8 as 11000010 10101001 = 0xC2 0xA9 and character U+2260 = 0010 0010 0110 0000 (not equalto) is encoded as: 11100010 10001001 10100000 = 0xE2 0x89 0xA0The official name and spelling of this encoding is UTF-8, where UTFstands for UCS Transformation Format. Please donot write UTF-8 in any documentation text in other ways (such as utf8or UTF_8), unless of course you refer to a variable name and not theencoding itself.An important note for developers of UTF-8 decoding routines:For security reasons, a UTF-8 decoder mustnot accept UTF-8 sequences that are longer than necessary toencode a character. For example, the character U+000A (line feed) mustbe accepted from a UTF-8 stream only in the form 0x0A, but notin any of the following five possible overlong forms: 0xC0 0x8A 0xE0 0x80 0x8A 0xF0 0x80 0x80 0x8A 0xF8 0x80 0x80 0x80 0x8A 0xFC 0x80 0x80 0x80 0x80 0x8AAny overlong UTF-8 sequence could be abused to bypass UTF-8substring tests that look only for the shortest possible encoding. Alloverlong UTF-8 sequences start with one of the following bytepatterns:1100000x (10xxxxxx)11100000 100xxxxx (10xxxxxx)11110000 1000xxxx (10xxxxxx 10xxxxxx)11111000 10000xxx (10xxxxxx 10xxxxxx10xxxxxx)11111100 100000xx (10xxxxxx 10xxxxxx10xxxxxx 10xxxxxx)Also note that the code positions U+D800 to U+DFFF (UTF-16surrogates) as well as U+FFFE and U+FFFF must not occur in normalUTF-8 or UCS-4 data. UTF-8 decoders should treat them like malformedor overlong sequences for safety reasons.Markus Kuhn’s UTF-8 decoderstress test file contains a systematic collection of malformed andoverlong UTF-8 sequences and will help you to verify the robustness ofyour decoder.Who invented UTF-8?The encoding known today as UTF-8 was invented by Ken Thompson. It wasborn during the evening hours of 1992-09-02 in a New Jersey diner,where he designed it in the presence of Rob Pike on a placemat(see Rob Pike’s UTF-8 history). Itreplaced an earlier attempt to design a FSS/UTF (file system safe UCStransformation format) that was circulated in an X/Open workingdocument in August 1992 by Gary Miller (IBM), Greger Leijonhufvud andJohn Entenmann (SMI) as a replacement for the division-heavy UTF-1encoding from the first edition of ISO 10646-1. By the end of thefirst week of September 1992, Pike and Thompson had turned AT&TBell Lab’s Plan 9into the world’s first operating system to use UTF-8. They reported about their experienceat the USENIXWinter 1993 Technical Conference, San Diego, January 25-29, 1993,Proceedings, pp. 43-50. FSS/UTF was briefly also referred to as UTF-2and later renamed into UTF-8, and pushed through the standards processby the X/Open Joint Internationalization Group XOJIG.Where do I find nice UTF-8 example files?A few interesting UTF-8 example files for tests and demonstrationsare:UTF-8Sampler web page by the Kermit projectMarkus Kuhn’s example plain-textfiles, including among others the classic demo, decoder test, TeX repertoire, WGL4 repertoire, euro testpages, and Robert Brady’s IPA lyrics.Unicode TranscriptionsGeneratorfor Indic Unicode test filesWhat different encodings are there?Both the UCS and Unicode standards are first of all large tablesthat assign to every character an integer number. If you use the term“UCS”, “ISO 10646”, or “Unicode”, this just refers to a mappingbetween characters and integers. This does not yet specify how tostore these integers as a sequence of bytes in memory.ISO 10646-1 defines the UCS-2 and UCS-4 encodings. These aresequences of 2 bytes and 4 bytes per character, respectively. ISO10646 was from the beginning designed as a 31-bit character set (withpossible code positions ranging from U-00000000 to U-7FFFFFFF),however it took until 2001 for the first characters to be assignedbeyond the Basic Multilingual Plane (BMP), that is beyond the first216 character positions (see ISO 10646-2 and Unicode 3.1).UCS-4 can represent all UCS and Unicode characters, UCS-2 canrepresent only those from the BMP (U+0000 to U+FFFF).“Unicode” originally implied that the encoding was UCS-2 and itinitially didn’t make any provisions for characters outside the BMP(U+0000 to U+FFFF). When it became clear that more than 64k characterswould be needed for certain special applications (historic alphabetsand ideographs, mathematical and musical typesetting, etc.), Unicodewas turned into a sort of 21-bit character set with possible codepoints in the range U-00000000 to U-0010FFFF. The 2×1024 surrogatecharacters (U+D800 to U+DFFF) were introduced into the BMP to allow1024×1024 non-BMP characters to be represented as a sequence oftwo 16-bit surrogate characters. This way UTF-16 was born, which representsthe extended “21-bit” Unicode in a way backwards compatible withUCS-2. The term UTF-32 wasintroduced in Unicode to describe a 4-byte encoding of the extended“21-bit” Unicode. UTF-32 is the exact same thing as UCS-4, except thatby definition UTF-32 is never used to represent characters aboveU-0010FFFF, while UCS-4 can cover all 231 code positions upto U-7FFFFFFF. The ISO 10646 working group has agreed to modify theirstandard to exclude code positions beyond U-0010FFFF, in order to turnthe new UCS-4 and UTF-32 into practically the same thing.In addition to all that, UTF-8 was introducedto provide an ASCII backwards compatible multi-byte encoding. Thedefinitions of UTF-8 in UCS and Unicode differed originally slightly,because in UCS, up to 6-byte long UTF-8 sequences were possible torepresent characters up to U-7FFFFFFF, while in Unicode only up to4-byte long UTF-8 sequences are defined to represent characters up toU-0010FFFF. (The difference was in essence the same as between UCS-4and UTF-32.)No endianess is implied by the encoding names UCS-2, UCS-4, UTF-16,and UTF-32, though ISO 10646-1 says that Bigendian should be preferredunless otherwise agreed. It has become customary to append the letters“BE” (Bigendian, high-byte first) and “LE” (Littleendian, low-bytefirst) to the encoding names in order to explicitly specify a byteorder.In order to allow the automatic detection of the byte order, it hasbecome customary on some platforms (notably Win32) to start everyUnicode file with the character U+FEFF (ZERO WIDTH NO-BREAK SPACE),also known as the Byte-Order Mark (BOM). Its byte-swapped equivalentU+FFFE is not a valid Unicode character, therefore it helps tounambiguously distinguish the Bigendian and Littleendian variants ofUTF-16 and UTF-32.A full featured character encoding converter will have to providethe following 13 encoding variants of Unicode and UCS:UCS-2, UCS-2BE, UCS-2LE, UCS-4, UCS-4LE, UCS-4BE, UTF-8, UTF-16,UTF-16BE, UTF-16LE, UTF-32, UTF-32BE, UTF-32LEWhere no byte order is explicitly specified, use the byte order ofthe CPU on which the conversion takes place and in an input streamswap the byte order whenever U+FFFE is encountered. The differencebetween outputting UCS-4 versus UTF-32 and UTF-16 versus UCS-2 lies inhandling out-of-range characters. The fallback mechanism fornon-representable characters has to be activated in UTF-32 (forcharacters > U-0010FFFF) or UCS-2 (for characters > U+FFFF) even whereUCS-4 or UTF-16 respectively would offer a representation.Really just of historic interest are UTF-1, UTF-7, SCSU and adozen other less widely publicised UCS encoding proposals with variousproperties, none of which ever enjoyed any significant use. Their useshould be avoided.A good encoding converter will also offer options for adding orremoving the BOM:Unconditionally prefix the output text with U+FEFF.Prefix the output text with U+FEFF unless it is already there.Remove the first character if it is U+FEFF.It has also been suggested to use the UTF-8 encoded BOM (0xEF 0xBB0xBF) as a signature to mark the beginning of a UTF-8 file. Thispractice should definitely not be used on POSIXsystems for several reasons:On POSIX systems, the locale and not magic file type codes definethe encoding of plain text files. Mixing the two concepts would add alot of complexity and break existing functionality.Adding a UTF-8 signature at the start of a file would interferewith many established conventions such as the kernel looking for “#!”at the beginning of a plaintext executable to locate the appropriateinterpreter.Handling BOMs properly would add undesirable complexity even tosimple programs like cat or grep that mixcontents of several files into one.In addition to the encoding alternatives, Unicode also specifiesvarious NormalizationForms, which provide reasonable subsets of Unicode, especially toremove encoding ambiguities caused by the presence of precomposed andcompatibility characters:Normalization Form D (NFD): Split up (decompose)precomposed characters into combining sequences where possible,e.g. use U+0041 U+0308 (LATIN CAPITAL LETTER A, COMBINING DIAERESIS)instead of U+00C4 (LATIN CAPITAL LETTER A WITH DIAERESIS). Also avoiddeprecated characters, e.g. use U+0041 U+030A (LATIN CAPITAL LETTER A,COMBINING RING ABOVE) instead of U+212B (ANGSTROM SIGN).Normalization Form C (NFC): Use precomposed charactersinstead of combining sequences where possible, e.g. use U+00C4 (“Latincapital letter A with diaeresis”) instead of U+0041 U+0308 (“Latincapital letter A”, “combining diaeresis”). Also avoid deprecatedcharacters, e.g. use U+00C5 (LATIN CAPITAL LETTER A WITH RING ABOVE)instead of U+212B (ANGSTROM SIGN).NFC is the preferred form forLinux and WWW.Normalization Form KD (NFKD): Like NFD, but avoid inaddition the use of compatibility characters, e.g. use “fi” instead ofU+FB01 (LATIN SMALL LIGATURE FI).Normalization Form KC (NFKC): Like NFC, but avoid inaddition the use of compatibility characters, e.g. use “fi” instead ofU+FB01 (LATIN SMALL LIGATURE FI).A full-featured character encoding converter should also offerconversion between normalization forms. Care should be used withmapping to NFKD or NFKC, as semantic information might be lost (forinstance U+00B2 (SUPERSCRIPT TWO) maps to 2) and extra mark-upinformation might have to be added to preserve it (e.g.,<SUP>2</SUP> in HTML).What programming languages support Unicode?More recent programming languages that were developed after around1993 already have special data types for Unicode/ISO 10646-1characters. This is the case with Ada95, Java, TCL, Perl, Python, C#and others.ISO C 90 specifies mechanisms to handle multi-byte encoding andwide characters. These facilities were improved with Amendment 1 to ISO C90 in 1994 and even further improvements were made in the ISO C 99 standard. Thesefacilities were designed originally with various East-Asian encodingsin mind. They are on one side slightly more sophisticated than whatwould be necessary to handle UCS (handling of “shift sequences”), butalso lack support for more advanced aspects of UCS (combiningcharacters, etc.). UTF-8 is an example of what the ISO C standardcalls multi-byte encoding. The type wchar_t, which inmodern environments is usually a signed 32-bit integer, can be used tohold Unicode characters.Unfortunately, wchar_t was already widely used forvarious Asian 16-bit encodings throughout the 1990s. Therefore, theISO C 99 standard was bound by backwards compatibility. It could notbe changed to require wchar_t to be used with UCS, likeJava and Ada95 managed to do. However, the C compiler can at leastsignal to an application that wchar_t is guaranteed to holdUCS values in all locales. To do so, it defines the macro__STDC_ISO_10646__ to be an integer constant of the formyyyymmL. The year and month refer to the version ofISO/IEC 10646 and its amendments that have been implemented. Forexample, __STDC_ISO_10646__ == 200009L if theimplementation covers ISO/IEC 10646-1:2000.How should Unicode be used under Linux?Before UTF-8 emerged, Linux users all over the world had to usevarious different language-specific extensions of ASCII. Most popularwere ISO 8859-1 and ISO 8859-2 in Europe, ISO 8859-7 in Greece, KOI-8/ ISO 8859-5 / CP1251 in Russia, EUC and Shift-JIS in Japan, BIG5 in Taiwan, etc.This made the exchange of files difficult and application software hadto worry about various small differences between these encodings.Support for these encodings was usually incomplete, untested, andunsatisfactory, because the application developers rarely used allthese encodings themselves.Because of these difficulties, major Linux distributors andapplication developers are now phasing out these older legacyencodings in favour of UTF-8. UTF-8 support has improved dramaticallyover the last few years and many people now use UTF-8 on a daily basisintext files (source code, HTML files, email messages, etc.)file namesstandard input and standard output, pipesenvironment variablescut and paste selection bufferstelnet, modem, and serial port connections to terminal emulatorsand in any other places where byte sequences used to be interpreted inASCII.In UTF-8 mode, terminal emulators such as xterm or the Linuxconsole driver transform every keystroke into the corresponding UTF-8sequence and send it to the stdin of the foreground process.Similarly, any output of a process on stdout is sent to the terminalemulator, where it is processed with a UTF-8 decoder and thendisplayed using a 16-bit font.Full Unicode functionality with all bells and whistles (e.g.high-quality typesetting of the Arabic and Indic scripts) can only beexpected from sophisticated multi-lingual word-processing packages.What Linux supports today on a broad base is far simpler and mainlyaimed at replacing the old 8- and 16-bit character sets. Linuxterminal emulators and command line tools usually only support a Level1 implementation of ISO 10646-1 (no combining characters), and onlyscripts such as Latin, Greek, Cyrillic, Armenian, Georgian, CJK, andmany scientific symbols are supported that need no further processingsupport. At this level, UCS support is very comparable to ISO 8859support and the only significant difference is that we have nowthousands of different characters available, that characters can berepresented by multibyte sequences, and that ideographicChinese/Japanese/Korean characters require two terminal characterpositions (double-width).Level 2 support in the form of combining characters for selectedscripts (in particular Thai)and Hangul Jamo is in parts also available (i.e., some fonts, terminalemulators and editors support it via simple overstringing), butprecomposed characters should be preferred over combining charactersequences where available. More formally, the preferred way ofencoding text in Unicode under Linux should be Normalization FormC as defined in Unicode TechnicalReport #15.One influential non-POSIX PC operating system vendor (whom we shallleave unnamed here) suggested that all Unicode files should start withthe character ZERO WIDTH NOBREAK SPACE (U+FEFF), which is in this rolealso referred to as the “signature” or “byte-order mark (BOM)”, inorder to identify the encoding and byte-order used in a file.Linux/Unix does not use any BOMs and signatures. Theywould break far too many existing ASCII syntax conventions (such asscripts starting with #!). On POSIX systems, the selectedlocale identifies already the encoding expected in all input andoutput files of a process. It has also been suggested to call UTF-8files without a signature “UTF-8N” files, but this non-standard termis usually not used in the POSIX world.Before you switch to UTF-8 under Linux, update your installation toa recent distribution with up-to-date UTF-8 support. This isparticular the case if you use an installation older than SuSE 9.1 orRed Hat 8.0. Before these, UTF-8 support was not yet mature enough tobe recommendable for daily use.RedHat Linux 8.0 (September 2002) was the first distribution to takethe leap of switching to UTF-8 as the default encoding for mostlocales. The only exceptions were Chinese/Japanese/Korean locales, forwhich there were at the time still too many specialized toolsavailable that did not yet support UTF-8. This first mass deploymentof UTF-8 under Linux caused most remaining issues to be ironed outrather quickly during 2003. SuSELinux then switched its default locales to UTF-8 as well, as ofversion 9.1(May 2004). It was followed by Ubuntu Linux, the firstDebian-derivative that switched to UTF-8 as the system-wide defaultencoding. With the migration of the three most popular Linuxdistributions, UTF-8 related bugs have now been fixed in practicallyall well-maintained Linux tools. Other distributions can be expectedto follow soon.How do I have to modify my software?If you are a developer, there are several approaches to add UTF-8support. We can split them into two categories, which I will call softand hard conversion. In soft conversion, data is kept in its UTF-8form everywhere and only very few software changes are necessary. Inhard conversion, any UTF-8 data that the program reads will beconverted into wide-character arrays and will be handled as sucheverywhere inside the application. Strings will only be converted backto UTF-8 at output time. Internally, a character remains a fixed-sizememory object.We can also distinguish hard-wired and locale-dependent approachesfor supporting UTF-8, depending on how much the string processingrelies on the standard library. C offers a number of string processingfunctions designed to handle arbitrary locale-specific multibyteencodings. An application programmer who relies entirely on these canremain unaware of the actual details of the UTF-8 encoding. Chancesare then that by merely changing the locale setting, several othermulti-byte encodings (such as EUC) will automatically be supported aswell. The other way a programmer can go is to hardcode knowledge aboutUTF-8 into the application. This may lead in some situations tosignificant performance improvements. It may be the best approach forapplications that will only be used with ASCII and UTF-8.Even where support for every multi-byte encoding supported by libcis desired, it may well be worth to add extra code optimized forUTF-8. Thanks to UTF-8’s self-synchronizing features, it can beprocessed very efficiently. The locale-dependent libc string functionscan be two orders of magnitude slower than equivalent hardwired UTF-8procedures. A bad teaching example was GNU grep 2.5.1, which reliedentirely on locale-dependent libc functions such asmbrlen() for its generic multi-byte encoding support.This made it about 100× slower in multibyte mode than insingle-byte mode! Other applications with hardwired support for UTF-8regular expressions (e.g., Perl 5.8) do not suffer this dramaticslowdown.Most applications can do very fine with just soft conversion. Thisis what makes the introduction of UTF-8 on Unix feasible at all. Toname two trivial examples, programs such as cat andecho do not have to be modified at all. They can remaincompletely ignorant as to whether their input and output is ISO 8859-2or UTF-8, because they handle just byte streams without processingthem. They only recognize ASCII characters and control codes such as'\n' which do not change in any way under UTF-8.Therefore the UTF-8 encoding and decoding is done for theseapplications completely in the terminal emulator.A small modification will be necessary for any program thatdetermines the number of characters in a string by counting the bytes.With UTF-8, as with other multi-byte encodings, where the length of atext string is of concern, programmers have to distinguish clearlybetweenthe number of bytes,the number of characters,the display width (e.g., the number of cursor position cells in aVT100 terminal emulator)of a string.C’s strlen(s) function always counts the number ofbytes. This is the number relevant, for example, for memorymanagement (determination of string buffer sizes). Where the output ofstrlen is used for such purposes, no change will be necessary.The number of characters can be counted in C in a portableway using mbstowcs(NULL,s,0). This works for UTF-8 likefor any other supported encoding, as long as the appropriate localehas been selected. A hard-wired technique to count the number ofcharacters in a UTF-8 string is to count all bytes except those in therange 0x80 – 0xBF, because these are just continuation bytes and notcharacters of their own. However, the need to count characters arisessurprisingly rarely in applications.In applications written for ASCII or ISO 8859, a far more commonuse of strlen is to predict the number ofcolumns that the cursor of the terminal will advance if a stringis printed. With UTF-8, neither a byte nor a character count willpredict the display width, because ideographic characters (Chinese,Japanese, Korean) will occupy two column positions, whereas controland combining characters occupy none. To determine the width of astring on the terminal screen, it is necessary to decode the UTF-8sequence and then use the wcwidth function to test thedisplay width of each character, or wcswidth to measurethe entire string.For instance, the ls program had to be modified,because without knowing the column widths of filenames, it cannotformat the table layout in which it presents directories to the user.Similarly, all programs that assume somehow that the output ispresented in a fixed-width font and format it accordingly have tolearn how to count columns in UTF-8 text. Editor functions such asdeleting a single character have to be slightly modified to delete allbytes that might belong to one character. Affected were for instanceeditors (vi, emacs, readline,etc.) as well as programs that use the ncurses library.Any Unix-style kernel can do fine with soft conversion and needsonly very minor modifications to fully support UTF-8. Most kernelfunctions that handle strings (e.g. file names, environment variables,etc.) are not affected at all by the encoding. Modifications werenecessary in Linux the following places:The console display and keyboard driver (another VT100 emulator)have to encode and decode UTF-8 and should support at least somesubset of the Unicode character set. This had already been availablein Linux as early as kernel 1.2 (send ESC %G to the console toactivate UTF-8 mode).External file system drivers such as VFAT and WinNT have toconvert file name character encodings. UTF-8 is one of the availableconversion options, and the mount command has to tell thekernel driver that user processes shall see UTF-8 file names. SinceVFAT and WinNT use already Unicode anyway, UTF-8 is the only availableencoding that guarantees a lossless conversion here.The tty driver of any POSIX system supports a “cooked” mode, inwhich some primitive line editing functionality is available. In orderto allow the character-erase function (which is activated when youpress backspace) to work properly with UTF-8, someone needs to tell itnot count continuation bytes in the range 0x80-0xBF as characters, butto delete them as part of a UTF-8 multi-byte sequence. Since thekernel is ignorant of the libc locale mechanics, another mechanism isneeded to tell the tty driver about UTF-8 being used. Linux kernelversions 2.6 or newer support a bit IUTF8 in the c_iflag membervariable of struct termios. If it is set, the “cooked” mode lineeditor will treat UTF-8 multi-byte sequences correctly. This mode canbe set from the command shell with “stty iutf8”. Xterm and friendsshould set this bit automatically when called in a UTF-8 locale.C support for Unicode and UTF-8Starting with GNU glibc 2.2, the type wchar_t isofficially intended to be used only for 32-bit ISO 10646 values,independent of the currently used locale. This is signalled toapplications by the definition of the __STDC_ISO_10646__macro as required by ISO C99. The ISO C multi-byte conversionfunctions (mbsrtowcs(), wcsrtombs(), etc.)are fully implemented in glibc 2.2 or higher and can be used toconvert between wchar_t and any locale-dependentmultibyte encoding, including UTF-8, ISO 8859-1, etc.For example, you can write #include <stdio.h> #include <locale.h> int main() { if (!setlocale(LC_CTYPE, "")) { fprintf(stderr, "Can't set the specified locale! " "Check LANG, LC_CTYPE, LC_ALL.\n"); return 1; } printf("%ls\n", L"Schöne Grüße"); return 0; }Call this program with the locale setting LANG=de_DEand the output will be in ISO 8859-1. Call it withLANG=de_DE.UTF-8 and the output will be in UTF-8. The%ls format specifier in printf callswcsrtombs in order to convert the wide character argumentstring into the locale-dependent multi-byte encoding.Many of C’s string functions are locale-independent and they justlook at zero-terminated byte sequences: strcpy strncpy strcat strncat strcmp strncmp strdup strchr strrchr strcspn strspn strpbrk strstr strtokSome of these (e.g. strcpy) can equally be used for single-byte(ISO 8859-1) and multi-byte (UTF-8) encoded character sets, as theyneed no notion of how many byte long a character is, while others(e.g., strchr) depend on one character being encoded in a single charvalue and are of less use for UTF-8 (strchr still works fine if youjust search for an ASCII character in a UTF-8 string).Other C functions are locale dependent and work in UTF-8 localesjust as well: strcoll strxfrmHow should the UTF-8 mode be activated?If your application is soft converted and does not use the standardlocale-dependent C multibyte routines (mbsrtowcs(),wcsrtombs(), etc.) to convert everything intowchar_t for processing, then it might have to find out insome way, whether it is supposed to assume that the text data ithandles is in some 8-bit encoding (like ISO 8859-1, where 1 byte = 1character) or UTF-8. Once everyone uses only UTF-8, you can just makeit the default, but until then both the classical 8-bit sets and UTF-8may still have to be supported.The first wave of applications with UTF-8 support used a whole lotof different command line switches to activate their respective UTF-8modes, for instance the famous xterm -u8. That turned outto be a very bad idea. Having to remember a special command lineoption or other configuration mechanism for every applicationis very tedious, which is why command line options arenot the proper way of activating a UTF-8 mode.The proper way to activate UTF-8 is the POSIX locale mechanism. Alocale is a configuration setting that contains information aboutculture-specific conventions of software behaviour, including thecharacter encoding, the date/time notation, alphabetic sorting rules,the measurement system and common office paper size, etc. The names oflocales usually consist of ISO639-1 language and is available, you can use a linesuch as utf8_mode = (strcmp(nl_langinfo(CODESET), "UTF-8") == 0);in order to detect whether the current locale uses the UTF-8encoding. You have of course to add a setlocale(LC_CTYPE,"") at the beginning of your application to set the localeaccording to the environment variables first. The standard functioncall nl_langinfo(CODESET) is also what localecharmap calls to find the name of the encoding specified by thecurrent locale for you. It is available on pretty much every modernUnix now. FreeBSD added nl_langinfo(CODESET) support withversion 4.6 (2002-06). If you need an autoconf test for theavailability of nl_langinfo(CODESET), here is the oneBruno Haible suggested:======================== m4/codeset.m4 ================================#serial AM1dnl From Bruno Haible.AC_DEFUN([AM_LANGINFO_CODESET],[ AC_CACHE_CHECK([for nl_langinfo and CODESET], am_cv_langinfo_codeset, [AC_TRY_LINK([#include <langinfo.h>], [char* cs = nl_langinfo(CODESET);], am_cv_langinfo_codeset=yes, am_cv_langinfo_codeset=no) ]) if test $am_cv_langinfo_codeset = yes; then AC_DEFINE(HAVE_LANGINFO_CODESET, 1, [Define if you have <langinfo.h> and nl_langinfo(CODESET).]) fi])=======================================================================[You could also try to query the locale environment variablesyourself without using setlocale(). In the sequenceLC_ALL, LC_CTYPE, LANG, lookfor the first of these environment variables that has a value. Makethe UTF-8 mode the default (still overridable by command lineswitches) when this value contains the substring UTF-8,as this indicates reasonably reliably that the C library has beenasked to use a UTF-8 locale. An example code fragment that does thisis char *s; int utf8_mode = 0; if (((s = getenv("LC_ALL")) && *s) || ((s = getenv("LC_CTYPE")) && *s) || ((s = getenv("LANG")) && *s)) { if (strstr(s, "UTF-8")) utf8_mode = 1; }This relies of course on all UTF-8 locales having the name of theencoding in their name, which is not always the case, therefore thenl_langinfo() query is clearly the better method. If youare really concerned that calling nl_langinfo() might notbe portable enough, there is also Markus Kuhn’s portable public domainnl_langinfo(CODESET)emulator for systems that do not have the real thing (and anotherone from Bruno Haible), and you can use the norm_charmap() function tostandardize the output of the nl_langinfo(CODESET) ondifferent platforms.]How do I get a UTF-8 version of xterm?The xtermversion that comes with XFree864.0 or higher (maintained by ThomasDickey) includes UTF-8 support. To activate it, start xterm in aUTF-8 locale and use a font with iso10646-1 encoding, forinstance with LC_CTYPE=en_GB.UTF-8 xterm \ -fn '-Misc-Fixed-Medium-R-SemiCondensed--13-120-75-75-C-60-ISO10646-1'and then cat some example file, such as UTF-8-demo.txtin the newly started xterm and enjoy what you see.If you are not using XFree86 4.0 or newer, then you canalternatively download the latest xtermdevelopment version separately and compile it yourself with“./configure --enable-wide-chars ; make” or alternativelywith “xmkmf; make Makefiles; make; make install; makeinstall.man”.If you do not have UTF-8 locale support available, use command lineoption -u8 when you invoke xterm to switch input andoutput to UTF-8.How much of Unicode does xterm support?Xterm in XFree86 4.0.1 only supported Level 1 (no combiningcharacters) of ISO 10646-1 with fixed character width andleft-to-right writing direction. In other words, the terminalsemantics were basically the same as for ISO 8859-1, except that itcan now decode UTF-8 and can access 16-bit characters.With XFree86 4.0.3, two important functions were added:automatic switching to a double-width font for CJK ideographssimple overstriking combining charactersIf the selected normal font is X × Y pixelslarge, then xterm will attempt to load in addition a2X × Y pixels large font (same XLFD, exceptfor a doubled value of the AVERAGE_WIDTH property). Itwill use this font to represent all Unicode characters that have beenassigned the East Asian Wide (W) or East Asian FullWidth(F) property in Markus Kuhn’s free wcwidth()implementation can be used by applications on platforms where theC library does not yet provide an equivalent function to find, howmany column positions a character or string will occupy on a UTF-8terminal emulator screen.Markus Kuhn’s transtab is atransliteration table for applications that have to make a best-effortconversion from Unicode to ASCII or some 8-bit character set. Itcontains a comprehensive list of substitution strings for Unicodecharacters, comparable to the fallback notations that people usecommonly in email and on typewriters to represent unavailablecharacters. The table comes in ISO/IEC TR 14652 format, to allowsimple inclusion into POSIX locale definition files.The Pango – Unicode and ComplexText Processing project added full-featured Unicode support to GTK+.Qt supported the use of*-ISO10646-1 fonts since version 2.0.A UTF-8 extension for theFast Light Tool Kit was prepared byJean-Marc Lienher, based on his Xutf8 Unicode display library.What packages with UTF-8 support are currently under development?Native Unicode support is planned for Emacs 23. If you areinterested in contributing/testing, please join theemacs-devel @gnu.org mailing list.The Linux ConsoleProject works on a complete revision of the VT100 emulator builtinto the Linux kernel, which will improve the simplistic UTF-8 supportalready there.How does UTF-8 support work under Solaris?Starting with Solaris 2.8, UTF-8 is at least partially supported.To use it, just set one of the UTF-8 locales, for instance by typing setenv LANG en_US.UTF-8in a C shell.Now the dtterm terminal emulator can be used to inputand output UTF-8 text and the mp print filter will printUTF-8 files on PostScript printers. The en_US.UTF-8locale is at the moment supported by Motif and CDE desktopapplications and libraries, but not by OpenWindows, XView, andOPENLOOK DeskSet applications and libraries.For more information, read Sun’s |
|
