If you are a native English speaker, particularly an American, you may never have thought much about what is required for the internationalization of programs for the simple reason that all the programs you use already speak your language. There are four general areas that require attention when writing an internationalized application:

When thinking about applications that run in other languages, it is important to recognize the distinction between an internationalized application and a multilingual application. A text editor that works in any given locale is internationalized; a mail reading program that labels its push buttons with text in the language of the locale is internationalized, but if it also allows a user to compose mail in a second language and include excerpts from a message in a third language, then it is multilingual. The requirements and problems of multilingual applications are not yet well understood, and the X Consortium made a considered decision that R5 would support internationalized applications but not explicitly support multilingual ones.

The following sections continue this introduction to internationalization with a description of the ANSI-C setlocale mechanism and a further discussion of character encoding and text representation issues.

setlocale(LC_ALL, "");

language[_territory[.codeset]]

Text Representation in an Internationalized Application

Think for a minute about the fundamentals of text representation by computer. Remember that characters displayed by your computer are represented by numbers. The correspondence between numbers and characters (on most American computers) is defined by the ASCII (American Standard Code for Information Interchange) encoding. There is nothing special about ASCII except that it is one of the most firmly established standards of the computer world. Text composed in one encoding (ASCII, for example) and displayed in another (perhaps EBCDIC, still used by IBM mainframes) will be nonsense because the number-to-character mappings of the encodings are not the same.

We've been using the term encoding rather loosely. Before we consider text representation any further, some definitions are appropriate. A character is an abstract element of text, distinct from a font glyph, which is the actual image that gets displayed. A character set is simply a set of characters; there are no numbers associated with those characters. We are all familiar with the character set used by ASCII. The Latin-1 character set used by many Western European Latin-based languages is an extension of ASCII that contains the accented characters required by many of those languages. An encoding is any numeric representation of the characters in a character set. The term codeset is sometimes used as a synonym for encoding. A charset (not the same as a character set) is an encoding in which all characters have the same number of bits. ASCII is a 7-bit encoding, for example, and is therefore a charset. Figure 10-1 diagrams the relationship between character sets, charsets, fonts, and font glyphs.

Figure 10-1. Character sets, encodings, charsets, fonts, and glyphs

The last two fields of an X font name specify a charset. By definition, the index of a font glyph in the font is the same as the encoding of the corresponding character in that charset. When the encoding of a locale is a charset, this obviously simplifies matters a great deal: text in the locale can be displayed using glyphs from a single font, and the character encoding can be used directly as the index of the corresponding font glyph.

Not all languages can be represented with a single charset, however. Japanese text, for example, commonly requires Japanese ideographic characters, Japanese phonetic characters, and Latin characters. Each of these character sets has its own standard fixed-width encoding, and is therefore a charset. Note, however, that the ideographic charset is 16-bits wide while the phonetic and Latin charsets are 8-bits wide. Full Japanese text display requires a font for each charset, and Japanese text representation requires a “super-encoding” that combines each of the component encodings. There are, in fact, several encodings commonly used for Japanese text. What they have in common is the use of “shift sequences” to indicate which charset the following character belongs to.

It is crucial to the concept of a locale that each locale has a single well-defined encoding. Many languages have only a single standardized encoding. If a language can be encoded in more than one standard way, each encoding defines a locale of its own, and the name of the encoding is part of the name of the locale.

ISO8859-1 and Other Encodings

If you examine the names of the X fonts on your system (using xlsfonts) you will probably find that most of them have the charset “iso8859-1.” This charset is sometimes called “Latin-1” and was designed to be suitable for use by most Western European languages (Greek being a notable exception). The character set of ISO8859-1 comprises all the ASCII characters plus a wide variety of accented and special characters. (You can take a look at the characters using the xfd program.) Because there are fewer than 256 characters in the set, ISO8859-1 can use a state-independent 8-bit encoding. This means that all characters are 8 bits long, and there are no special shift sequences that modify the interpretation of characters. Because there are not any shift sequences, it is possible to use the encoding of all Latin-1 characters directly as font indices.

ISO8859-1 contains a superset of the ASCII characters. Every character in the ASCII character set has the same encoding in Latin-1 as it does in ASCII. (But Latin-1 does not define any control characters such as linefeed, backspace or the bell character.) Because it is an 8-bit encoding, Latin-1 strings can be represented using the usual C null-terminated array of char. Because the characters are a uniform 8 bits and because strings do not contain embedded shift states, it is possible to use Latin-1 strings with the standard C string manipulation routines (strlen, strcat, etc.) In conjunction with the ANSI-C internationalization facilities, the careful design of ISO8859-1 means that most programs originally written for ASCII use can easily be ported for use in most Western European countries.

But it is not so simple once we try to go beyond Western Europe and Latin-based alphabets. Japanese text, for example, commonly uses (at least within the computer industry) words written in the Latin alphabet along with phonetic characters from the katakana and hiragana alphabets and ideographic kanji characters. Each of these types of text has its own charset (8- or 16-bit), but they must be combined into a single encoding for Japanese text. This is done with shift sequences, bytes embedded in the running text which control the character set in which the following character will be interpreted. It is possible to use “locking shifts” which modify the interpretation of the next and subsequent characters, but this scheme is infrequently used because it makes strings of text very difficult to manipulate.

Compound Text is another text representation that is used in X applications. Compound Text strings identify their encoding using embedded escape sequences (they can also have multiple sub-strings with multiple encodings) and are therefore locale-independent. The Compound Text representation was standardized as part of X11R4 for use as a text interchange format for interclient communication. It is often used to encode text properties and for the transfer of text via selections, and is not intended for text representation internal to an application. There are new R5 routines that convert X property values to and from the Compound Text representation. Note that Compound Text is not the same thing as the Compound Strings used by the Motif widget set.

Multi-byte Strings and Wide-character Strings

Strings in encodings that contain shift sequences and characters with non-uniform width can be stored in standard NULL-terminated arrays of characters, but can be difficult to work with in this form: the number of characters in a string cannot be assumed to be equal to the number of bytes, and it is not possible to iterate through the characters in a string by simply incrementing a pointer. On the other hand, strings of char are usefully passed to standard functions like strcat and strcpy, and assuming a terminal that understands the encoding, functions like printf work correctly with these strings.

As an alternative to these multi-byte strings, ANSI-C defines a wide-character type, wchar_t, in which each character has a fixed size and occupies one array element in the string. (The wchar_t is 2 bytes on some systems, 4 bytes on others, and may be 1 byte on systems that support nothing but the default C locale.) ANSI-C defines functions to convert between multi-byte and wide-character strings: mblen, mbstowcs, mbtowc, wcstombs, and wctomb. ^[32] As you can see here, and as you will see with the R5 internationalized text input and output functions, “multi-byte” is commonly abbreviated “mb” in function names, and “wide character” is abbreviated “wc.” Multi-byte strings are usually more compact than wide-character strings, but wide-character strings are easier to work with. Note that ANSI-C does not provide wide-character string manipulation functions. There is, however, a contributed library of wide character functions that is shipped with the MIT R5 release; see the directory contrib/lib/Xwchar.

In an internationalized application, you must take care to handle all strings properly. Unfortunately the ANSI-C library does not provide adequate functions or conventions for sophisticated internationalized text manipulation. Note, though, that many applications can do internationalized text input and output without performing any manipulations on that text. The following list gives a few guidelines for handling internationalized strings:

• Multi-byte strings are null-terminated. There is no single convention for the termination of wide character strings, but strings passed to wcstombs are null-terminated. As was the case before R5, X text output and input functions take and return strings with a count of the characters they contain.

• If an encoding is state-dependent (i.e., if it uses locking shifts) multi-byte strings are assumed to begin in the default shift state of the encoding. There is no convention for the shift state at the end of a string, so when concatenating two strings, the first may need to be reset to the default shift state in order to guarantee correct interpretation of the second. In practice, state-dependent encodings are rarely used.

• None of the C library string-handling functions work with wide-character strings.

• The following C string-handling functions may be safely used with multi-byte strings (in a state-independent encoding): strcat, strcmp, strcpy, strlen, strncmp. Note that the string comparison routines are only useful to check for byte-for-byte equality. To compare strings for sorting, use strcoll.

• Multi-byte strings can be written to file or output streams. Assuming a terminal that operates in the current locale, printing a multi-byte string to stdout or stderr will cause the correct text to be displayed.

• Multi-byte strings can be read from files or from the stdin input stream. If the file is encoded in the current locale, or the terminal operates in the locale, then the strings that are read will be meaningful.

An internationalized X application begins in the same way as a ANSI-C terminal-based internationalized program: with a call to setlocale. An X program, however, generally goes two steps further.

Immediately after calling setlocale, an application should call XSupportsLocale() to determine if the Xlib implementation supports the current locale. This function takes no arguments and return a Bool. If this function returns False, an application will typically print a “Locale not supported” message and exit.

After verifying that the locale is supported, an application should call XSetLocaleModifiers(). A “locale modifier” can be thought of as an extension to the name of a locale; it specifies more information about the desired localized behavior of an application. R5 as shipped by MIT recognizes one locale modifier, used to specify the input method (see Chapter 10, “Internationalization”) to be used for internationalized text input for the locale.

XSetLocaleModifiers() allows the programmer to specify a list of modifiers (usually none) which will be concatenated with a list of user-specified modifiers from an operating system environment variable (XMODIFIERS in POSIX). The strings passed to XSetLocaleModifiers() and set in the XMODIFIERS environment variable are a series of concatenated “@category=value” strings. Thus to specify that the “Xwnmo” input method should be used by an application, a user might set the XMODIFIERS as follows:

setenv XMODIFIERS @im=_XWNMO

Example 10-1 shows code that uses setlocale and the two functions described here to correctly establish its locale.

#include <stdio.h>

#include <X11/Xlib.h>

/*
 * include <locale.h> or the non-standard X substitutes
 * depending on the X_LOCALE compilation flag
 */
#include <X11/Xlocale.h>

main(argc, argv)
int argc;
char *argv[];
{
    char *program_name = argv[0];
    /*
     * The error messages in this program are all in English.
     * In a truly internationalized program, they would not be
     * hardcoded; they would be looked up in a database of some sort.
     */
    if (setlocale(LC_ALL, "") == NULL) {
        (void) fprintf(stderr, "%s: cannot set locale., program_name);
        exit(1);
    }
    if (!XSupportsLocale()) {
        (void) fprintf(stderr, "%s: X does not support locale %s.,
                       program_name, setlocale(LC_ALL, NULL));
        exit(1);
    }
    if (XSetLocaleModifiers("") == NULL) {
        (void) fprintf(stderr, "%s: Warning: cannot set locale modifiers.,
                       program_name);
    }
        .
        .
        .
}

Not all systems support the setlocale function, but X can be built for these systems by defining the X_LOCALE compilation flag. When writing programs in an environment that does not have setlocale, include the header file <X11/Xlocale.h>. If this file is compiled with X_LOCALE defined, it defines setlocale as a macro for an Xlib-internal function. Otherwise, it simply includes the standard header <locale.h> to get the correct declaration of the real setlocale.

Before R5, the Xlib drawing routines made the fundamental assumption that the encoding of a character was equal to the index of the character's glyph in the font. As explained in “Text Representation in an Internationalized Application” this is a useful and valid assumption when text in a language can be most naturally encoded as an 8- or 16-bit wide charset. Unfortunately, it is not valid in many important cases.

R5 bases its new text output routines on a new Xlib abstraction, the XFontSet. An XFontSet is bound to the locale in which it is created, and contains all the fonts needed to display text in that locale, or all the independent charsets used in the encoding of that locale. Technical Japanese text, for example, often mixes Latin with Japanese characters, so for a Japanese locale, fonts might be required with the charsets jisx0208.1983-0 for Kanji ideographic characters, jisx0201.1976-0 for Kana phonetic characters, and iso8859-1 for Latin characters.

Drawing internationalized text in R5 is conceptually very similar to drawing text in X11R4--there are routines that allow you to query font metrics, measure strings, and draw strings. The new R5 functions use an XFontSet rather than an XFontStruct or a font specified in a graphics context. The drawing and measuring routines interpret text in the encoding of the locale of the fontset, and correctly map wide or multi-byte characters to the corresponding font glyph (or glyphs).

XFontSet fontset;
char **missing_charsets;
int num_missing_charsets = 0;
char *default_string;
int i;
        .
        .
        .
fontset = XCreateFontSet(dpy,
                         "-misc-fixed-*-*-*-*-*-130-75-75-*-*-*-*",
                         &missing_charsets, &num_missing_charsets,
                         &default_string);
/*
 * if there are charsets for which no fonts can
 * be found, print a warning message.
 */
if (num_missing_charsets > 0) {
    (void)fprintf(stderr, "%s: The following charsets are missing:,
                  program_name);
    for(i=0; i < num_missing_charsets; i++)
        (void)fprintf(stderr, "%s: %s, program_name,
                      missing_charsets[i]);
    (void)fprintf(stderr, "%s: The string %s will be used in place,
                  program_name, default_string);
    (void)fprintf(stderr, "%s: of any characters from those sets.,
                  program_name);
    XFreeStringList(missing_charsets);
}
        .
        .
        .

typedef struct {
    XRectangle max_ink_extents;          /* over all drawable characters */
    XRectangle max_logical_extents;      /* over all drawable characters */
} XFontSetExtents;

typedef struct {
    char        *chars;                 /* pointer to string */
    int         nchars;                 /* number of bytes in string */
    int         delta;                  /* pixel delta between strings */
    XFontSet    font_set;               /* fonts, None means don't change */
} XmbTextItem;
typedef struct {
    wchar_t     *chars;                 /* pointer to wide char string */
    int         nchars;                 /* number of wide characters */
    int         delta;                  /* pixel delta between strings */
    XFontSet    font_set;               /* fonts, None means don't change */
} XwcTextItem;

#include <X11/Xlib.h>

/*
 * This function draws a specified multi-byte string centered in
 * a specified region of a window.
 */
void DrawCenteredMbString(dpy, w, fontset, gc,
                          str, num_bytes, x, y, width, height)
Display *dpy;
Window w;
XFontSet fontset;
GC gc;
char *str;
int num_bytes;
int x, y, width, height;
{
    XRectangle boundingbox;
    XRectangle dummy;
    int originx, originy;
   /*
    * Figure out how big the string will be.
    * We should be able to pass NULL instead of &dummy, but
    * XmbTextExtents is buggy in the Xsi implementation.
    * Also, it should return the escapement of the string, but doesn't.
    */
    (void) XmbTextExtents(fontset, str, num_bytes,
                          &dummy, &boundingbox);
   /*
    * The string we want to center may be drawn left-to-right,
    * right-to-left, or some of both, so computing the
    * drawing origin is a little tricky.  The bounding box's x
    * and y coordinates are the upper left hand corner and are
    * relative to the drawing origin.
    * if boundingbox.x is 0, the string is pure left-to-right.
    * If it is equal to -boundingbox.width then the string is pure
    * right-to-left, but it may not be either of these, so what
    * we've got to do is choose the origin so that the bounding box
    * is centered in the window without assuming that the origin is
    * at one end or another of the string.
    */
    originx = x + (width - boundingbox.width)/2 - boundingbox.x;
    originy = y + (height - boundingbox.height)/2 - boundingbox.y;
   /*
    * now draw the string
    */
    XmbDrawImageString(dpy, w, fontset, gc,
                       originx, originy,
                       str, num_bytes);
}

Perhaps the most fundamental concern of internationalization is the encoding of strings. So far we've considered text drawing and string input, and have used multi-byte or wide-character strings in the encoding of the locale. Because X is a networked window system, however, an X client must communicate with the X server, usually with a window manager, sometimes with a session manager, and often with other clients through the X selection mechanism (which is used to implement copy-and-paste). When we allow the internationalization of X programs, we must confront the issues of communication between clients that use different locales, and of communication between an internationalized client and a “locale-neutral” X server. Furthermore we must make decisions about the encodings of any other strings used in the X and Xt specifications.

Some of the issues that must be considered are the appropriate encoding for color and font names passed to the X server, the encoding of bitmap files, the encoding of strings selected in one client and copied to another, and the encoding of resource values and names. When making decisions on questions like these, the designers of X internationalization had several choices. They could specify that particular strings were:

Compound text is an encoding designed to represent text from any locale. As such it is well suited to be a standard string format for clients that communicate using string properties. It does not, however, address the problem of converting strings from one locale to another, and often this is simply not possible. In most cases it is not meaningful to select text from an application running in one locale and paste it into an application running in a different one. This is the realm of multilingual applications which are not addressed by R5.

Note that the above list refers to the COMPOUND_TEXT and STRING encodings. These capitalized names refer to the Atom names used in the ICCCM to specify the type of a “Property.” The ICCCM also specifies a selection conversion target Atom, TEXT, which simply means a string in whatever encoding is convenient for the selection owner.

Sometimes the best choice of encodings is ASCII. It may seem unfair to non-English locales that the ASCII encoding should be singled out for special treatment, but for strings that are to be shared between X client and X server (such as Display, Property, and font and color names) some standard encoding must be specified. Because ASCII is widespread and is the usual encoding for C programming, it is a natural choice. In many cases, though, it is not the specific ASCII encoding that is important, but the fact that there is some common encoding for all the characters used by ASCII. R5 never actually refers to ASCII. Instead, it defines the X Portable Character Set as a set of basic characters that must exist in all locales supported by Xlib. Those characters are:

a..z A..Z 0..9
!"#$%&'()*+,-./:;<=>?@[\]^_‘{|}~
<space>, <tab>, and <newline>

R5 also defines the Host Portable Character Encoding as the encoding for that character set. The encoding itself is not defined; the only requirement is that the same encoding is used for all locales on a given host machine. A string in the Host Portable Character Encoding is understood to contain only characters from the X Portable Character Set. Finally, the Latin Portable Character Encoding is the characters of the X Portable Character Set encoded as a subset of the Latin-1 encoding. (Latin-1 is itself a superset of ASCII.) Note that if an X client running on one host has a different portable encoding than an X server running on a different host, then translation from one encoding to the other will be required (for color names, font names, etc.) and would be done by the Xlib communication layer. In practice, however, it is likely that all systems will simply use an encoding which is a superset of ASCII, (with the possible exception of mainframes that use EBCDIC) and therefore all characters in the X Portable Character Set will share a single, standard (ASCII) encoding. Appendix K of Volume Two summarizes all the encodings.

String-encoding issues arise throughout Xlib, and particularly so for functions that involve X properties and resource databases. The internationalization of client-to-window-manager and client-to-client communication via properties is described in 10.5 below and the internationalization of X resource databases is discussed in 10.6. Here we itemize the remaining changes to the Xlib specification that involve string encodings. Table 10-1 lists Xlib functions and the encodings of the strings that are passed in and out of them. These are not so much changes to the Xlib specification as clarifications of it to make the encodings explicit.

You'll need to understand non-internationalized interclient communication before reading this; see Chapter 12.

When writing an internationalized application it is not safe to assume that all interclient communication with text properties will be done with Latin-1 or ASCII strings. R5 provides some new functions that do not make this assumption. The first is a convenience routine for communication with window managers. XmbSetWMProperties() is a function very similar to XSetWMProperties(), except that the window_name and icon_name arguments are multi-byte strings (rather than XTextProperty pointers) in the encoding of the locale. If these strings can be converted to the STRING encoding (Latin-1 plus newline and tab), then their corresponding WM_NAME and WM_ICON_NAME properties are created with type STRING. If this conversion cannot be performed, the strings are converted to Compound Text (this conversion can always be done, by the definition of Compound Text), and the properties are created with type COMPOUND_TEXT. Note that there is no wide-character version of this function.

Since X properties have a single contiguous block of data as their value, they cannot directly represent types such as char **. But sometimes such a complex type must be represented (imagine a text editor setting a property to a set of disjointed selected strings). To allow this, X11R4 defined the XTextProperty structure (shown in Example 10-6) and the functions XStringListToTextProperty() and XTextPropertyToStringList().

typedef struct {
        unsigned char *value;   /* property data */
        Atom encoding;          /* type of property */
        int format;             /* 8, 16, or 32 */
        unsigned long nitems;   /* number of items in value */
} XTextProperty;

These functions assume input strings are in Latin-1 and always create properties of type STRING, which is not correct behavior in internationalized applications. So R5 provides the new functions Xmb/XwcTextListToTextProperty() and Xmb/XwcTextPropertyToTextList() which operate correctly with localized strings, converting between text encoded in the locale and STRING or COMPOUND_TEXT types. The Xmb/wcTextListToTextProperty() functions take a new argument of type XICCEncodingStyle, which is shown in Example 10-7.

typedef enum {
        XStringStyle,           /* STRING */
        XCompoundTextStyle,     /* COMPOUND_TEXT */
        XTextStyle,             /* text in owner's encoding (current locale) */
        XStdICCTextStyle        /* STRING, else COMPOUND_TEXT */
} XICCEncodingStyle;

The style argument to these functions specifies how the text is to be converted. The possible values have the following meanings:

The returned XTextProperty is suitable to pass to XSetTextProperty().

The other two routines, Xmb/XwcTextPropertyToTextList(), perform the conversion in the opposite direction. They are passed an XTextProperty (obtained with a call to XGetTextProperty(), perhaps) and return an array of pointers to char * or an array of pointers to wchar_t *. These routines do not require an argument of type XICCEncodingStyle; they always convert from the encoding of the property to the encoding of the current locale if such a conversion is possible. The application is responsible for freeing the memory allocated by these functions. To free the array of multi-byte strings (and the strings themselves) returned by XmbTextPropertyToTextList() use XFreeStringList(), which is a pre-R5 function. To free the array of wide-character strings (and the strings themselves) allocated by XwcTextPropertyToTextList() use the new function XwcFreeStringList().

These four functions return an integer. The possible values and their meanings are as follows:

Table 10-2 shows the possible results of the conversions performed by Xmb/XwcTextListToTextProperty() and Xmb/XwcTextPropertyToTextList().

When there are unconvertible characters in a string, the conversion functions substitute a locale-dependent default string (encoded in the current locale). The value of the default string may be queried with XDefaultString(), and may be the empty string (""). There is no way to set the value of the default string. The default string is independent of the default string used by the R5 text-drawing routines when an XFontSet does not contain all the characters needed to represent text in a locale.

We've seen that X resources are a useful way to allow the localization of strings--rather than hardcoding its strings, an X client can look them all up by name from a locale-dependent resource file. The twist here is that although resource values can be localized, and may contain text in the encoding of the locale, resource names must still be hardcoded into the application. As you might expect, R5 specifies that resource names in the Host Portable Character Encoding are always supported, and that any other encodings are implementation-dependent. What this means is that a Chinese user who wishes to customize the behavior of an application written by a Japanese programmer will have to specify values for resources that are named using Latin characters in the X Portable Character Set. Those resource names may be English phonetic representations of Japanese words which are mnemonic to the Japanese programmer, but which are meaningless to the Chinese (or American) user. This situation is unfortunate but there is no way around it within the scope of the X Resource Manager mechanisms. If resource names are to be localized, they would have to be looked up in a database as well, and then we would need hardcoded names for the names. Another approach would be to use resource numbers in place of resource names. These remain constant across all locales, but where a resource name is mnemonic to the original programmer, at least, a resource number would be mnemonic to no one.

When a resource file or string are parsed into an XrmDatabase(), that parsing is done in the current locale, and the database is bound to that locale even if the current locale changes. We can speak of the “locale of the database” in the same way that we speak of the “locale of the XFontSet.” To determine the locale of a database, call XrmLocaleOfDatabase().

The internationalization of resources requires additions to the Xlib specification to make explicit the encoding and interpretation of the strings that are passed in and out of the Xrm functions. Table 10-4 lists the resource manager functions that have been respecified.

This chapter has covered a lot of tricky material. The following guidelines summarize the requirements for ANSI-C and R5-based internationalization: