Locale refers to a set of local customs that determine many aspects of software input and output formatting, including natural language, culture, character sets and encodings, and formatting and sorting rules. The locale of a program is the set of such parameters that are currently selected. For information on the method for selecting locales, see “Additional Reading on Internationalization” below.

Internationalization is the process of making a program capable of running in multiple locales without recompiling. To put it another way, an internationalized program is one that can be easily localized without changing the program itself. (See “Localization (l10n),” below, for an explanation of the term “localization.”)

---

Note: The word “internationalization” consists of an i followed by 18 letters followed by an n. It is thus often abbreviated “i18n” in informal writing. On similar principles, “localization” is often abbreviated “l10n.”

---

A program written for a specific locale may be difficult to run in a different environment. Rewriting such a program to operate in each desired environment would be tedious and costly.

Your goal as a developer should thus be to write locale-independent programs, programs that make no assumptions about languages, local customs, or coded character sets. Such internationalized applications can run in a user's native environment following native conventions with native messages, without recompiling or relinking. A single copy of an internationalized program can be used by a world of different users.

Localization is the act of providing an internationalized application with the environment and data it needs to operate in a particular locale. For example, adding German system messages to IRIX is a part of localizing IRIX for the German locale.

Nationalized programs run in only one language and are governed by one set of customs; in other words, in a nationalized program the locale is built into the application. Even if the application doesn't use ASCII or English, as long as it is a single-language program it is nationalized, not internationalized. Most older UNIX programs can be thought of as being nationalized for the United States.

Consider two applications, hello and bonjour. The application hello always produces the output

Hello, world.

and bonjour always produces

Bonjour, tout le monde.

Neither hello nor bonjour are internationalized; they are both nationalized.

There are no special requirements for writing or porting nationalized applications, whether they are text or graphics programs. Terminal-based programs work on suitable terminals, including internationalized terminal emulators. “Suitable” means that the terminal supports any necessary fonts and understands the encoding of the application output. Graphics programs simply do as they have always done. Applications using existing interfaces to operate in non-English or non-ASCII environments should continue to compile and run under an internationalized operating system.

A multilingual program is one that uses several different locales at the same time. Examples are described in “Multilingual Support”.

Applications need not perform every aspect of their work in the same locale. Although this approach is not recommended, an application could (for example) perform most of its activities in the English locale but use French sorting rules. You can use locale categories to do this kind of locale-mixing. (Mixing locale categories is not the same as multilingual support—see “Multilingual Support.”)

The category argument is a symbolic constant that tells setlocale() which items in a locale to change. Table 16-1 lists the available category choices.

Categories correspond to databases that contain relevant information for each defined locale. The locations of these databases are given in the “Location of Locale-Specific Data”.

The setlocale() function attempts to set the locale of the specified category to the specified locale. You should almost always pass the empty string as the locale parameter to conform to user preferences.

On success, setlocale() returns the new value of the category. If setlocale() couldn't set the category to the value requested, it returns NULL and does not change locale.

An empty string passed as the locale parameter is special. It specifies that the locale should be chosen based on environment variables. This is the way a user specifies a preferred locale, and that preference should almost always be honored. The variables are checked hierarchically, depending on category, as shown in Table 16-2; for instance, if the category is LC_COLLATE, an empty-string locale parameter indicates that the locale should be chosen based on the value of the environment variable LC_COLLATE—or, if that value is undefined, the value of the environment variable LANG, which should contain the name of the locale that the user wishes to work in.

Specifying the category LC_ALL attempts to set each category individually to the value of the appropriate environment variable.

If no non-null environment variable is available, setlocale() returns the name of the current locale.

Here are the possibilities for specifying the locale parameter:

Except for XPG/4 message catalogs, locale-specific data (that is, the “compiled” files containing the collation information, monetary information, and so on) are located in /usr/lib/locale/<locale>/<category>, where <locale> and <category> are the names of the locale and category, respectively. For example, the database for the LC_COLLATE category of the French locale fr would be in /usr/lib/locale/fr/LC_COLLATE.

There will probably be multiple locales symbolically linked to each other, usually in cases where a specific locale name points to the more general case. For example, /usr/lib/locale/POSIX might point to /usr/lib/locale/C.

A locale string is of the form

language[_territory[.encoding]][@modifier]...

where

Language data is implementation specific; databases for the language en (English) might contain British cultural data in England and American cultural data in the United States. If other than the default settings are required, the territory field may be used. For example, the above cases could be more strictly defined by setting LANG to en_GB or en_US. Full rigor might lead to en_GB.ISO 8859-1 for England and en_US.ISO 8859-1 for the USA.

ANSI C has defined a special locale value of C. The C locale is guaranteed to work on all compliant systems and provides the user with the system's default locale. This default is typically American English and ASCII, but need not be. POSIX has also defined a special locale value, POSIX, which is identical to the C locale.

The length of the locale string may not exceed NL_LANGMAX characters (NL_LANGMAX is defined in /usr/include/limits.h). However, XPG/4 recommends that this string (not counting modifiers) not exceed 14 characters.

There can be only one locale at a time associated with any given process in an internationalized system. Therefore, although multilingual applications—which give the appearance of using more than one locale at a time—can be created, internationalization does not provide inherent support for them. Here are two examples of multilingual programs:

In writing a multilingual application, the first task is identifying the locales for the program to run in and when they apply. (There is no standard method for performing this task.) Once the application has chosen the desired locales, it must do one of the following:

The LANG environment variable and the locale variables provide the freedom to configure a locale, but they do not protect the user from creating a nonsensical combination of settings. For example, you are allowed to set LANG to fr (French) and LC_COLLATE to ja_JP.EUC (Japanese). In such a case, string routines would assume text encoded in 8859-1—except for the sorting routines, which might assume French text and Japanese sorting rules. This would likely result in arbitrary-seeming behavior.

The IRIX filesystem does not contain information about what encoding should be associated with any given data. Thus, applications must assume that data presented to an application in some locale is properly encoded for that locale. In other words, a file is interpreted differently depending on locale; there is no way to ask the file what it thinks its encoding is.

For example, you may have created a file while in a Japanese locale using EUC. Later, you might try printing it while in a French locale. The results will likely resemble a random collection of Latin 1 characters.

This problem applies to almost all stored strings. Most strings are uninterpreted sequences of nonzero bytes. This includes, for example, filenames. You can, if you want to, name your files using Chinese characters in a Chinese locale, but the names will look odd to anyone who runs /bin/ls on the same filesystem using a non-Chinese locale.

Multibyte strings are very easy to pass around. They efficiently use space (both data and disk space), since “extra” bytes are used only for characters that require them. MB strings can be read and written without regard to their contents, as long as the strings remain intact. Displaying MB strings on a terminal is done with the usual routines: printf(), puts(), and so on. Many programs (such as cat) need never concern themselves with the multibyte nature of MB strings, since they operate on bytes rather than on characters; so MB strings are often used for string I/O.

Manipulation of individual characters in an MB string can be difficult, since finding a particular character or position in a string is nontrivial (see “Handling Multibyte Characters,” below). Therefore, it is common to convert to WC strings for that kind of work.

Usually, multibyte characters are handled just like char strings. Editing such strings, however, requires some care.

You cannot tell how many bytes are in a particular character until you look at the character. You cannot look at the nth character in a string without looking at all the previous n - 1 characters, because you cannot tell where a character starts without knowing where the previous character ends. Given a byte, you don't know its position within a character. Thus, we say the string has state or is context-sensitive; that is, the interpretation we assign to any given byte depends on where we are in a character.

This analysis of characters is locale-dependent, and therefore must be done by routines that understand locale.

Multibyte characters and strings are convertible to wchars using mbtowc() for individual characters and mbstowcs() for strings (see the mbtowc(3) and mbstowcs() reference pages).

To find out how many bytes make up a given single MB character, use mblen(), as shown in Example 16-1 (see also the mblen(3) reference page).

#include <stdlib.h>
. . .
size_t n;
int len;
char *pStr;
. . .
len = mblen(pStr, n); /* examine no more than n bytes */

It is the application's responsibility to ensure that pStr points to the beginning of a character, not to the middle of a character.

The maximum number of bytes in a multibyte character is MB_LEN_MAX, which is defined in limits.h. The maximum number of bytes in a character under the current locale is given by the macro MB_CUR_MAX, defined in stdlib.h.

Since strlen() simply counts bytes before the first NULL, it tells you how many bytes are in an MB string.

When mbstowcs() converts MB strings to WC strings, it returns the number of characters converted. This is the simplest way to count characters in an MB string.

---

Note: Many code segments that deal with individual characters within a string are better served by wide character strings. Because counting often involves conversion, such segments are often better served by working with a WC string, then converting back.

---

Getting the length without performing the conversion is straightforward, but not as simple. mbtowc() converts one character and returns the number of bytes used, but returns the same information without conversion if a NULL is passed as the address of the WC destination. Thus

len = mblen(pStr, n);

is equivalent to

len = mbtowc((wchar_t *) NULL, pStr, n);

In fact, mblen() calls mbtowc() to perform its count. Therefore, counting characters in an MB string without converting would look like the code in Example 16-2.

int cLen;
char *tStr = pStr;
numChars = 0;
cLen = mbtowc((wchar_t *) NULL, tStr, MB_CUR_MAX);
while (cLen > 0) {
    tStr += cLen;
    numChars++;
    cLen = mbtowc((wchar_t *) NULL, tStr, MB_CUR_MAX);
    if (cLen == -1)
        numChars = cLen; /* invalid MB character */
}

The single advantage of WC strings is that all characters are the same size. Thus, a string can be treated as an array, and a program can simply index into the array in order to modify its contents. Most applications' char manipulation routines work with little modification other than a type change to wchar_t, with appropriate attention to byte count and sizeof().

So, when applications have significant string editing to perform, they typically keep the strings in WC format while doing that editing. Those WC strings may or may not be converted to or from MB strings at other points in the application.

Wide characters are often large and are not as space efficient as multibyte strings. Applications that do not need to perform string editing probably shouldn't use wchars. If an application intends to both maintain and edit large numbers of strings, then the developer needs to make size and complexity trade-off decisions.

Analogs to the routines defined in string.h and stdio.h are supplied in libw.a and defined in widec.h. This includes routines such as getwchar(), putwchar(), putws(), wscpy(), wslen(), and wsrchr() (see the wcstring(3) reference page).

Wide characters and strings are convertible to MB strings via wctomb() and wcstombs(), respectively.

printf() function, detailed in the printf(3S) reference page, examines LC_NUMERIC and chooses the appropriate decimal radix. If none is available, it tries to use ASCII period. No further locale-specific formatting is done directly by printf(). However, see “Variably Ordered Referencing of printf() Arguments,” for a way to handle locale-specific ordering of syntactic elements in messages.

The localeconv() function, detailed in the localeconv(3C) reference page, can be called to find out about numeric formatting data, including the decimal radix (inappropriately called decimal_point), the grouping separator (inappropriately called thousands_sep), the grouping rules, and a great deal of monetary formatting information.

The localeconv() function leaves actual use of formatting information other than the decimal radix to the application.

The strfmon() function, detailed in the strfmon(3S) reference page, is new with IRIX version 6.2. Like sprintf(), strfmon() takes an output area, a format string that contains conversion specifications, and one or more argument values to be converted. It creates an output string containing fixed data and converted values.

Only two conversion types are supported: %i to convert a double value to international currency representation, and %n to convert a double value to national currency representation. You can use strfmon() to format currency values as strings, and then use printf() or other functions to write the formatted strings.

To meet the requirements of local customs, the X/Open Native Language System (NLS) interface provides a set of library functions that allow cultural data appropriate to the user to be determined at run-time.

Regular expressions provide pattern-matching facilities for text. A variety of regular expression support libraries are supplied with IRIX. Most of them parse regular expressions in terms of machine collating sequences, the English language, and the ASCII coded character set.

When a program deals with internationalized input text, it is important to extend regular expression facilities to cover internationalized strings and coded character sets. It is difficult to write regular expressions that apply to more than one language, or to languages with accented/multi-character collating elements because of limitations in syntax.

Application programs can use the wsregexp function library, documented in the wsregexp(3W) reference page, to support internationalized regular expression behavior.

The American National Standards Committee X3J11 standard for the C programming language includes a number of library functions that are defined to operate internationally; that is, they modify their operation in a manner appropriate to the user's native language and cultural environment.

The X/Open definition includes the international functions in Table 16-4 as defined in Draft ANSI X3.159, Programming Language C. ANSI functions that are enhanced by the X/Open definition are marked with an asterisk.

Draft ANSI X3.159, Programming Language C also defines a number of multi-byte functions, and an additional function for manipulating monetary values. At this stage, the X/Open definition is only guaranteed to work correctly for single-byte 8-bit characters, and thus does not include the multi-byte functions.

In addition, X/Open defines internationalized regular expression compile and match functions, native language message-handling functions, and native language versions of the error-handling functions (see Table 16-5).

Configuration data identify the languages supported on a system in terms of the recognized settings of language, territory, and codeset. Each valid combination of these settings has its own set of collating sequence, character classification and shift tables, language information data, and message catalogs.

Collating sequence tables define the collating sequence for each supported language. The binary values of characters in the associated coded character set are used as indices into the table, individual entries of which indicate the relative position of that character in the language collating sequence. The interface definition supports the following capabilities:

These capabilities extend to providing support for the relative ordering of collating elements within an equivalent class (for example, where two characters are first compared for equality ignoring accents, and if equal, are then ordered by accent sequence).

These contain the lookup tables for character classification. Each character code from the defined coded character set is used as an index into the relevant language lookup table. Each entry language lookup table contains a series of flags identifying the truth or falsehood of a particular language assertion, such as

Shift tables contain the corresponding upper- and lower-case combinations for each character defined in a coded character set. Thus, the upshifted or downshifted value of a character can be determined by accessing the relevant character entry in the shift table.

Language information (or langinfo) contains message text specific to a particular localization. The library function nl_langinfo() provides a procedural interface to this data, allowing applications to discover cultural and language-specific information at run-time. Individual items of langinfo data are identified by constants in Volume 2, XSI System Interfaces and Headers, <langinfo.h>.

Information specific to a culture or language includes the following:

A few utilities distributed with IRIX, in particular grep (see the grep(1) reference page) support internationalized regular expressions, which provide additional syntax for matching character classes, sequences, or ranges. The internationalized regular expressions supported by the wsregexp library are as shown in Table 16-7.

Within square brackets, a period (.) that is not part of a [[.c.]] sequence, a colon (:) that is not part of a [[:class:]] sequence, and an equals sign (=) that is not part of a [[=c=]] sequence matches itself.

Table 16-8 shows examples of simple regular expressions.

The list below identifies the minimum set of utilities that provide 8-bit transparency on all X/Open compliant systems. The definitions of these commands, in terms of their syntax and parameters, are not changed by the operation of NLS.

The cc, yacc, and lex commands provide 8-bit transparency for characters contained in character strings, character constants, and comment strings. An 8-bit character string enables a programmer to define default messages in languages other than English. The support of 8-bit characters in identifier names is implementation defined.

The 8-bit operation of commands that communicate with other systems cannot be guaranteed in all circumstances. For example, intersystem mail may be restricted to 7-bit data by the underlying network, 8-bit data and filenames may not be portable to noninternationalized systems, and so forth. Under these circumstances, it is recommended that you use only characters defined in the ASCII 7-bit range of characters for data transfer between machines, and you use only characters defined in the Portable Filename Character Set for naming remote files.

The list below shows library functions usable by internationalized application programs

Also, all functions defined in the X/Open Portability Guide, Volume 2, XSI System Interfaces and Headers, and X/Open Portability Guide, Volume 3, XSI Curses Interface, provide 8-bit transparency on X/Open compliant systems.

catopen() locates and opens a message catalog file:

#include <nl_types.h>
nl_catd catopen(char *name, int unused);

The argument name is used to locate the catalog. Usually, this is a simple, relative pathname that is combined with environment variables to indicate the path to the catalog (see “XPG/4 Catalog Location” for details). However, the catalog assumes names that begin with “/ ” are absolute pathnames. Use of a hard-coded pathname like this is strongly discouraged; it doesn't allow the user to specify the catalog's locale through environment variables.

When an application is finished using a message catalog, it should close the catalog and free the descriptor using catclose():

int catclose(nl_catd);

Catalogs contain sets of numbered messages. The application developer must know the contents of the catalog in order to specify the set and number of a message to be obtained.

catgets() is used to retrieve strings from a message catalog (see the catopen(3) and catgets(3) reference pages). Example 16-3 shows a program that reads the first message from the first message set in the appropriate catalog, and displays the result.

#include <stdio.h>
#include <locale.h>
#include <nl_types.h>
 
#define SET1      1
#define WRLD_MSG  1
 
int main(){
    nl_catd msgd;
    char *message;
    setlocale(LC_ALL, ""); 
 
msgd = catopen("hw",0);
    message = catgets(msgd, SET1, WRLD_MSG,"Hello, world\n");
    printf(message);
    catclose(msgd);
}

The previous example uses printf() instead of puts() in order to make a point: the format string of printf() came from a catalog. Note the crucial difference between these two statements:

printf(catgets(msgd, set, num, defaultStr));
printf("%s", catgets(msgd, set, num, defaultStr));

In the first statement, the catalog provides the printf() formatting string, possibly containing conversion specifications and escape sequences. In the second statement, the string from the catalog is treated as data and not interpreted for conversion specifications. For further discussion of issues relating to this important distinction, see “Variably Ordered Referencing of printf() Arguments.”

XPG/4 message catalogs are located using the environment variable NLSPATH. The default NLSPATH is /usr/lib/locale/%L/LC_MESSAGES/%N, where %L is filled in by the LANG environment variable and %N is filled in by the name argument to catopen(). NLSPATH can specify multiple pathnames in ordered precedence, much like the PATH variable. The following is a sample NLSPATH assignment:

NLSPATH=/usr/lib/locale/%L/LC_MESSAGES/%N:/usr/local/lib/locale/%L/LC_MESSAGES/
%N:/usr/defaults/%N 

Message catalogs are of this general form (these forms are detailed in the gencat(1) reference page):

$set n comment 
a message-a\n
b message-b\n
c message-c\n
$quote "
d " message-d "
$this is a comment

Each message is identified by a message number and a set. Sets are often used to separate messages into more easily usable groups, such as error messages, help messages, directives, and so on. Alternatively, you could use a different set for each source file, containing all of that source file's messages.

$set n specifies the beginning of set n, where n is a set identifier in the range from 1 to NL_SETMAX. All messages following the $set statement belong to set n until either a $delset or another $set is reached. You can skip set numbers (for example, you can have a set 3 without having a set 2), but the set numbers that you use must be listed in ascending numerical order (and every set must have a number). Any string following the set identifier on the same line is considered a comment.

$delset n deletes the set n from a message catalog.

$quote c specifies a quote character, c, which can be used to surround message text so that trailing spaces or null (empty) messages are visible in a message source line. By default, there is no quote character and messages are separated by newlines. To continue a message onto a second line, add a backslash to the end of the first line:

$set 1
1 Hello, world.
2 here is a long \
string.\n
3 Hello again.
n message-text-n

Message #2 in set #1 is “here is a long string.\n”.

After creating the message catalog sources, you need to compile them into binary form using gencat, which has the following syntax:

gencat catfile msgfile [msgfile ...]

where catfile is the target message catalog and msgfile is the message source file (see the gencat(1) reference page). If an old catfile exists, gencat attempts to merge new entries with the old. gencat “resolves” set and message number conflicts with new information replacing the old.

The catfile then needs to be placed in a location where catopen() can find it; see the “XPG/4 Catalog Location”.

An MNLS catalog source file contains a list of strings separated by new lines. For an empty string, an empty line is used. Strings are referenced by line number in the original source file.

Applications access the catalog by line number, so it's very important not to change the line numbers of existing catalog entries. This means that, when you want to add a new string to an existing catalog source, you should always append it to the end of the file—if you put it in the middle of the file, then you change the line number for subsequent strings.

The following tools can help you compile MNLS message catalogs:

When a file of strings is ready to be compiled, simply run mkmsgs and put the results in the directory /usr/lib/locale/localename/LC_MESSAGES.

One difference between MNLS and XPG/4 catalog functions is that the MNLS catalog can be used from commands, and hence it can be used to internationalize a shell script. The following table summarizes MNLS functions that have both a command line and a function library version:

MNLS message catalogs do not need to be specifically opened. The catalog of choice can be set explicitly once, or it can be specified every time a string is needed.

To specify the default message catalog to be used by subsequent calls to MNLS functions that reference catalogs, use setcat():

#include <pfmt.h>
char *setcat(const char *catalog);

catalog is limited to 14 characters, and may contain no character equal to zero or to the ASCII codes for slash (/) or colon (:). (See the setcat(3) reference page.)

setcat() doesn't check to see if the catalog name is valid; it just stores the string for future reference. For an example of use, see the following topic. The catalog indicated by the string must be found in the directory /usr/lib/locale/localename/LC_MESSAGES.

MNLS message catalogs do not need to be specifically opened. The catalog of choice can be set explicitly once, or it can be specified in each reference call. Strings are read from a catalog via gettxt() (see the gettxt(3) reference page):

#include <unistd.h>
char *gettxt(const char *msgid, const char *defaultStr);

msgid is a string containing two fields separated by a colon:

msgfilename:msgnumber

The msgfilename is a catalog name as described previously in the “Specifying MNLS Catalogs”. For example, to get message 10 from the MQ catalog, you could use either:

char *str = gettxt("MQ:10", "Hello, world.\n");

or

setcat("MQ");
str = gettxt(":10", "Hello, world.\n");

pfmt() is one of the most important routines dealing with MNLS catalogs, because it is used to produce most system diagnostic messages. pfmt() formats like printf() and produces standard error message formats (see the pfmt(3) reference page for the function, or pfmt(1) for shell use). It can usually be used in place of perror(). For example,

pfmt(stderr, MM_ERROR, "MQ:64:Permission denied");

would produce, by default (such as when the Mozambique locale is unavailable),

ERROR: Permission denied.

The syntax of pfmt() is

#include <pfmt.h>
int pfmt(FILE *stream, long flags, char *format, ... );

The flags are used to indicate severity, type, or control details to pfmt(). The format string includes information specifying which message from which catalog to look for. Flag details are discussed in the following section. The format is discussed in the “Format Strings for pfmt()”.

pfmt() flags are composed of several groups; specify no more than one from each group. Specify multiple flags by using OR. The groups are as follows:

pfmt() prints messages in the form label:severity:text. Severity is specified in the flags. The text comes from a message catalog (or a default) as specified in the format, and the label is specified earlier by the application.

In the example above, if no label has been set, we get only the output:

ERROR: Permission denied.

Typically, an application sets the label once early in its life; subsequent error messages have the label prepended. For example

setlabel("UX:myprog");
...
pfmt(stderr, MM_ERROR, "MQ:64:Permission denied");

would produce (by default)

UX:myprog: ERROR: Permission denied.

For details, consult the pfmt(3) and setlabel(3) reference pages.

pfmt() format strings are of this form:

[[catalog:]messagenum:]defaultstring 

The catalog field is in the format described in “Specifying MNLS Catalogs”. messagenum is the message number in the catalog to use as the format. defaultstring specifies the string to use if the catalog lookup fails for any reason.

An important feature of pfmt() is its ability to refer to format arguments in format-specified order just as printf() does. See “Variably Ordered Referencing of printf() Arguments” for details.

fmtmsg() is a comprehensive formatter using the MNLS catalogs and “standard” formats. You probably won't need to use it; most applications should get by with pfmt(), gettxt(), and printf(). Consult the fmtmsg(3) reference page for details.

You can internationalize the strings defined in the LEGEND and MENUCMD rules in the File Typing Rule (FTR) file. To internationalize these rules, precede the string with the following:

:[catalogname:]msgnumber: 

catalogname is optional and should be a valid MNLS catalog; msgnumber is the line number in catalogname. If you omit catalogname, the uxsgidesktop catalog is used by default.

You can use these rules to create your own FTR catalog. For example, an entry looks like this:

LEGEND :mycatalog:7:Archive 8mm Tape Drive 

This entry uses line 7 from the catalog, mycatalog, as the LEGEND for this FTR. If mycatalog is not available, or line 7 is not accessible from mycatalog, “Archive 8mm Tape Drive” is used as the LEGEND.

LEGEND :7:Archive 8mm Tape Drive 

This entry uses line 7 from the uxsgidesktop catalog, if available. Otherwise, “Archive 8mm Tape Drive” is used.

The next example,

MENUCMD \`mycatalog:9:Eject Tape\' /usr/sbin/eject /dev/tape

displays line 9 from mycatalog, if available. Otherwise “Eject Tape” is displayed on the menu that pops up when you click an icon that uses this FTR.

You can internationalize strings in the command part of MENUCMD and CMD rules by using gettxt or any other convenient policy detailed in this section. For example

CMD OPEN xconfirm -t "Tape tool not available"

can be internationalized to

CMD OPEN xconfirm -t "`gettxt mycatalog:376 'Tape tool not available'`"

In this example, gettxt is invoked to access line 376 from the catalog, mycatalog, and the string returned by gettxt is passed to xconfirm for display. If line 376 from mycatalog is not accessible, then gettxt returns the string “Tape tool not available.”

For more information about FTRs, see the IRIX Interactive Desktop Integration Guide.

There is no built-in support for vertical text. Applications may draw strings vertically only by laying out the text manually.

In previous releases of X, there was no general support for character sets other than Latin 1. X11R6, however, does allow other character sets.

X11R6 includes the definition of the X Portable Character Set, which is required to exist in all locales supported by Xlib. There is no encoding defined for this set; it is only a character set. The set—which is similar to printable ASCII plus the newline and tab—consists of these characters:

abcdefghijklmnoqrstuvwxyz
ABCDEFGHIJKLMNOPQRSTUVWXYZ
0123456789
!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~
<space> <tab> <newline>

The Host Portable Character Encoding is the encoding of the X Portable Character Set on the Xlib host. This encoding is part of X, and is thus independent of locale—the coding remains the same for all locales supported by the host.

Strings used or returned by Xlib routines are either in the Host Portable Character Encoding or a locale-specific encoding. The Xlib reference pages specify which encodings are used where. Some string constructs (such as TextProperty) contain information regarding their own encoding.

Full use of X11R6's internationalization features means calling some new routines supplied in the X11R6 Xlib. While all old Xlib applications work with the new Xlib, developers should change their code in places. These are described below.

If you're using Xt (with a widget set such as IRIS IM, Motif, or XaW), then don't use setlocale(). Instead, use

XtSetLanguageProc(NULL, NULL, NULL)

If you're using a toolkit other than Xt, call setlocale() as early as possible after execution begins.

Initialize Xlib's internationalization state after calling setlocale(). Xlib is being initialized, not a server or server-specific object, so a server connection is not necessary.

if ( setlocale(LC_ALL, "") == NULL )
    exit_with_error();
if ( ! XSupportsLocale() )
    exit_with_other_error();
if ( XSetLocaleModifiers("") == NULL)
    give_warning();

XSetLocaleModifiers() is required only for input. Just as passing an empty string to setlocale() honors the user's environment, so does passing an empty string to XSetLocaleModifiers().

To render strings encoded in EUC in Japanese, an application would need fonts encoded in 8859-1, JIS X 208, and JIS X 201. The application doesn't need to know which characters in a string go with which font, since it doesn't deal with locale specifics. So it creates a fontset that is made from a list of user-specified fonts (under the assumption that the localizer has provided an appropriate list). Rendering is then done using that fontset. The locale-aware rendering system chooses the appropriate fonts for each character being rendered, from the supplied list. You can find additional information about EUC in “Asian Languages.”

A fontset specification is just a string, enumerating XLFD names of fonts. (See X Logical Font Description Conventions, an MIT X Consortium standard, as well as “Font Names”.) This string can include wild card characters. For example, a specification of 16-point “fixed” fonts might be as follows:

char *fontSetSpecString = "*fixed-medium-r-normal*150*";

Based on the fonts available, a particular server might expand this to a string such as:

-jis-fixed-medium-r-normal--16-150-75-75-c-160-jisx0208.1983-0
-sony-fixed-medium-r-normal--16-150-75-75-c-80-iso8859-1
-sony-fixed-medium-r-normal--16-150-75-75-c-80-jisx0201.1976-0

Specifying the fontset by simply enumerating the fonts is perfectly acceptable:

char *fontSetSpecString =
"-jis-fixed-medium-r-normal*150-75-75*jisx0208.1983-0,\
-sony-fixed-medium-r-normal*150-75-75*iso8859-1,\
-sony-fixed-medium-r-normal*150-75-75*jisx0201.1976-0";

A German locale would work with only the ISO font; a Japanese locale might use all three; a Chinese locale would have trouble with this fontset.

The developer should specify a default fontset suitable for the default locale. Furthermore, developers should ensure that the application accepts localized fontset specifications via resources (or message catalogs) or command line options. Localizers are responsible for providing default fontset specifications suitable for their locales.

Creating fontsets in X is simply a matter of providing a string that names the fonts, as described above.

XFontSet fontset;
char *base_name;  /* should get from resource */
char **missingCharsetList;
int missingCharsetCount;
char *defaultStringForMissingCharsets; 
base_name = "*fixed-medium-r*150*"; /* use resources! */ 
fontset = XCreateFontSet(display, base_name,
                       &missingCharsetList,
                       &missingCharsetCount,
                       &defaultStringForMissingCharsets);

The locale in effect at create time is bound to the fontset. Fontsets are freed with XFreeFontSet().

Fontsets are used when rendering text with X11R6 Xmb or Xwc text rendering routines. These routines are described in “Text Rendering Routines.”

A sample program demonstrating some of the concepts in this section is given in Chapter 11 of the Xlib Programming Manual, Volume One. Looking carefully at that code may be easier than starting from scratch.

The old GL function qdevice() has a hard-coded view of a keyboard (see /usr/include/gl/device.h for details). Some flexibility, particularly for Europe, is available if you queue KEYBD instead of individual keys, but the GL has no general solution to non-ASCII input. There is no supported way to input Chinese (for instance) to the old GL.

OpenGL does not contain input code but leaves that to the operating environment, which in IRIX means X.

In short, support for internationalized input means a departure from qread(). Under IRIX, that means using mixed-model input, all the more reason to use a toolkit.

When a client connects to the X server, the server announces its range of keycodes and exports a table of keysyms. Each key event the client receives has a single byte keycode, which directly represents a physical key, and a single byte state, which represents currently engaged modifier keys, such as Shift or Alt.

---

Note: The mapping of state bits to modifiers is done by another table acquired from the server.

---

Keysyms are well defined, and there has been an attempt to have a keysym for every engraving one might possibly find on any keyboard, anywhere. (An engraving is the image imprinted on a physical key.) These are contained in /usr/include/X11/keysymdef.h. Keysyms represent the engravings on the actual keys, but not their meanings. The server's idea of the keysym table can be changed by clients, and clients may receive KeyMap events when this remapping happens, but such events don't happen often.

When a client receives a Key event, it asks Xlib to use the keycode to index into its keysym table to find a list of keysyms. (This list is usually very short. Most keys have only one or two engravings on them.) Using the state byte, Xlib chooses a keysym from the list to find out what was engraved on the key the user pressed.

At this point, the client can choose to act on the keysym itself (if, for instance, it was a backspace) or it can ask for a character string represented by the keysym (or both). Generating such a string is tricky; it is discussed in “Input Methods (IMs),” below.

Details on X keyboard support can be found in X Window System, Third Edition, from Digital Press. Details on input methods are also available in that book, as well as in the Xlib Programming Manual, Volume One.

There are two ways to compose characters that do not exist on a keyboard: explicit and implicit. It is common for an application to be modal and switch between the two. For example, Japanese input of kana is often done via implicit composition.

Users switch between a mode where input is interpreted as romaji (Latin characters) and a mode where input is translated to kana.

Furthermore, both styles may operate simultaneously. While an application is supporting implicit composition of certain characters, other characters may be composable via explicit composition.

Not every keystroke produces a character, even if the associated keysym normally implies character text. The event-to-string translation routines figure out what result a given set of keystrokes should produce (see “Using XLookupString(), XwcLookupString(), and XmbLookupString()” in this section).

Character composition from the user's aspect is discussed in the compose(5) and composetable(5) reference pages.

Explicit Composition		Explicit composition is requested when the user presses the Compose key and then types a key sequence that corresponds to the desired character. For example, to compose the character ñ under some keymaps, you might press the Compose key and then type ~n. Note: The xmodmap(1) reference page tells how to map the XK_Multi_key keysym onto whatever key you want to use as Compose.
Implicit Composition		Implicit composition mimics many existing European typewriters that have “dead” keys: keys that type a character but do not advance the carriage. When a special “dead” key is struck, the system attempts to compose a character using the next character struck. For example, on a keyboard that had a diaeresis (¨) and an O, but no Ö, you would strike ¨ and then O to compose Ö. Implicit composition support usually comes with some specified way to leave characters uncomposed.

IRIX currently supports 16 keyboard layouts: American, Belgian, Czech, Danish, English, French, German, Italian, Norwegian, Polish, Portuguese, Russian, Spanish, Swedish, Swiss and Turkish. The American keyboard needs only ASCII.

XOpenIM() opens an input method appropriate for the locale and modifiers in effect when it is called (see the XOpenIM(3X11) reference page). The locale is bound to that IM and cannot be changed. (But you could open another IM if you wanted to switch later.) Strings returned by XmbLookupString() and XwcLookupString() are encoded in the locale that was current when the IM was opened, regardless of current input context.

The syntax is

XIM XOpenIM(Display *dpy, XrmDataBase db, char *res_name,
            char *res_class);

The res_name is the resource name of the application, res_class is the resource class, and db is the resource database that the input method should use for looking up resources private to itself. Any of these can be NULL. The fragment in Example 16-7 shows how easy it is to open an input method.

XIM im;
im = XOpenIM(dpy, NULL, NULL, NULL);
if (im == NULL)
    exit_with_error();

XOpenIM() finds the IM appropriate for the current locale. If XSupportsLocale() has returned good status (see “Initialization for Xlib Programming”) and XOpenIM() fails, something is amiss with the administration of the system.

XSetLocaleModifiers() determines configure locale modifiers. The local host X locale modifiers announcer (the XMODIFIERS environment variable) is appended to the modifier list to provide default values on the locale host. The modifier list argument is a null-terminated string containing zero or more concatenated expressions of this form:

@category=value

For example, if you want to connect Input Method Server xwnmo, set modifiers _XWNMO as follows:

XSetLocaleModifiers("@im=_XWNMO");

Or, set environment variable XMODIFIERS to the string @im=_XWNMO and execute

XSetLocaleModifiers("");

---

Note: The library routines are not prepared for the possibility of XSupportsLocale() succeeding and XOpenIM() failing, so it's up to application developers to deal with such an eventuality. (This circumstance could occur, for example, if the IM died after XSupportsLocale() was called.) This topic is under some debate in the MIT X consortium. If XSetLocaleModifiers() is wrong, XOpenIM() will fail.

---

Most of the complexity associated with IM use comes from configuring an input context to work with the IM. Input contexts are discussed in “Input Contexts (ICs)”.

To close an input method, call XCloseIM().

The Root Window style has a pre-edit area and a status area in a window owned by the IM as a descendant of the root. The application does not manage the pre-edit data, the pre-edit area, the status data, or the status area. Everything is left to the input method to do in its own window, as illustrated in Figure 16-1.

Figure 16-1. Root Window Input

The Off-the-Spot style places a pre-edit area and a status area in the window being used, usually in reserved space away from the place where input appears. The application manages the pre-edit area and status area, but allows the IM to update the data there. (The application provides information regarding foreground and background colors, fonts, and so on.) A window using Off-the-Spot input style might look like that shown in Figure 16-2.

Figure 16-2. Off-the-Spot Input

The Over-the-Spot style involves the IM creating a small, pre-edit window over the point of insertion. The window is owned and managed by the IM as a descendant of the root, but it gives the user the impression that input is being entered in the right place; in fact, the pre-edit window often has no borders and is invisible to the user, giving the appearance of On-the-Spot input. The application manages the status area as in Off-the-Spot, but specifies the location of the editing so that the IM can place pre-edit data over that spot.

On-the-Spot input is by far the most complex for the application developer. The IM delivers all pre-edit data via callbacks to the application, which must perform in-place editing—complete with insertion and deletion and so on. This approach usually involves a great deal of string and text rendering support at the input generation level, above and beyond the effort required for completed input. Since this may mean a lot of updating of surrounding data or other display management, everything is left to the application. There is little chance an IM could ever know enough about the application to be able to help it provide user feedback. The IM therefore provides status and edit information via callbacks.

Done well, this style can be the most intuitive one for a user.

A style describes how an IM presents its pre-edit and status information to the user. An IM supplies information detailing its presentation capabilities. The information comes in the form of flags combined with OR. The flags to use with each style are as follows:

For example, if you wanted a style variable to match an Over-the-Spot IM style, you could write:

XIMStyle over = XIMPreeditPosition | XIMStatusArea;

If an IM returns XIMStatusNone (not to be confused with XIMStatusNothing), it means the IM will not supply status information.

An input method supports one or more styles. It's up to the application to find a style that is supported by both the IM and the application. If several exist, the application must choose. If none exist, the application is in trouble.

The IM may be able to support multiple styles—for example, both Off-the-Spot and Root Window. The application may be able to do, in order of preference, Over-the-Spot, Off-the-Spot, and Root Window. The application should determine that the best match in this case is Off-the-Spot.

First, discover what the IM can do, then set up a variable describing what the application can do, as shown in Example 16-8.

XIMStyles *IMcando;
XIMStyle  clientCanDo; /* note type difference */
XIMStyle  styleWeWillUse = NULL;
XGetImValues(im, XNQueryInputStyle, &IMcando, NULL);
clientCanDo =
/*none*/ XIMPreeditNone | XIMStatusNone |
/*over*/ XIMPreeditPosition | XIMStatusArea |
/*off*/  XIMPreeditArea | XIMStatusArea |
/*root*/ XIMPreeditNothing | XIMStatusNothing;

A client should always be able to handle the case of XIMPreeditNone | XIMStatusNone, which is likely in a Western locale. To the application, this is not very different from a RootWindow style, but it comes with less overhead.

Once you know what the application can handle, look through the IM styles for a match, as shown in Example 16-9.

for(i=0; i < IMcando->count_styles; i++) {
    XIMStyle tmpStyle;
    tmpStyle = IMcando->support_styles[i];
    if ( ((tmpStyle & clientCanDo) == tmpStyle) )
            styleWeWillUse = tmpStyle;
}
if (styleWeWillUse = NULL)
    exit_with_error();
XFree(IMcando);
/* styleWeWillUse is set, which is what we were after */

There are several pieces of information an input method may require, depending on the input context and style chosen by the application. The input method can acquire any such information it needs from the input context, ignoring any information that does not affect the style or IM.

A full description of every item of information available to the IM is supplied in X Window System, Third Edition. The following is a brief list:

When an IM is going to provide state, it needs some simple X information with which to do its work. For example, if an IM is going to draw status information in a client window in an Off-the-Spot style, it needs to know where the area is, what color and font to render text in, and so on. The application gives this data to the IC for use by the IM.

As with the “IC Values” section, full details are available in X Window System, Third Edition.

Creating an input context is a simple matter of calling XCreateIC() with a variable-length list of parameters specifying IC values. Example 16-10 shows a simple example that works for the root window.

XVaNestedList arglist;
XIC ic;
arglist = XVaCreateNestedList(0, XNFontSet, fontset,
                           XNForeground,
                           WhitePixel(dpy, screen),
                           XNBackground,
                           BlackPixel(dpy, screen),
                           NULL);
ic = XCreateIC(im, XNInputStyle, styleWeWillUse,
              XNClientWindow, window, XNFocusWindow, window,
              XNStatusAttributes, arglist,
              XNPreeditAttributes, arglist, NULL);
XFree(arglist);
if (ic == NULL)
    exit_with_error();

A multi-window application may choose to use several input contexts. But for simplicity, assume that the application just wants to get to the internationalized input using one method in one window.

Using the IC is a matter of making sure you check events the IC wants, and of setting IC focus. If you are setting up a window for the first time, you know the event mask you want, and you can use it directly. If you are attaching an IC to a previously configured window, you should query the window and add in the new event mask.

unsigned long imEventMask;
XGetWindowAttributes(dpy, win, &winAtts);
XGetICValues(ic, XNFilterEvents, &imEventMask, NULL);
imEventMask |= winAtts.your_event_mask;
XSelectInput(dpy, window, imEventMask);
XSetICFocus(ic);

At this point, the window is ready to be used.

Every event received by your application should be fed to the IM via XFilterEvent(), which returns a value telling you whether or not to disregard the event. IMs asks you to disregard the event if they have extracted the data and plan on giving it to you later, possibly in some other form. All events (not just KeyPress and KeyRelease events) go to XFilterEvent().

If you compacted the event processing into a single routine, a typical event loop would look something like the code in Example 16-12.

Xevent event;
while (TRUE) {
    XNextEvent(dpy, &event);
    if (XFilterEvent(&event, None))
        continue;
    DealWithEvent(&event);
}

When using an input method, you should replace calls to XLookupString() with calls to XwcLookupString() or XmbLookupString(). The MB and WC versions have very similar interfaces. The examples below arbitrarily use XmbLookupString(), but apply to both versions.

There are two new situations to deal with:

Dealing with the former is a matter of maintaining an arena, as in Example 16-13.

To tell the application what to pay attention to for a given event, XmbLookupString() returns a status value in a passed parameter, equal to one of the following:

XmbLookupString() also returns the length of the string in question. Note that XmbLookupString() returns the length of the string in bytes, while XwcLookupString() returns the length of the string in characters.

The example below should help show how these functions work. Most event processors perform a switch on the event type; assume you have done that and have received a KeyPress event.

case KeyPress:
{
    Keysym keysym;
    Status status;
    int buflength;
    static int bufsize = 16;
    static char *buf = NULL;
    if (buf == NULL) {
        buf = malloc(bufsize);
        if (buf < 0) StopSequence();
    }
    buflength = XmbLookupString(ic, &event, buf, bufsize,
                             &keysym, &status);
    /* first, check to see if that worked */
    if (status == XBufferOverflow) {
        buf = realloc(buf, (bufsize = buflength));
        buflength = XmbLookupString(ic, &event, buf, bufsize,
                                 &keysym, &status);
    }
    /* We have a valid status. Check that */
    switch(status) {
    case XLookupKeysym:
        DealWithKeysym(keysym);
        break;
    case XLookupBoth:
        DealWithKeysym(keysym);
        /* **FALL INTO** charcter case */
    case XLookupChars:
        DealWithString(buf, buflength);
    case XLookupNone:
        break;
    } /* end switch(status) */
} /* end case KeyPress segment */
break; /* we are in a switch(event.type) statement */ 

Most toolkits provide form, pane, rowcolumn, or other layout objects that calculate layout depending on the “natural” (localized) size of the objects involved. Most use some hints provided by the developer that can regulate this layout. For example, some IRIS IM widgets providing these services are XmForm, XmPanedWindow, and XmRowColumn.

Dynamic layout is probably the simplest way to prevent localization difficulties.

---

Note: The IRIS IM product is the Silicon Graphics port of the OSF/Motif product, and should not be confused with IM, the abbreviation for Input Methods.

---

Under certain circumstances, an application may insist on having a predefined layout. When this is so, the application must provide objects that are constructed to allow localization. A “Quit” button that just barely allows room for the Latin 1 string “Quit” is not likely to suffice when localizers attempt to fit their translations into that small space.

In order to enforce constant layout, the developer incurs the heavy responsibility of making sure the objects are localizable. This means a lot of investigation; the “there, that ought to be enough” approach is chancy at best.

Some toolkits provide for layout control by run-time reading of strings or other data files. Applications that use such toolkits can easily finesse the layout issue by providing the capability for localization of the layout, as well as localization of the contents of the layout. This provides each localizer maximum freedom in presenting the application to the local users. The application developer is responsible for providing localizers with instructions and the mechanisms necessary to produce layout data.

IRIX provides an interactive method of laying out widgets for IRIS IM and Xaw (the Athena Widget Set): a utility called editres. With editres, you can construct and edit resources and see how your widgets will look on the screen; the program even generates a usable app-defaults file for you. But note that if you hard-code any resources into your IRIS IM code, you won't be able to edit them using this method.

Various Asian character sets have been developed, some of which are considered standard. Encodings for these sets are less standardized. Asian character sets usually require larger-than-byte character types like those described in “Multibyte Characters.” Table 16-11 lists some of these standard character sets. Note that some of these character sets have multiple associated codesets, usually designated by appending the year the codeset was adopted to the character set name. (For example, JIS X 208-1983 is different from JIS X 208-1990.)

EUC is Extended UNIX Code, an encoding methodology that supports concurrent use of four codesets in one encoding. It employs two special “shift state” bytes:

ss1 = 0x8e
ss2 = 0x8f

These are used to identify codesets within a string. The EUC encoding scheme uses the following patterns to indicate which codeset is in use at any given time:

Codeset #0: 0xxxxxxx
Codeset #1: 1xxxxxxx [ 1xxxxxxx ...]
Codeset #2: ss1 1xxxxxxx [ 1xxxxxxx ...]
Codeset #3: ss2 1xxxxxxx [ 1xxxxxxx ...]

So if ss1 appears in a string, it means that the next character—however many bytes long it is—should be interpreted as a character from codeset #2. If there are multiple characters in a row from codeset #2, each one is preceded by ss1. Similarly, ss2 indicates that the following character belongs to codeset #3. If any other byte whose high bit is 1 appears in the string (without being preceded by ss1 or ss2), it is interpreted as all or part of a character from codeset #1.

In EUC, codeset #1 is always ASCII. The other codesets are implementation- or user-defined. This is why EUC cannot support Latin 1 in Asian locales.

EUC implementations exist (but are not standardized) for all ideographic Asian languages.