| 1 |
.\" Copyright 1995-2001, Markus Kuhn <[email protected]> |
| 2 |
.\" Copyright, the authors of the Linux man-pages project |
| 3 |
.\" |
| 4 |
.\" SPDX-License-Identifier: GPL-2.0-or-later |
| 5 |
.\" |
| 6 |
.TH unicode 7 2026-02-11 "Linux man-pages 6.18" |
| 7 |
.SH NAME |
| 8 |
unicode \- universal character set |
| 9 |
.SH DESCRIPTION |
| 10 |
The international standard ISO/IEC 10646 defines the |
| 11 |
Universal Character Set (UCS). |
| 12 |
UCS contains all characters of all other character set standards. |
| 13 |
It also guarantees "round-trip compatibility"; |
| 14 |
in other words, |
| 15 |
conversion tables can be built such that no information is lost |
| 16 |
when a string is converted from any other encoding to UCS and back. |
| 17 |
.P |
| 18 |
UCS contains the characters required to represent practically all |
| 19 |
known languages. |
| 20 |
This includes not only the Latin, Greek, Cyrillic, |
| 21 |
Hebrew, Arabic, Armenian, and Georgian scripts, but also Chinese, |
| 22 |
Japanese and Korean Han ideographs as well as scripts such as |
| 23 |
Hiragana, Katakana, Hangul, Devanagari, Bengali, Gurmukhi, Gujarati, |
| 24 |
Oriya, Tamil, Telugu, Kannada, Malayalam, Thai, Lao, Khmer, Bopomofo, |
| 25 |
Tibetan, Runic, Ethiopic, Canadian Syllabics, Cherokee, Mongolian, |
| 26 |
Ogham, Myanmar, Sinhala, Thaana, Yi, and others. |
| 27 |
For scripts not yet |
| 28 |
covered, research on how to best encode them for computer usage is |
| 29 |
still going on and they will be added eventually. |
| 30 |
This might |
| 31 |
eventually include not only Hieroglyphs and various historic |
| 32 |
Indo-European languages, but even some selected artistic scripts such |
| 33 |
as Tengwar, Cirth, and Klingon. |
| 34 |
UCS also covers a large number of |
| 35 |
graphical, typographical, mathematical, and scientific symbols, |
| 36 |
including those provided by TeX, Postscript, APL, MS-DOS, MS-Windows, |
| 37 |
Macintosh, OCR fonts, as well as many word processing and publishing |
| 38 |
systems, and more are being added. |
| 39 |
.P |
| 40 |
The UCS standard (ISO/IEC 10646) describes a |
| 41 |
31-bit character set architecture |
| 42 |
consisting of 128 24-bit |
| 43 |
.IR groups , |
| 44 |
each divided into 256 16-bit |
| 45 |
.I planes |
| 46 |
made up of 256 8-bit |
| 47 |
.I rows |
| 48 |
with 256 |
| 49 |
.I column |
| 50 |
positions, one for each character. |
| 51 |
Part 1 of the standard (ISO/IEC 10646-1) |
| 52 |
defines the first 65534 code positions (0x0000 to 0xfffd), which form |
| 53 |
the |
| 54 |
.I Basic Multilingual Plane |
| 55 |
(BMP), that is plane 0 in group 0. |
| 56 |
Part 2 of the standard (ISO/IEC 10646-2) |
| 57 |
adds characters to group 0 outside the BMP in several |
| 58 |
.I "supplementary planes" |
| 59 |
in the range 0x10000 to 0x10ffff. |
| 60 |
There are no plans to add characters |
| 61 |
beyond 0x10ffff to the standard, therefore of the entire code space, |
| 62 |
only a small fraction of group 0 will ever be actually used in the |
| 63 |
foreseeable future. |
| 64 |
The BMP contains all characters found in the |
| 65 |
commonly used other character sets. |
| 66 |
The supplemental planes added by |
| 67 |
ISO/IEC 10646-2 cover only more exotic characters for special scientific, |
| 68 |
dictionary printing, publishing industry, higher-level protocol and |
| 69 |
enthusiast needs. |
| 70 |
.P |
| 71 |
The representation of each UCS character as a 2-byte word is referred |
| 72 |
to as the UCS-2 form (only for BMP characters), |
| 73 |
whereas UCS-4 is the representation of each character by a 4-byte word. |
| 74 |
In addition, there exist two encoding forms UTF-8 |
| 75 |
for backward compatibility with ASCII processing software and UTF-16 |
| 76 |
for the backward-compatible handling of non-BMP characters up to |
| 77 |
0x10ffff by UCS-2 software. |
| 78 |
.P |
| 79 |
The UCS characters 0x0000 to 0x007f are identical to those of the |
| 80 |
classic US-ASCII |
| 81 |
character set and the characters in the range 0x0000 to 0x00ff |
| 82 |
are identical to those in |
| 83 |
ISO/IEC\~8859-1 (Latin-1). |
| 84 |
.SS Combining characters |
| 85 |
Some code points in UCS |
| 86 |
have been assigned to |
| 87 |
.IR "combining characters" . |
| 88 |
These are similar to the nonspacing accent keys on a typewriter. |
| 89 |
A combining character just adds an accent to the previous character. |
| 90 |
The most important accented characters have codes of their own in UCS; |
| 91 |
however, the combining character mechanism allows us to add accents |
| 92 |
and other diacritical marks to any character. |
| 93 |
The combining characters |
| 94 |
always follow the character which they modify. |
| 95 |
For example, the German |
| 96 |
character Umlaut-A ("Latin capital letter A with diaeresis") can |
| 97 |
either be represented by the precomposed UCS code 0x00c4, or |
| 98 |
alternatively as the combination of a normal "Latin capital letter A" |
| 99 |
followed by a "combining diaeresis": 0x0041 0x0308. |
| 100 |
.P |
| 101 |
Combining characters are essential for instance for encoding the Thai |
| 102 |
script or for mathematical typesetting and users of the International |
| 103 |
Phonetic Alphabet. |
| 104 |
.SS Implementation levels |
| 105 |
As not all systems are expected to support advanced mechanisms like |
| 106 |
combining characters, ISO/IEC 10646-1 specifies the following three |
| 107 |
.I implementation levels |
| 108 |
of UCS: |
| 109 |
.TP 0.9i |
| 110 |
Level 1 |
| 111 |
Combining characters and Hangul Jamo |
| 112 |
(a variant encoding of the Korean script, where a Hangul syllable |
| 113 |
glyph is coded as a triplet or pair of vowel/consonant codes) are not |
| 114 |
supported. |
| 115 |
.TP |
| 116 |
Level 2 |
| 117 |
In addition to level 1, combining characters are now allowed for some |
| 118 |
languages where they are essential (e.g., Thai, Lao, Hebrew, |
| 119 |
Arabic, Devanagari, Malayalam). |
| 120 |
.TP |
| 121 |
Level 3 |
| 122 |
All UCS characters are supported. |
| 123 |
.P |
| 124 |
The Unicode 3.0 Standard |
| 125 |
published by the Unicode Consortium |
| 126 |
contains exactly the UCS Basic Multilingual Plane |
| 127 |
at implementation level 3, as described in ISO/IEC 10646-1:2000. |
| 128 |
Unicode 3.1 added the supplemental planes of ISO/IEC 10646-2. |
| 129 |
The Unicode standard and |
| 130 |
technical reports published by the Unicode Consortium provide much |
| 131 |
additional information on the semantics and recommended usages of |
| 132 |
various characters. |
| 133 |
They provide guidelines and algorithms for |
| 134 |
editing, sorting, comparing, normalizing, converting, and displaying |
| 135 |
Unicode strings. |
| 136 |
.SS Unicode under Linux |
| 137 |
Under GNU+Linux, the C type |
| 138 |
.I wchar_t |
| 139 |
is a signed 32-bit integer type. |
| 140 |
Its values are always interpreted |
| 141 |
by the C library as UCS |
| 142 |
code values (in all locales), a convention that is signaled by the GNU |
| 143 |
C library to applications by defining the constant |
| 144 |
.B __STDC_ISO_10646__ |
| 145 |
as specified in the ISO C99 standard. |
| 146 |
.P |
| 147 |
UCS/Unicode can be used just like ASCII in input/output streams, |
| 148 |
terminal communication, plaintext files, filenames, and environment |
| 149 |
variables in the ASCII compatible UTF-8 multibyte encoding. |
| 150 |
To signal the use of UTF-8 as the character |
| 151 |
encoding to all applications, a suitable |
| 152 |
.I locale |
| 153 |
has to be selected via environment variables (e.g., |
| 154 |
"LANG=en_GB.UTF-8"). |
| 155 |
.P |
| 156 |
The |
| 157 |
.B nl_langinfo(CODESET) |
| 158 |
function returns the name of the selected encoding. |
| 159 |
Library functions such as |
| 160 |
.BR wctomb (3) |
| 161 |
and |
| 162 |
.BR mbsrtowcs (3) |
| 163 |
can be used to transform the internal |
| 164 |
.I wchar_t |
| 165 |
characters and strings into the system character encoding and back |
| 166 |
and |
| 167 |
.BR wcwidth (3) |
| 168 |
tells how many positions (0\[en]2) the cursor is advanced by the |
| 169 |
output of a character. |
| 170 |
.SS Private Use Areas (PUA) |
| 171 |
In the Basic Multilingual Plane, |
| 172 |
the range 0xe000 to 0xf8ff will never be assigned to any characters by |
| 173 |
the standard and is reserved for private usage. |
| 174 |
For the Linux |
| 175 |
community, this private area has been subdivided further into the |
| 176 |
range 0xe000 to 0xefff which can be used individually by any end-user |
| 177 |
and the Linux zone in the range 0xf000 to 0xf8ff where extensions are |
| 178 |
coordinated among all Linux users. |
| 179 |
The registry of the characters |
| 180 |
assigned to the Linux zone is maintained by LANANA and the registry |
| 181 |
itself is |
| 182 |
.I Documentation/admin\-guide/unicode.rst |
| 183 |
in the Linux kernel sources. |
| 184 |
.P |
| 185 |
Two other planes are reserved for private usage, plane 15 |
| 186 |
(Supplementary Private Use Area-A, range 0xf0000 to 0xffffd) |
| 187 |
and plane 16 (Supplementary Private Use Area-B, range |
| 188 |
0x100000 to 0x10fffd). |
| 189 |
.SS Literature |
| 190 |
.IP \[bu] 3 |
| 191 |
Information technology \[em] Universal Multiple-Octet Coded Character |
| 192 |
Set (UCS) \[em] Part 1: Architecture and Basic Multilingual Plane. |
| 193 |
International Standard ISO/IEC 10646-1, International Organization |
| 194 |
for Standardization, Geneva, 2000. |
| 195 |
.IP |
| 196 |
This is the official specification of UCS. |
| 197 |
Available from |
| 198 |
.UR http:\://www.iso.ch/ |
| 199 |
.UE . |
| 200 |
.IP \[bu] |
| 201 |
The Unicode Standard, Version 3.0. |
| 202 |
The Unicode Consortium, Addison-Wesley, |
| 203 |
Reading, MA, 2000, ISBN 0-201-61633-5. |
| 204 |
.IP \[bu] |
| 205 |
S.\& Harbison, G.\& Steele. |
| 206 |
C: A Reference Manual. |
| 207 |
Fourth edition, |
| 208 |
Prentice Hall, Englewood Cliffs, 1995, ISBN 0-13-326224-3. |
| 209 |
.IP |
| 210 |
A good reference book about the C programming language. |
| 211 |
The fourth |
| 212 |
edition covers the 1994 Amendment 1 to the ISO C90 standard, which |
| 213 |
adds a large number of new C library functions for handling wide and |
| 214 |
multibyte character encodings, but it does not yet cover ISO C99, |
| 215 |
which improved wide and multibyte character support even further. |
| 216 |
.IP \[bu] |
| 217 |
Unicode Technical Reports. |
| 218 |
.RS |
| 219 |
.UR http:\://www.unicode.org\:/reports/ |
| 220 |
.UE |
| 221 |
.RE |
| 222 |
.IP \[bu] |
| 223 |
Markus Kuhn: UTF-8 and Unicode FAQ for UNIX/Linux. |
| 224 |
.RS |
| 225 |
.UR http:\://www.cl.cam.ac.uk\:/\[ti]mgk25\:/unicode.html |
| 226 |
.UE |
| 227 |
.RE |
| 228 |
.IP \[bu] |
| 229 |
Bruno Haible: Unicode HOWTO. |
| 230 |
.RS |
| 231 |
.UR http:\://www.tldp.org\:/HOWTO\:/Unicode\-HOWTO.html |
| 232 |
.UE |
| 233 |
.RE |
| 234 |
.\" .SH AUTHOR |
| 235 |
.\" Markus Kuhn <[email protected]> |
| 236 |
.SH SEE ALSO |
| 237 |
.BR locale (1), |
| 238 |
.BR setlocale (3), |
| 239 |
.BR charsets (7), |
| 240 |
.BR utf\-8 (7) |