| 1 |
.\" Copyright 1996, Eric S. Raymond <[email protected]> |
| 2 |
.\" Copyright, Andries E. Brouwer <[email protected]> |
| 3 |
.\" Copyright, the authors of the Linux man-pages project |
| 4 |
.\" |
| 5 |
.\" SPDX-License-Identifier: GPL-2.0-or-later |
| 6 |
.\" |
| 7 |
.TH charsets 7 2026-02-08 "Linux man-pages 6.18" |
| 8 |
.SH NAME |
| 9 |
charsets \- character set standards and internationalization |
| 10 |
.SH DESCRIPTION |
| 11 |
This manual page gives an overview on different character set standards |
| 12 |
and how they were used on Linux before Unicode became ubiquitous. |
| 13 |
Some of this information is still helpful for people working with legacy |
| 14 |
systems and documents. |
| 15 |
.P |
| 16 |
Standards discussed include such as |
| 17 |
ASCII, GB 2312, ISO/IEC\~8859, JIS, KOI8-R, KS, and Unicode. |
| 18 |
.P |
| 19 |
The primary emphasis is on character sets that were actually used by |
| 20 |
locale character sets, not the myriad others that could be found in data |
| 21 |
from other systems. |
| 22 |
.SS ASCII |
| 23 |
ASCII (American Standard Code For Information Interchange) is the original |
| 24 |
7-bit character set, originally designed for American English. |
| 25 |
Also known as US-ASCII. |
| 26 |
It is currently described by the ISO/IEC\~646:1991 IRV |
| 27 |
(International Reference Version) standard. |
| 28 |
.P |
| 29 |
Various ASCII variants replacing the dollar sign with other currency |
| 30 |
symbols and replacing punctuation with non-English alphabetic |
| 31 |
characters to cover German, French, Spanish, and others in 7 bits |
| 32 |
emerged. |
| 33 |
All are deprecated; |
| 34 |
glibc does not support locales whose character sets are not true |
| 35 |
supersets of ASCII. |
| 36 |
.P |
| 37 |
As Unicode, when using UTF-8, is ASCII-compatible, plain ASCII text |
| 38 |
still renders properly on modern UTF-8 using systems. |
| 39 |
.SS ISO/IEC\~8859 |
| 40 |
ISO/IEC\~8859 is a series of 15 8-bit character sets, all of which have ASCII |
| 41 |
in their low (7-bit) half, invisible control characters in positions |
| 42 |
128 to 159, and 96 fixed-width graphics in positions 160\[en]255. |
| 43 |
.P |
| 44 |
Of these, the most important is ISO/IEC\~8859-1 |
| 45 |
("Latin Alphabet No.\& 1" / Latin-1). |
| 46 |
It was widely adopted and supported by different systems, |
| 47 |
and is gradually being replaced with Unicode. |
| 48 |
The ISO/IEC\~8859-1 characters are also the first 256 characters of Unicode. |
| 49 |
.P |
| 50 |
Console support for the other ISO/IEC\~8859 character sets is available under |
| 51 |
Linux through user-mode utilities (such as |
| 52 |
.BR setfont (8)) |
| 53 |
that modify keyboard bindings and the EGA graphics |
| 54 |
table and employ the "user mapping" font table in the console |
| 55 |
driver. |
| 56 |
.P |
| 57 |
Here are brief descriptions of each character set: |
| 58 |
.TP |
| 59 |
ISO/IEC\~8859-1 (Latin-1) |
| 60 |
Latin-1 covers many European languages such as Albanian, Basque, |
| 61 |
Danish, English, Faroese, Galician, Icelandic, Irish, Italian, |
| 62 |
Norwegian, Portuguese, Spanish, and Swedish. |
| 63 |
The lack of the ligatures |
| 64 |
Dutch IJ/ij, |
| 65 |
French œ, |
| 66 |
and „German“ quotation marks |
| 67 |
was considered tolerable. |
| 68 |
.TP |
| 69 |
ISO/IEC\~8859-2 (Latin-2) |
| 70 |
Latin-2 supports many Latin-written Central and East European |
| 71 |
languages such as Bosnian, Croatian, Czech, German, Hungarian, Polish, |
| 72 |
Slovak, and Slovene. |
| 73 |
Replacing Romanian ș/ț with ş/ţ |
| 74 |
was considered tolerable. |
| 75 |
.TP |
| 76 |
ISO/IEC\~8859-3 (Latin-3) |
| 77 |
Latin-3 was designed to cover of Esperanto, Maltese, and Turkish, but |
| 78 |
ISO/IEC\~8859-9 later superseded it for Turkish. |
| 79 |
.TP |
| 80 |
ISO/IEC\~8859-4 (Latin-4) |
| 81 |
Latin-4 introduced letters for North European languages such as |
| 82 |
Estonian, Latvian, and Lithuanian, but was superseded by ISO/IEC\~8859-10 and |
| 83 |
ISO/IEC\~8859-13. |
| 84 |
.TP |
| 85 |
ISO/IEC\~8859-5 |
| 86 |
Cyrillic letters supporting Bulgarian, Byelorussian, Macedonian, |
| 87 |
Russian, Serbian, and (almost completely) Ukrainian. |
| 88 |
It was never widely used, see the discussion of KOI8-R/KOI8-U below. |
| 89 |
.TP |
| 90 |
ISO/IEC\~8859-6 |
| 91 |
Was created for Arabic. |
| 92 |
The ISO/IEC\~8859-6 glyph table is a fixed font of separate |
| 93 |
letter forms, but a proper display engine should combine these |
| 94 |
using the proper initial, medial, and final forms. |
| 95 |
.TP |
| 96 |
ISO/IEC\~8859-7 |
| 97 |
Was created for Modern Greek in 1987, |
| 98 |
updated in 2003. |
| 99 |
.TP |
| 100 |
ISO/IEC\~8859-8 |
| 101 |
Supports Modern Hebrew without niqud (punctuation signs). |
| 102 |
Niqud and full-fledged Biblical Hebrew were |
| 103 |
outside the scope of this character set. |
| 104 |
.TP |
| 105 |
ISO/IEC\~8859-9 (Latin-5) |
| 106 |
This is a variant of Latin-1 |
| 107 |
that replaces Icelandic letters with Turkish ones. |
| 108 |
.TP |
| 109 |
ISO/IEC\~8859-10 (Latin-6) |
| 110 |
Latin-6 added the Inuit (Greenlandic) and Sami (Lappish) letters |
| 111 |
that were missing in Latin-4 |
| 112 |
to cover the entire Nordic area. |
| 113 |
.TP |
| 114 |
ISO/IEC\~8859-11 |
| 115 |
Supports the Thai alphabet and is nearly identical to the TIS-620 standard. |
| 116 |
.TP |
| 117 |
ISO/IEC\~8859-12 |
| 118 |
This character set does not exist. |
| 119 |
.TP |
| 120 |
ISO/IEC\~8859-13 (Latin-7) |
| 121 |
Supports the Baltic Rim languages; |
| 122 |
in particular, |
| 123 |
it includes Latvian characters not found in Latin-4. |
| 124 |
.TP |
| 125 |
ISO/IEC\~8859-14 (Latin-8) |
| 126 |
This is the Celtic character set, covering Old Irish, Manx, Gaelic, |
| 127 |
Welsh, Cornish, and Breton. |
| 128 |
.TP |
| 129 |
ISO/IEC\~8859-15 (Latin-9) |
| 130 |
Latin-9 is similar to the widely used Latin-1 but replaces some less |
| 131 |
common symbols with the Euro sign and French and Finnish letters that |
| 132 |
were missing in Latin-1. |
| 133 |
.TP |
| 134 |
ISO/IEC\~8859-16 (Latin-10) |
| 135 |
This character set covers many Southeast European languages, |
| 136 |
and most importantly supports Romanian more completely than Latin-2. |
| 137 |
.SS KOI8-R / KOI8-U |
| 138 |
KOI8-R is a non-ISO character set popular in Russia before Unicode. |
| 139 |
The lower half is ASCII; |
| 140 |
the upper is a Cyrillic character set somewhat better designed than |
| 141 |
ISO/IEC\~8859-5. |
| 142 |
KOI8-U, based on KOI8-R, has better support for Ukrainian. |
| 143 |
Neither of these sets are ISO/IEC\~2022 compatible, |
| 144 |
unlike the ISO/IEC\~8859 series. |
| 145 |
.P |
| 146 |
Console support for KOI8-R is available under Linux through user-mode |
| 147 |
utilities that modify keyboard bindings and the EGA graphics table, |
| 148 |
and employ the "user mapping" font table in the console driver. |
| 149 |
.SS GB 2312 |
| 150 |
GB 2312 is a mainland Chinese national standard character set used |
| 151 |
to express simplified Chinese. |
| 152 |
Just like JIS X 0208, characters are |
| 153 |
mapped into a 94x94 two-byte matrix used to construct EUC-CN. |
| 154 |
EUC-CN |
| 155 |
is the most important encoding for Linux and includes ASCII and |
| 156 |
GB 2312. |
| 157 |
Note that EUC-CN is often called as GB, GB 2312, or CN-GB. |
| 158 |
.SS Big5 |
| 159 |
Big5 was a popular character set in Taiwan to express traditional |
| 160 |
Chinese. |
| 161 |
(Big5 is both a character set and an encoding.) |
| 162 |
It is a superset of ASCII. |
| 163 |
Non-ASCII characters are expressed in two bytes. |
| 164 |
Bytes 0xa1\[en]0xfe are used as leading bytes for two-byte characters. |
| 165 |
Big5 and its extension were widely used in Taiwan and Hong Kong. |
| 166 |
It is not ISO/IEC\~2022 compliant. |
| 167 |
.\" Thanks to Tomohiro KUBOTA for the following sections about |
| 168 |
.\" national standards. |
| 169 |
.SS JIS X 0208 |
| 170 |
JIS X 0208 is a Japanese national standard character set. |
| 171 |
Though there are some more Japanese national standard character sets (like |
| 172 |
JIS X 0201, JIS X 0212, and JIS X 0213), this is the most important one. |
| 173 |
Characters are mapped into a 94x94 two-byte matrix, |
| 174 |
whose each byte is in the range 0x21\[en]0x7e. |
| 175 |
Note that JIS X 0208 is a character set, not an encoding. |
| 176 |
This means that JIS X 0208 |
| 177 |
itself is not used for expressing text data. |
| 178 |
JIS X 0208 is used |
| 179 |
as a component to construct encodings such as EUC-JP, Shift_JIS, |
| 180 |
and ISO/IEC\~2022-JP. |
| 181 |
EUC-JP is the most important encoding for Linux |
| 182 |
and includes ASCII and JIS X 0208. |
| 183 |
In EUC-JP, JIS X 0208 |
| 184 |
characters are expressed in two bytes, each of which is the |
| 185 |
JIS X 0208 code plus 0x80. |
| 186 |
.SS KS X 1001 |
| 187 |
KS X 1001 is a Korean national standard character set. |
| 188 |
Just as |
| 189 |
JIS X 0208, characters are mapped into a 94x94 two-byte matrix. |
| 190 |
KS X 1001 is used like JIS X 0208, as a component |
| 191 |
to construct encodings such as EUC-KR, Johab, and ISO/IEC\~2022-KR. |
| 192 |
EUC-KR is the most important encoding for Linux and includes |
| 193 |
ASCII and KS X 1001. |
| 194 |
KS C 5601 is an older name for KS X 1001. |
| 195 |
.SS ISO/IEC\~2022 and ISO/IEC\~4873 |
| 196 |
The ISO/IEC\~2022 and ISO/IEC\~4873 standards describe a font-control model |
| 197 |
based on VT100 practice. |
| 198 |
This model is (partially) supported |
| 199 |
by the Linux kernel and by |
| 200 |
.BR xterm (1). |
| 201 |
Several ISO/IEC\~2022-based character encodings have been defined, |
| 202 |
especially for Japanese. |
| 203 |
.P |
| 204 |
There are 4 graphic character sets, called G0, G1, G2, and G3, |
| 205 |
and one of them is the current character set for codes with |
| 206 |
high bit zero (initially G0), and one of them is the current |
| 207 |
character set for codes with high bit one (initially G1). |
| 208 |
Each graphic character set has 94 or 96 characters, and is |
| 209 |
essentially a 7-bit character set. |
| 210 |
It uses codes either |
| 211 |
040\[en]0177 (041\[en]0176) or 0240\[en]0377 (0241\[en]0376). |
| 212 |
G0 always has size 94 and uses codes 041\[en]0176. |
| 213 |
.P |
| 214 |
Switching between character sets is done using the shift functions |
| 215 |
.B \[ha]N |
| 216 |
(SO or LS1), |
| 217 |
.B \[ha]O |
| 218 |
(SI or LS0), ESC n (LS2), ESC o (LS3), |
| 219 |
ESC N (SS2), ESC O (SS3), ESC \[ti] (LS1R), ESC } (LS2R), ESC | (LS3R). |
| 220 |
The function |
| 221 |
.RI LS n |
| 222 |
makes character set |
| 223 |
.RI G n |
| 224 |
the current one |
| 225 |
for codes with high bit zero. |
| 226 |
The function |
| 227 |
.RI LS n R |
| 228 |
makes character set |
| 229 |
.RI G n |
| 230 |
the current one |
| 231 |
for codes with high bit one. |
| 232 |
The function |
| 233 |
.RI SS n |
| 234 |
makes character set |
| 235 |
.RI G n |
| 236 |
.RI ( n =2 |
| 237 |
or 3) |
| 238 |
the current one for the next character only (regardless of the value |
| 239 |
of its high order bit). |
| 240 |
.P |
| 241 |
A 94-character set is designated as |
| 242 |
.RI G n |
| 243 |
character set |
| 244 |
by an escape sequence ESC ( xx (for G0), ESC ) xx (for G1), |
| 245 |
ESC * xx (for G2), ESC + xx (for G3), where xx is a symbol |
| 246 |
or a pair of symbols found in the ISO/IEC\~2375 International |
| 247 |
Register of Coded Character Sets. |
| 248 |
For example, ESC ( @ selects the ISO/IEC\~646 character set as G0, |
| 249 |
ESC ( A selects the UK standard character set (with pound |
| 250 |
instead of number sign), ESC ( B selects ASCII (with dollar |
| 251 |
instead of currency sign), ESC ( M selects a character set |
| 252 |
for African languages, ESC ( ! A selects the Cuban character |
| 253 |
set, and so on. |
| 254 |
.P |
| 255 |
A 96-character set is designated as |
| 256 |
.RI G n |
| 257 |
character set |
| 258 |
by an escape sequence ESC \- xx (for G1), ESC .\& xx (for G2) |
| 259 |
or ESC / xx (for G3). |
| 260 |
For example, ESC \- G selects the Hebrew alphabet as G1. |
| 261 |
.P |
| 262 |
A multibyte character set is designated as |
| 263 |
.RI G n |
| 264 |
character set |
| 265 |
by an escape sequence ESC $ xx or ESC $ ( xx (for G0), |
| 266 |
ESC $ ) xx (for G1), ESC $ * xx (for G2), ESC $ + xx (for G3). |
| 267 |
For example, ESC $ ( C selects the Korean character set for G0. |
| 268 |
The Japanese character set selected by ESC $ B has a more |
| 269 |
recent version selected by ESC & @ ESC $ B. |
| 270 |
.P |
| 271 |
ISO/IEC\~4873 stipulates a narrower use of character sets, where G0 |
| 272 |
is fixed (always ASCII), so that G1, G2, and G3 |
| 273 |
can be invoked only for codes with the high order bit set. |
| 274 |
In particular, |
| 275 |
.B \[ha]N |
| 276 |
and |
| 277 |
.B \[ha]O |
| 278 |
are not used anymore, ESC ( xx |
| 279 |
can be used only with xx=B, and ESC ) xx, ESC * xx, ESC + xx |
| 280 |
are equivalent to ESC \- xx, ESC .\& xx, ESC / xx, respectively. |
| 281 |
.SS TIS-620 |
| 282 |
TIS-620 is a Thai national standard character set |
| 283 |
and a superset of ASCII. |
| 284 |
In the same fashion as the ISO/IEC\~8859 series, |
| 285 |
Thai characters are mapped into 0xa1\[en]0xfe. |
| 286 |
.SS Unicode |
| 287 |
Unicode (ISO/IEC 10646) is a standard which aims to |
| 288 |
unambiguously represent every character in every human language. |
| 289 |
Unicode's structure permits 20.1 bits to encode every character. |
| 290 |
Since most computers don't include 20.1-bit integers, |
| 291 |
Unicode is usually encoded as 32-bit integers internally |
| 292 |
and either a series of 16-bit integers (UTF-16) |
| 293 |
(needing two 16-bit integers only when encoding certain rare characters) |
| 294 |
or a series of 8-bit bytes (UTF-8). |
| 295 |
.P |
| 296 |
Linux represents Unicode using the 8-bit Unicode Transformation Format |
| 297 |
(UTF-8). |
| 298 |
UTF-8 is a variable length encoding of Unicode. |
| 299 |
It uses |
| 300 |
1 byte to code 7 bits, |
| 301 |
2 bytes for 11 bits, |
| 302 |
3 bytes for 16 bits, |
| 303 |
4 bytes for 21 bits, |
| 304 |
5 bytes for 26 bits, |
| 305 |
6 bytes for 31 bits. |
| 306 |
.P |
| 307 |
Let 0,1,x stand for a zero, one, or arbitrary bit. |
| 308 |
A byte 0xxxxxxx |
| 309 |
stands for the Unicode 00000000 0xxxxxxx |
| 310 |
which codes the same symbol as the ASCII 0xxxxxxx. |
| 311 |
Thus, |
| 312 |
ASCII goes unchanged into UTF-8, |
| 313 |
and people using only ASCII do not notice any change: |
| 314 |
not in code, |
| 315 |
and not in file size. |
| 316 |
.P |
| 317 |
A byte 110xxxxx is the start of a 2-byte code, |
| 318 |
and 110xxxxx 10yyyyyy is assembled into 00000xxx xxyyyyyy. |
| 319 |
A byte 1110xxxx is the start of a 3-byte code, |
| 320 |
and 1110xxxx 10yyyyyy 10zzzzzz is assembled into xxxxyyyy yyzzzzzz. |
| 321 |
(When UTF-8 is used to code the 31-bit ISO/IEC 10646 |
| 322 |
then this progression continues up to 6-byte codes.) |
| 323 |
.P |
| 324 |
For most texts in ISO/IEC\~8859 character sets, |
| 325 |
this means that the characters outside of ASCII are now coded with two bytes. |
| 326 |
This tends to expand ordinary text files by only one or two percent. |
| 327 |
For Russian or Greek texts, |
| 328 |
this expands ordinary text files by 100%, |
| 329 |
since text in those languages is mostly outside of ASCII. |
| 330 |
For Japanese users this means that |
| 331 |
the 16-bit codes now in common use will take three bytes. |
| 332 |
While there are algorithmic conversions from some character sets |
| 333 |
(especially ISO/IEC\~8859-1) |
| 334 |
to Unicode, |
| 335 |
general conversion requires carrying around conversion tables, |
| 336 |
which can be quite large for 16-bit codes. |
| 337 |
.P |
| 338 |
Note that UTF-8 is self-synchronizing: |
| 339 |
10xxxxxx is a tail, |
| 340 |
any other byte is the head of a code. |
| 341 |
Note that the only way ASCII bytes occur in a UTF-8 stream, |
| 342 |
is as themselves. |
| 343 |
In particular, |
| 344 |
there are no embedded NULs (\[aq]\[rs]0\[aq]) or \[aq]/\[aq]s |
| 345 |
that form part of some larger code. |
| 346 |
.P |
| 347 |
Since ASCII, and, in particular, NUL and \[aq]/\[aq], are unchanged, |
| 348 |
the kernel does not notice that UTF-8 is being used. |
| 349 |
It does not care at all what the bytes it is handling stand for. |
| 350 |
.P |
| 351 |
Rendering of Unicode data streams is |
| 352 |
typically handled through "subfont" tables |
| 353 |
which map a subset of Unicode to glyphs. |
| 354 |
Internally, |
| 355 |
the kernel uses Unicode to describe the subfont loaded in video RAM. |
| 356 |
This means that in the Linux console in UTF-8 mode, |
| 357 |
one can use a character set with 512 different symbols. |
| 358 |
This is not enough for Japanese, Chinese, and Korean, |
| 359 |
but it is enough for most other purposes. |
| 360 |
.SH SEE ALSO |
| 361 |
.BR iconv (1), |
| 362 |
.BR ascii (7), |
| 363 |
.BR iso_8859\-1 (7), |
| 364 |
.BR unicode (7), |
| 365 |
.BR utf\-8 (7) |