Debian Codegraph @

upstream/debian-unstable/man7/charsets.7

raw
1 .\" Copyright 1996, Eric S. Raymond <[email protected]>
2 .\" Copyright, Andries E. Brouwer <[email protected]>
3 .\" Copyright, the authors of the Linux man-pages project
4 .\"
5 .\" SPDX-License-Identifier: GPL-2.0-or-later
6 .\"
7 .TH charsets 7 2026-02-08 "Linux man-pages 6.18"
8 .SH NAME
9 charsets \- character set standards and internationalization
10 .SH DESCRIPTION
11 This manual page gives an overview on different character set standards
12 and how they were used on Linux before Unicode became ubiquitous.
13 Some of this information is still helpful for people working with legacy
14 systems and documents.
15 .P
16 Standards discussed include such as
17 ASCII, GB 2312, ISO/IEC\~8859, JIS, KOI8-R, KS, and Unicode.
18 .P
19 The primary emphasis is on character sets that were actually used by
20 locale character sets, not the myriad others that could be found in data
21 from other systems.
22 .SS ASCII
23 ASCII (American Standard Code For Information Interchange) is the original
24 7-bit character set, originally designed for American English.
25 Also known as US-ASCII.
26 It is currently described by the ISO/IEC\~646:1991 IRV
27 (International Reference Version) standard.
28 .P
29 Various ASCII variants replacing the dollar sign with other currency
30 symbols and replacing punctuation with non-English alphabetic
31 characters to cover German, French, Spanish, and others in 7 bits
32 emerged.
33 All are deprecated;
34 glibc does not support locales whose character sets are not true
35 supersets of ASCII.
36 .P
37 As Unicode, when using UTF-8, is ASCII-compatible, plain ASCII text
38 still renders properly on modern UTF-8 using systems.
39 .SS ISO/IEC\~8859
40 ISO/IEC\~8859 is a series of 15 8-bit character sets, all of which have ASCII
41 in their low (7-bit) half, invisible control characters in positions
42 128 to 159, and 96 fixed-width graphics in positions 160\[en]255.
43 .P
44 Of these, the most important is ISO/IEC\~8859-1
45 ("Latin Alphabet No.\& 1" / Latin-1).
46 It was widely adopted and supported by different systems,
47 and is gradually being replaced with Unicode.
48 The ISO/IEC\~8859-1 characters are also the first 256 characters of Unicode.
49 .P
50 Console support for the other ISO/IEC\~8859 character sets is available under
51 Linux through user-mode utilities (such as
52 .BR setfont (8))
53 that modify keyboard bindings and the EGA graphics
54 table and employ the "user mapping" font table in the console
55 driver.
56 .P
57 Here are brief descriptions of each character set:
58 .TP
59 ISO/IEC\~8859-1 (Latin-1)
60 Latin-1 covers many European languages such as Albanian, Basque,
61 Danish, English, Faroese, Galician, Icelandic, Irish, Italian,
62 Norwegian, Portuguese, Spanish, and Swedish.
63 The lack of the ligatures
64 Dutch IJ/ij,
65 French œ,
66 and „German“ quotation marks
67 was considered tolerable.
68 .TP
69 ISO/IEC\~8859-2 (Latin-2)
70 Latin-2 supports many Latin-written Central and East European
71 languages such as Bosnian, Croatian, Czech, German, Hungarian, Polish,
72 Slovak, and Slovene.
73 Replacing Romanian ș/ț with ş/ţ
74 was considered tolerable.
75 .TP
76 ISO/IEC\~8859-3 (Latin-3)
77 Latin-3 was designed to cover of Esperanto, Maltese, and Turkish, but
78 ISO/IEC\~8859-9 later superseded it for Turkish.
79 .TP
80 ISO/IEC\~8859-4 (Latin-4)
81 Latin-4 introduced letters for North European languages such as
82 Estonian, Latvian, and Lithuanian, but was superseded by ISO/IEC\~8859-10 and
83 ISO/IEC\~8859-13.
84 .TP
85 ISO/IEC\~8859-5
86 Cyrillic letters supporting Bulgarian, Byelorussian, Macedonian,
87 Russian, Serbian, and (almost completely) Ukrainian.
88 It was never widely used, see the discussion of KOI8-R/KOI8-U below.
89 .TP
90 ISO/IEC\~8859-6
91 Was created for Arabic.
92 The ISO/IEC\~8859-6 glyph table is a fixed font of separate
93 letter forms, but a proper display engine should combine these
94 using the proper initial, medial, and final forms.
95 .TP
96 ISO/IEC\~8859-7
97 Was created for Modern Greek in 1987,
98 updated in 2003.
99 .TP
100 ISO/IEC\~8859-8
101 Supports Modern Hebrew without niqud (punctuation signs).
102 Niqud and full-fledged Biblical Hebrew were
103 outside the scope of this character set.
104 .TP
105 ISO/IEC\~8859-9 (Latin-5)
106 This is a variant of Latin-1
107 that replaces Icelandic letters with Turkish ones.
108 .TP
109 ISO/IEC\~8859-10 (Latin-6)
110 Latin-6 added the Inuit (Greenlandic) and Sami (Lappish) letters
111 that were missing in Latin-4
112 to cover the entire Nordic area.
113 .TP
114 ISO/IEC\~8859-11
115 Supports the Thai alphabet and is nearly identical to the TIS-620 standard.
116 .TP
117 ISO/IEC\~8859-12
118 This character set does not exist.
119 .TP
120 ISO/IEC\~8859-13 (Latin-7)
121 Supports the Baltic Rim languages;
122 in particular,
123 it includes Latvian characters not found in Latin-4.
124 .TP
125 ISO/IEC\~8859-14 (Latin-8)
126 This is the Celtic character set, covering Old Irish, Manx, Gaelic,
127 Welsh, Cornish, and Breton.
128 .TP
129 ISO/IEC\~8859-15 (Latin-9)
130 Latin-9 is similar to the widely used Latin-1 but replaces some less
131 common symbols with the Euro sign and French and Finnish letters that
132 were missing in Latin-1.
133 .TP
134 ISO/IEC\~8859-16 (Latin-10)
135 This character set covers many Southeast European languages,
136 and most importantly supports Romanian more completely than Latin-2.
137 .SS KOI8-R / KOI8-U
138 KOI8-R is a non-ISO character set popular in Russia before Unicode.
139 The lower half is ASCII;
140 the upper is a Cyrillic character set somewhat better designed than
141 ISO/IEC\~8859-5.
142 KOI8-U, based on KOI8-R, has better support for Ukrainian.
143 Neither of these sets are ISO/IEC\~2022 compatible,
144 unlike the ISO/IEC\~8859 series.
145 .P
146 Console support for KOI8-R is available under Linux through user-mode
147 utilities that modify keyboard bindings and the EGA graphics table,
148 and employ the "user mapping" font table in the console driver.
149 .SS GB 2312
150 GB 2312 is a mainland Chinese national standard character set used
151 to express simplified Chinese.
152 Just like JIS X 0208, characters are
153 mapped into a 94x94 two-byte matrix used to construct EUC-CN.
154 EUC-CN
155 is the most important encoding for Linux and includes ASCII and
156 GB 2312.
157 Note that EUC-CN is often called as GB, GB 2312, or CN-GB.
158 .SS Big5
159 Big5 was a popular character set in Taiwan to express traditional
160 Chinese.
161 (Big5 is both a character set and an encoding.)
162 It is a superset of ASCII.
163 Non-ASCII characters are expressed in two bytes.
164 Bytes 0xa1\[en]0xfe are used as leading bytes for two-byte characters.
165 Big5 and its extension were widely used in Taiwan and Hong Kong.
166 It is not ISO/IEC\~2022 compliant.
167 .\" Thanks to Tomohiro KUBOTA for the following sections about
168 .\" national standards.
169 .SS JIS X 0208
170 JIS X 0208 is a Japanese national standard character set.
171 Though there are some more Japanese national standard character sets (like
172 JIS X 0201, JIS X 0212, and JIS X 0213), this is the most important one.
173 Characters are mapped into a 94x94 two-byte matrix,
174 whose each byte is in the range 0x21\[en]0x7e.
175 Note that JIS X 0208 is a character set, not an encoding.
176 This means that JIS X 0208
177 itself is not used for expressing text data.
178 JIS X 0208 is used
179 as a component to construct encodings such as EUC-JP, Shift_JIS,
180 and ISO/IEC\~2022-JP.
181 EUC-JP is the most important encoding for Linux
182 and includes ASCII and JIS X 0208.
183 In EUC-JP, JIS X 0208
184 characters are expressed in two bytes, each of which is the
185 JIS X 0208 code plus 0x80.
186 .SS KS X 1001
187 KS X 1001 is a Korean national standard character set.
188 Just as
189 JIS X 0208, characters are mapped into a 94x94 two-byte matrix.
190 KS X 1001 is used like JIS X 0208, as a component
191 to construct encodings such as EUC-KR, Johab, and ISO/IEC\~2022-KR.
192 EUC-KR is the most important encoding for Linux and includes
193 ASCII and KS X 1001.
194 KS C 5601 is an older name for KS X 1001.
195 .SS ISO/IEC\~2022 and ISO/IEC\~4873
196 The ISO/IEC\~2022 and ISO/IEC\~4873 standards describe a font-control model
197 based on VT100 practice.
198 This model is (partially) supported
199 by the Linux kernel and by
200 .BR xterm (1).
201 Several ISO/IEC\~2022-based character encodings have been defined,
202 especially for Japanese.
203 .P
204 There are 4 graphic character sets, called G0, G1, G2, and G3,
205 and one of them is the current character set for codes with
206 high bit zero (initially G0), and one of them is the current
207 character set for codes with high bit one (initially G1).
208 Each graphic character set has 94 or 96 characters, and is
209 essentially a 7-bit character set.
210 It uses codes either
211 040\[en]0177 (041\[en]0176) or 0240\[en]0377 (0241\[en]0376).
212 G0 always has size 94 and uses codes 041\[en]0176.
213 .P
214 Switching between character sets is done using the shift functions
215 .B \[ha]N
216 (SO or LS1),
217 .B \[ha]O
218 (SI or LS0), ESC n (LS2), ESC o (LS3),
219 ESC N (SS2), ESC O (SS3), ESC \[ti] (LS1R), ESC } (LS2R), ESC | (LS3R).
220 The function
221 .RI LS n
222 makes character set
223 .RI G n
224 the current one
225 for codes with high bit zero.
226 The function
227 .RI LS n R
228 makes character set
229 .RI G n
230 the current one
231 for codes with high bit one.
232 The function
233 .RI SS n
234 makes character set
235 .RI G n
236 .RI ( n =2
237 or 3)
238 the current one for the next character only (regardless of the value
239 of its high order bit).
240 .P
241 A 94-character set is designated as
242 .RI G n
243 character set
244 by an escape sequence ESC ( xx (for G0), ESC ) xx (for G1),
245 ESC * xx (for G2), ESC + xx (for G3), where xx is a symbol
246 or a pair of symbols found in the ISO/IEC\~2375 International
247 Register of Coded Character Sets.
248 For example, ESC ( @ selects the ISO/IEC\~646 character set as G0,
249 ESC ( A selects the UK standard character set (with pound
250 instead of number sign), ESC ( B selects ASCII (with dollar
251 instead of currency sign), ESC ( M selects a character set
252 for African languages, ESC ( ! A selects the Cuban character
253 set, and so on.
254 .P
255 A 96-character set is designated as
256 .RI G n
257 character set
258 by an escape sequence ESC \- xx (for G1), ESC .\& xx (for G2)
259 or ESC / xx (for G3).
260 For example, ESC \- G selects the Hebrew alphabet as G1.
261 .P
262 A multibyte character set is designated as
263 .RI G n
264 character set
265 by an escape sequence ESC $ xx or ESC $ ( xx (for G0),
266 ESC $ ) xx (for G1), ESC $ * xx (for G2), ESC $ + xx (for G3).
267 For example, ESC $ ( C selects the Korean character set for G0.
268 The Japanese character set selected by ESC $ B has a more
269 recent version selected by ESC & @ ESC $ B.
270 .P
271 ISO/IEC\~4873 stipulates a narrower use of character sets, where G0
272 is fixed (always ASCII), so that G1, G2, and G3
273 can be invoked only for codes with the high order bit set.
274 In particular,
275 .B \[ha]N
276 and
277 .B \[ha]O
278 are not used anymore, ESC ( xx
279 can be used only with xx=B, and ESC ) xx, ESC * xx, ESC + xx
280 are equivalent to ESC \- xx, ESC .\& xx, ESC / xx, respectively.
281 .SS TIS-620
282 TIS-620 is a Thai national standard character set
283 and a superset of ASCII.
284 In the same fashion as the ISO/IEC\~8859 series,
285 Thai characters are mapped into 0xa1\[en]0xfe.
286 .SS Unicode
287 Unicode (ISO/IEC 10646) is a standard which aims to
288 unambiguously represent every character in every human language.
289 Unicode's structure permits 20.1 bits to encode every character.
290 Since most computers don't include 20.1-bit integers,
291 Unicode is usually encoded as 32-bit integers internally
292 and either a series of 16-bit integers (UTF-16)
293 (needing two 16-bit integers only when encoding certain rare characters)
294 or a series of 8-bit bytes (UTF-8).
295 .P
296 Linux represents Unicode using the 8-bit Unicode Transformation Format
297 (UTF-8).
298 UTF-8 is a variable length encoding of Unicode.
299 It uses
300 1 byte to code 7 bits,
301 2 bytes for 11 bits,
302 3 bytes for 16 bits,
303 4 bytes for 21 bits,
304 5 bytes for 26 bits,
305 6 bytes for 31 bits.
306 .P
307 Let 0,1,x stand for a zero, one, or arbitrary bit.
308 A byte 0xxxxxxx
309 stands for the Unicode 00000000 0xxxxxxx
310 which codes the same symbol as the ASCII 0xxxxxxx.
311 Thus,
312 ASCII goes unchanged into UTF-8,
313 and people using only ASCII do not notice any change:
314 not in code,
315 and not in file size.
316 .P
317 A byte 110xxxxx is the start of a 2-byte code,
318 and 110xxxxx 10yyyyyy is assembled into 00000xxx xxyyyyyy.
319 A byte 1110xxxx is the start of a 3-byte code,
320 and 1110xxxx 10yyyyyy 10zzzzzz is assembled into xxxxyyyy yyzzzzzz.
321 (When UTF-8 is used to code the 31-bit ISO/IEC 10646
322 then this progression continues up to 6-byte codes.)
323 .P
324 For most texts in ISO/IEC\~8859 character sets,
325 this means that the characters outside of ASCII are now coded with two bytes.
326 This tends to expand ordinary text files by only one or two percent.
327 For Russian or Greek texts,
328 this expands ordinary text files by 100%,
329 since text in those languages is mostly outside of ASCII.
330 For Japanese users this means that
331 the 16-bit codes now in common use will take three bytes.
332 While there are algorithmic conversions from some character sets
333 (especially ISO/IEC\~8859-1)
334 to Unicode,
335 general conversion requires carrying around conversion tables,
336 which can be quite large for 16-bit codes.
337 .P
338 Note that UTF-8 is self-synchronizing:
339 10xxxxxx is a tail,
340 any other byte is the head of a code.
341 Note that the only way ASCII bytes occur in a UTF-8 stream,
342 is as themselves.
343 In particular,
344 there are no embedded NULs (\[aq]\[rs]0\[aq]) or \[aq]/\[aq]s
345 that form part of some larger code.
346 .P
347 Since ASCII, and, in particular, NUL and \[aq]/\[aq], are unchanged,
348 the kernel does not notice that UTF-8 is being used.
349 It does not care at all what the bytes it is handling stand for.
350 .P
351 Rendering of Unicode data streams is
352 typically handled through "subfont" tables
353 which map a subset of Unicode to glyphs.
354 Internally,
355 the kernel uses Unicode to describe the subfont loaded in video RAM.
356 This means that in the Linux console in UTF-8 mode,
357 one can use a character set with 512 different symbols.
358 This is not enough for Japanese, Chinese, and Korean,
359 but it is enough for most other purposes.
360 .SH SEE ALSO
361 .BR iconv (1),
362 .BR ascii (7),
363 .BR iso_8859\-1 (7),
364 .BR unicode (7),
365 .BR utf\-8 (7)