| 1 |
.\" Copyright 1996-2001, Markus Kuhn <[email protected]> |
| 2 |
.\" Copyright, the authors of the Linux man-pages project |
| 3 |
.\" |
| 4 |
.\" SPDX-License-Identifier: GPL-2.0-or-later |
| 5 |
.\" |
| 6 |
.TH UTF-8 7 2026-02-22 "Linux man-pages 6.18" |
| 7 |
.SH NAME |
| 8 |
UTF-8 \- an ASCII compatible multibyte Unicode encoding |
| 9 |
.SH DESCRIPTION |
| 10 |
The Unicode 3.0 character set occupies a 16-bit code space. |
| 11 |
The most obvious |
| 12 |
Unicode encoding (known as UCS-2) |
| 13 |
consists of a sequence of 16-bit words. |
| 14 |
Such strings can contain\[em]as part of many 16-bit characters\[em]bytes |
| 15 |
such as \[aq]\[rs]0\[aq] or \[aq]/\[aq], which have a |
| 16 |
special meaning in filenames and other C library function arguments. |
| 17 |
In addition, the majority of UNIX tools expect ASCII files and can't |
| 18 |
read 16-bit words as characters without major modifications. |
| 19 |
For these reasons, |
| 20 |
UCS-2 is not a suitable external encoding of Unicode |
| 21 |
in filenames, text files, environment variables, and so on. |
| 22 |
The ISO/IEC 10646 Universal Character Set (UCS), |
| 23 |
a superset of Unicode, occupies an even larger code |
| 24 |
space\[em]31\ bits\[em]and the obvious |
| 25 |
UCS-4 encoding for it (a sequence of 32-bit words) has the same problems. |
| 26 |
.P |
| 27 |
The UTF-8 encoding of Unicode and UCS |
| 28 |
does not have these problems and is the common way in which |
| 29 |
Unicode is used on UNIX-style operating systems. |
| 30 |
.SS Properties |
| 31 |
The UTF-8 encoding has the following nice properties: |
| 32 |
.IP \[bu] 3 |
| 33 |
UCS |
| 34 |
characters 0x00000000 to 0x0000007f (the classic US-ASCII |
| 35 |
characters) are encoded simply as bytes 0x00 to 0x7f (ASCII |
| 36 |
compatibility). |
| 37 |
This means that files and strings which contain only |
| 38 |
7-bit ASCII characters have the same encoding under both |
| 39 |
ASCII |
| 40 |
and |
| 41 |
UTF-8. |
| 42 |
.IP \[bu] |
| 43 |
All UCS characters greater than 0x7f are encoded as a multibyte sequence |
| 44 |
consisting only of bytes in the range 0x80 to 0xfd, so no ASCII |
| 45 |
byte can appear as part of another character and there are no |
| 46 |
problems with, for example, \[aq]\[rs]0\[aq] or \[aq]/\[aq]. |
| 47 |
.IP \[bu] |
| 48 |
The lexicographic sorting order of UCS-4 strings is preserved. |
| 49 |
.IP \[bu] |
| 50 |
All possible 2\[ha]31 UCS codes can be encoded using UTF-8. |
| 51 |
.IP \[bu] |
| 52 |
The bytes 0xc0, 0xc1, 0xfe, and 0xff are never used in the UTF-8 encoding. |
| 53 |
.IP \[bu] |
| 54 |
The first byte of a multibyte sequence which represents a single non-ASCII |
| 55 |
UCS character is always in the range 0xc2 to 0xfd and indicates how long |
| 56 |
this multibyte sequence is. |
| 57 |
All further bytes in a multibyte sequence |
| 58 |
are in the range 0x80 to 0xbf. |
| 59 |
This allows easy resynchronization and |
| 60 |
makes the encoding stateless and robust against missing bytes. |
| 61 |
.IP \[bu] |
| 62 |
UTF-8 encoded UCS characters may be up to six bytes long, however the |
| 63 |
Unicode standard specifies no characters above 0x10ffff, so Unicode characters |
| 64 |
can be only up to four bytes long in |
| 65 |
UTF-8. |
| 66 |
.SS Encoding |
| 67 |
The following byte sequences are used to represent a character. |
| 68 |
The sequence to be used depends on the UCS code number of the character: |
| 69 |
.TP |
| 70 |
0x00000000 \- 0x0000007F: |
| 71 |
.RI 0 xxxxxxx |
| 72 |
.TP |
| 73 |
0x00000080 \- 0x000007FF: |
| 74 |
.RI 110 xxxxx |
| 75 |
.RI 10 xxxxxx |
| 76 |
.TP |
| 77 |
0x00000800 \- 0x0000FFFF: |
| 78 |
.RI 1110 xxxx |
| 79 |
.RI 10 xxxxxx |
| 80 |
.RI 10 xxxxxx |
| 81 |
.TP |
| 82 |
0x00010000 \- 0x001FFFFF: |
| 83 |
.RI 11110 xxx |
| 84 |
.RI 10 xxxxxx |
| 85 |
.RI 10 xxxxxx |
| 86 |
.RI 10 xxxxxx |
| 87 |
.TP |
| 88 |
0x00200000 \- 0x03FFFFFF: |
| 89 |
.RI 111110 xx |
| 90 |
.RI 10 xxxxxx |
| 91 |
.RI 10 xxxxxx |
| 92 |
.RI 10 xxxxxx |
| 93 |
.RI 10 xxxxxx |
| 94 |
.TP |
| 95 |
0x04000000 \- 0x7FFFFFFF: |
| 96 |
.RI 1111110 x |
| 97 |
.RI 10 xxxxxx |
| 98 |
.RI 10 xxxxxx |
| 99 |
.RI 10 xxxxxx |
| 100 |
.RI 10 xxxxxx |
| 101 |
.RI 10 xxxxxx |
| 102 |
.P |
| 103 |
The |
| 104 |
.I xxx |
| 105 |
bit positions are filled with the bits of the character code number in |
| 106 |
binary representation, most significant bit first (big-endian). |
| 107 |
Only the shortest possible multibyte sequence |
| 108 |
which can represent the code number of the character can be used. |
| 109 |
.P |
| 110 |
The UCS code values 0xd800\[en]0xdfff (UTF-16 surrogates) as well as 0xfffe and |
| 111 |
0xffff (UCS noncharacters) should not appear in conforming UTF-8 streams. |
| 112 |
According to RFC 3629 no point above U+10FFFF should be used, |
| 113 |
which limits characters to four bytes. |
| 114 |
.SS Example |
| 115 |
The Unicode character 0xa9 = 1010 1001 (the copyright sign) is encoded |
| 116 |
in UTF-8 as |
| 117 |
.P |
| 118 |
.RS |
| 119 |
11000010 10101001 = 0xc2 0xa9 |
| 120 |
.RE |
| 121 |
.P |
| 122 |
and character 0x2260 = 0010 0010 0110 0000 (the "not equal" symbol) is |
| 123 |
encoded as: |
| 124 |
.P |
| 125 |
.RS |
| 126 |
11100010 10001001 10100000 = 0xe2 0x89 0xa0 |
| 127 |
.RE |
| 128 |
.SS Application notes |
| 129 |
Users have to select a UTF-8 locale, for example with |
| 130 |
.P |
| 131 |
.RS |
| 132 |
export LANG=en_GB.UTF-8 |
| 133 |
.RE |
| 134 |
.P |
| 135 |
in order to activate the UTF-8 support in applications. |
| 136 |
.P |
| 137 |
Application software that has to be aware of the used character |
| 138 |
encoding should always set the locale with for example |
| 139 |
.P |
| 140 |
.RS |
| 141 |
setlocale(LC_CTYPE, "") |
| 142 |
.RE |
| 143 |
.P |
| 144 |
and programmers can then test the expression |
| 145 |
.P |
| 146 |
.RS |
| 147 |
streq(nl_langinfo(CODESET), "UTF-8") |
| 148 |
.RE |
| 149 |
.P |
| 150 |
to determine whether a UTF-8 locale has been selected and whether |
| 151 |
therefore all plaintext standard input and output, terminal |
| 152 |
communication, plaintext file content, filenames, and environment |
| 153 |
variables are encoded in UTF-8. |
| 154 |
.P |
| 155 |
Programmers accustomed to single-byte encodings |
| 156 |
such as US-ASCII or ISO/IEC\~8859 |
| 157 |
have to be aware that two assumptions made so far are no longer valid |
| 158 |
in UTF-8 locales. |
| 159 |
Firstly, a single byte does not necessarily correspond any |
| 160 |
more to a single character. |
| 161 |
Secondly, since modern terminal emulators in UTF-8 |
| 162 |
mode also support Chinese, Japanese, and Korean |
| 163 |
double-width characters as well as nonspacing combining characters, |
| 164 |
outputting a single character does not necessarily advance the cursor |
| 165 |
by one position as it did in ASCII. |
| 166 |
Library functions such as |
| 167 |
.BR mbsrtowcs (3) |
| 168 |
and |
| 169 |
.BR wcswidth (3) |
| 170 |
should be used today to count characters and cursor positions. |
| 171 |
.P |
| 172 |
The official ESC sequence to switch from an ISO/IEC\~2022 |
| 173 |
encoding scheme (as used for instance by VT100 terminals) to |
| 174 |
UTF-8 is ESC % G |
| 175 |
("\[rs]x1b%G"). |
| 176 |
The corresponding return sequence from |
| 177 |
UTF-8 to ISO/IEC\~2022 is ESC % @ ("\[rs]x1b%@"). |
| 178 |
Other ISO/IEC\~2022 sequences (such as |
| 179 |
for switching the G0 and G1 sets) are not applicable in UTF-8 mode. |
| 180 |
.SS Security |
| 181 |
The Unicode and UCS standards require that producers of UTF-8 |
| 182 |
shall use the shortest form possible, for example, producing a two-byte |
| 183 |
sequence with first byte 0xc0 is nonconforming. |
| 184 |
Unicode 3.1 has added the requirement that conforming programs must not accept |
| 185 |
non-shortest forms in their input. |
| 186 |
This is for security reasons: if |
| 187 |
user input is checked for possible security violations, a program |
| 188 |
might check only for the ASCII |
| 189 |
version of "/../" or ";" or NUL and overlook that there are many |
| 190 |
non-ASCII ways to represent these things in a non-shortest UTF-8 |
| 191 |
encoding. |
| 192 |
.SS Standards |
| 193 |
ISO/IEC 10646-1:2000, Unicode 3.1, RFC\ 3629, Plan 9. |
| 194 |
.\" .SH AUTHOR |
| 195 |
.\" Markus Kuhn <[email protected]> |
| 196 |
.SH SEE ALSO |
| 197 |
.BR locale (1), |
| 198 |
.BR nl_langinfo (3), |
| 199 |
.BR setlocale (3), |
| 200 |
.BR charsets (7), |
| 201 |
.BR unicode (7) |