| 1 |
.. _unicode-howto: |
| 2 |
|
| 3 |
***************** |
| 4 |
Unicode HOWTO |
| 5 |
***************** |
| 6 |
|
| 7 |
:Release: 1.12 |
| 8 |
|
| 9 |
This HOWTO discusses Python's support for the Unicode specification |
| 10 |
for representing textual data, and explains various problems that |
| 11 |
people commonly encounter when trying to work with Unicode. |
| 12 |
|
| 13 |
|
| 14 |
Introduction to Unicode |
| 15 |
======================= |
| 16 |
|
| 17 |
Definitions |
| 18 |
----------- |
| 19 |
|
| 20 |
Today's programs need to be able to handle a wide variety of |
| 21 |
characters. Applications are often internationalized to display |
| 22 |
messages and output in a variety of user-selectable languages; the |
| 23 |
same program might need to output an error message in English, French, |
| 24 |
Japanese, Hebrew, or Russian. Web content can be written in any of |
| 25 |
these languages and can also include a variety of emoji symbols. |
| 26 |
Python's string type uses the Unicode Standard for representing |
| 27 |
characters, which lets Python programs work with all these different |
| 28 |
possible characters. |
| 29 |
|
| 30 |
Unicode (https://www.unicode.org/) is a specification that aims to |
| 31 |
list every character used by human languages and give each character |
| 32 |
its own unique code. The Unicode specifications are continually |
| 33 |
revised and updated to add new languages and symbols. |
| 34 |
|
| 35 |
A **character** is the smallest possible component of a text. 'A', 'B', 'C', |
| 36 |
etc., are all different characters. So are 'È' and 'Í'. Characters vary |
| 37 |
depending on the language or context you're talking |
| 38 |
about. For example, there's a character for "Roman Numeral One", 'Ⅰ', that's |
| 39 |
separate from the uppercase letter 'I'. They'll usually look the same, |
| 40 |
but these are two different characters that have different meanings. |
| 41 |
|
| 42 |
The Unicode standard describes how characters are represented by |
| 43 |
**code points**. A code point value is an integer in the range 0 to |
| 44 |
0x10FFFF (about 1.1 million values, the |
| 45 |
`actual number assigned <https://www.unicode.org/versions/latest/#Summary>`_ |
| 46 |
is less than that). In the standard and in this document, a code point is written |
| 47 |
using the notation ``U+265E`` to mean the character with value |
| 48 |
``0x265e`` (9,822 in decimal). |
| 49 |
|
| 50 |
The Unicode standard contains a lot of tables listing characters and |
| 51 |
their corresponding code points: |
| 52 |
|
| 53 |
.. code-block:: none |
| 54 |
|
| 55 |
0061 'a'; LATIN SMALL LETTER A |
| 56 |
0062 'b'; LATIN SMALL LETTER B |
| 57 |
0063 'c'; LATIN SMALL LETTER C |
| 58 |
... |
| 59 |
007B '{'; LEFT CURLY BRACKET |
| 60 |
... |
| 61 |
2167 'Ⅷ'; ROMAN NUMERAL EIGHT |
| 62 |
2168 'Ⅸ'; ROMAN NUMERAL NINE |
| 63 |
... |
| 64 |
265E '♞'; BLACK CHESS KNIGHT |
| 65 |
265F '♟'; BLACK CHESS PAWN |
| 66 |
... |
| 67 |
1F600 '😀'; GRINNING FACE |
| 68 |
1F609 '😉'; WINKING FACE |
| 69 |
... |
| 70 |
|
| 71 |
Strictly, these definitions imply that it's meaningless to say 'this is |
| 72 |
character ``U+265E``'. ``U+265E`` is a code point, which represents some particular |
| 73 |
character; in this case, it represents the character 'BLACK CHESS KNIGHT', |
| 74 |
'♞'. In |
| 75 |
informal contexts, this distinction between code points and characters will |
| 76 |
sometimes be forgotten. |
| 77 |
|
| 78 |
A character is represented on a screen or on paper by a set of graphical |
| 79 |
elements that's called a **glyph**. The glyph for an uppercase A, for example, |
| 80 |
is two diagonal strokes and a horizontal stroke, though the exact details will |
| 81 |
depend on the font being used. Most Python code doesn't need to worry about |
| 82 |
glyphs; figuring out the correct glyph to display is generally the job of a GUI |
| 83 |
toolkit or a terminal's font renderer. |
| 84 |
|
| 85 |
|
| 86 |
Encodings |
| 87 |
--------- |
| 88 |
|
| 89 |
To summarize the previous section: a Unicode string is a sequence of |
| 90 |
code points, which are numbers from 0 through ``0x10FFFF`` (1,114,111 |
| 91 |
decimal). This sequence of code points needs to be represented in |
| 92 |
memory as a set of **code units**, and **code units** are then mapped |
| 93 |
to 8-bit bytes. The rules for translating a Unicode string into a |
| 94 |
sequence of bytes are called a **character encoding**, or just |
| 95 |
an **encoding**. |
| 96 |
|
| 97 |
The first encoding you might think of is using 32-bit integers as the |
| 98 |
code unit, and then using the CPU's representation of 32-bit integers. |
| 99 |
In this representation, the string "Python" might look like this: |
| 100 |
|
| 101 |
.. code-block:: none |
| 102 |
|
| 103 |
P y t h o n |
| 104 |
0x50 00 00 00 79 00 00 00 74 00 00 00 68 00 00 00 6f 00 00 00 6e 00 00 00 |
| 105 |
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
| 106 |
|
| 107 |
This representation is straightforward but using it presents a number of |
| 108 |
problems. |
| 109 |
|
| 110 |
1. It's not portable; different processors order the bytes differently. |
| 111 |
|
| 112 |
2. It's very wasteful of space. In most texts, the majority of the code points |
| 113 |
are less than 127, or less than 255, so a lot of space is occupied by ``0x00`` |
| 114 |
bytes. The above string takes 24 bytes compared to the 6 bytes needed for an |
| 115 |
ASCII representation. Increased RAM usage doesn't matter too much (desktop |
| 116 |
computers have gigabytes of RAM, and strings aren't usually that large), but |
| 117 |
expanding our usage of disk and network bandwidth by a factor of 4 is |
| 118 |
intolerable. |
| 119 |
|
| 120 |
3. It's not compatible with existing C functions such as ``strlen()``, so a new |
| 121 |
family of wide string functions would need to be used. |
| 122 |
|
| 123 |
Therefore this encoding isn't used very much, and people instead choose other |
| 124 |
encodings that are more efficient and convenient, such as UTF-8. |
| 125 |
|
| 126 |
UTF-8 is one of the most commonly used encodings, and Python often |
| 127 |
defaults to using it. UTF stands for "Unicode Transformation Format", |
| 128 |
and the '8' means that 8-bit values are used in the encoding. (There |
| 129 |
are also UTF-16 and UTF-32 encodings, but they are less frequently |
| 130 |
used than UTF-8.) UTF-8 uses the following rules: |
| 131 |
|
| 132 |
1. If the code point is < 128, it's represented by the corresponding byte value. |
| 133 |
2. If the code point is >= 128, it's turned into a sequence of two, three, or |
| 134 |
four bytes, where each byte of the sequence is between 128 and 255. |
| 135 |
|
| 136 |
UTF-8 has several convenient properties: |
| 137 |
|
| 138 |
1. It can handle any Unicode code point. |
| 139 |
2. A Unicode string is turned into a sequence of bytes that contains embedded |
| 140 |
zero bytes only where they represent the null character (U+0000). This means |
| 141 |
that UTF-8 strings can be processed by C functions such as ``strcpy()`` and sent |
| 142 |
through protocols that can't handle zero bytes for anything other than |
| 143 |
end-of-string markers. |
| 144 |
3. A string of ASCII text is also valid UTF-8 text. |
| 145 |
4. UTF-8 is fairly compact; the majority of commonly used characters can be |
| 146 |
represented with one or two bytes. |
| 147 |
5. If bytes are corrupted or lost, it's possible to determine the start of the |
| 148 |
next UTF-8-encoded code point and resynchronize. It's also unlikely that |
| 149 |
random 8-bit data will look like valid UTF-8. |
| 150 |
6. UTF-8 is a byte oriented encoding. The encoding specifies that each |
| 151 |
character is represented by a specific sequence of one or more bytes. This |
| 152 |
avoids the byte-ordering issues that can occur with integer and word oriented |
| 153 |
encodings, like UTF-16 and UTF-32, where the sequence of bytes varies depending |
| 154 |
on the hardware on which the string was encoded. |
| 155 |
|
| 156 |
|
| 157 |
References |
| 158 |
---------- |
| 159 |
|
| 160 |
The `Unicode Consortium site <https://www.unicode.org>`_ has character charts, a |
| 161 |
glossary, and PDF versions of the Unicode specification. Be prepared for some |
| 162 |
difficult reading. `A chronology <https://www.unicode.org/history/>`_ of the |
| 163 |
origin and development of Unicode is also available on the site. |
| 164 |
|
| 165 |
On the Computerphile Youtube channel, Tom Scott briefly |
| 166 |
`discusses the history of Unicode and UTF-8 <https://www.youtube.com/watch?v=MijmeoH9LT4>`_ |
| 167 |
(9 minutes 36 seconds). |
| 168 |
|
| 169 |
To help understand the standard, Jukka Korpela has written `an introductory |
| 170 |
guide <https://jkorpela.fi/unicode/guide.html>`_ to reading the |
| 171 |
Unicode character tables. |
| 172 |
|
| 173 |
Another `good introductory article <https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/>`_ |
| 174 |
was written by Joel Spolsky. |
| 175 |
If this introduction didn't make things clear to you, you should try |
| 176 |
reading this alternate article before continuing. |
| 177 |
|
| 178 |
Wikipedia entries are often helpful; see the entries for "`character encoding |
| 179 |
<https://en.wikipedia.org/wiki/Character_encoding>`_" and `UTF-8 |
| 180 |
<https://en.wikipedia.org/wiki/UTF-8>`_, for example. |
| 181 |
|
| 182 |
|
| 183 |
Python's Unicode Support |
| 184 |
======================== |
| 185 |
|
| 186 |
Now that you've learned the rudiments of Unicode, we can look at Python's |
| 187 |
Unicode features. |
| 188 |
|
| 189 |
The String Type |
| 190 |
--------------- |
| 191 |
|
| 192 |
Since Python 3.0, the language's :class:`str` type contains Unicode |
| 193 |
characters, meaning any string created using ``"unicode rocks!"``, ``'unicode |
| 194 |
rocks!'``, or the triple-quoted string syntax is stored as Unicode. |
| 195 |
|
| 196 |
The default encoding for Python source code is UTF-8, so you can simply |
| 197 |
include a Unicode character in a string literal:: |
| 198 |
|
| 199 |
try: |
| 200 |
with open('/tmp/input.txt', 'r') as f: |
| 201 |
... |
| 202 |
except OSError: |
| 203 |
# 'File not found' error message. |
| 204 |
print("Fichier non trouvé") |
| 205 |
|
| 206 |
Side note: Python 3 also supports using Unicode characters in identifiers:: |
| 207 |
|
| 208 |
répertoire = "/tmp/records.log" |
| 209 |
with open(répertoire, "w") as f: |
| 210 |
f.write("test\n") |
| 211 |
|
| 212 |
If you can't enter a particular character in your editor or want to |
| 213 |
keep the source code ASCII-only for some reason, you can also use |
| 214 |
escape sequences in string literals. (Depending on your system, |
| 215 |
you may see the actual capital-delta glyph instead of a \u escape.) :: |
| 216 |
|
| 217 |
>>> "\N{GREEK CAPITAL LETTER DELTA}" # Using the character name |
| 218 |
'\u0394' |
| 219 |
>>> "\u0394" # Using a 16-bit hex value |
| 220 |
'\u0394' |
| 221 |
>>> "\U00000394" # Using a 32-bit hex value |
| 222 |
'\u0394' |
| 223 |
|
| 224 |
In addition, one can create a string using the :func:`~bytes.decode` method of |
| 225 |
:class:`bytes`. This method takes an *encoding* argument, such as ``UTF-8``, |
| 226 |
and optionally an *errors* argument. |
| 227 |
|
| 228 |
The *errors* argument specifies the response when the input string can't be |
| 229 |
converted according to the encoding's rules. Legal values for this argument are |
| 230 |
``'strict'`` (raise a :exc:`UnicodeDecodeError` exception), ``'replace'`` (use |
| 231 |
``U+FFFD``, ``REPLACEMENT CHARACTER``), ``'ignore'`` (just leave the |
| 232 |
character out of the Unicode result), or ``'backslashreplace'`` (inserts a |
| 233 |
``\xNN`` escape sequence). |
| 234 |
The following examples show the differences:: |
| 235 |
|
| 236 |
>>> b'\x80abc'.decode("utf-8", "strict") #doctest: +NORMALIZE_WHITESPACE |
| 237 |
Traceback (most recent call last): |
| 238 |
... |
| 239 |
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: |
| 240 |
invalid start byte |
| 241 |
>>> b'\x80abc'.decode("utf-8", "replace") |
| 242 |
'\ufffdabc' |
| 243 |
>>> b'\x80abc'.decode("utf-8", "backslashreplace") |
| 244 |
'\\x80abc' |
| 245 |
>>> b'\x80abc'.decode("utf-8", "ignore") |
| 246 |
'abc' |
| 247 |
|
| 248 |
Encodings are specified as strings containing the encoding's name. Python |
| 249 |
comes with roughly 100 different encodings; see the Python Library Reference at |
| 250 |
:ref:`standard-encodings` for a list. Some encodings have multiple names; for |
| 251 |
example, ``'latin-1'``, ``'iso_8859_1'`` and ``'8859``' are all synonyms for |
| 252 |
the same encoding. |
| 253 |
|
| 254 |
One-character Unicode strings can also be created with the :func:`chr` |
| 255 |
built-in function, which takes integers and returns a Unicode string of length 1 |
| 256 |
that contains the corresponding code point. The reverse operation is the |
| 257 |
built-in :func:`ord` function that takes a one-character Unicode string and |
| 258 |
returns the code point value:: |
| 259 |
|
| 260 |
>>> chr(57344) |
| 261 |
'\ue000' |
| 262 |
>>> ord('\ue000') |
| 263 |
57344 |
| 264 |
|
| 265 |
Converting to Bytes |
| 266 |
------------------- |
| 267 |
|
| 268 |
The opposite method of :meth:`bytes.decode` is :meth:`str.encode`, |
| 269 |
which returns a :class:`bytes` representation of the Unicode string, encoded in the |
| 270 |
requested *encoding*. |
| 271 |
|
| 272 |
The *errors* parameter is the same as the parameter of the |
| 273 |
:meth:`~bytes.decode` method but supports a few more possible handlers. As well as |
| 274 |
``'strict'``, ``'ignore'``, and ``'replace'`` (which in this case |
| 275 |
inserts a question mark instead of the unencodable character), there is |
| 276 |
also ``'xmlcharrefreplace'`` (inserts an XML character reference), |
| 277 |
``backslashreplace`` (inserts a ``\uNNNN`` escape sequence) and |
| 278 |
``namereplace`` (inserts a ``\N{...}`` escape sequence). |
| 279 |
|
| 280 |
The following example shows the different results:: |
| 281 |
|
| 282 |
>>> u = chr(40960) + 'abcd' + chr(1972) |
| 283 |
>>> u.encode('utf-8') |
| 284 |
b'\xea\x80\x80abcd\xde\xb4' |
| 285 |
>>> u.encode('ascii') #doctest: +NORMALIZE_WHITESPACE |
| 286 |
Traceback (most recent call last): |
| 287 |
... |
| 288 |
UnicodeEncodeError: 'ascii' codec can't encode character '\ua000' in |
| 289 |
position 0: ordinal not in range(128) |
| 290 |
>>> u.encode('ascii', 'ignore') |
| 291 |
b'abcd' |
| 292 |
>>> u.encode('ascii', 'replace') |
| 293 |
b'?abcd?' |
| 294 |
>>> u.encode('ascii', 'xmlcharrefreplace') |
| 295 |
b'ꀀabcd޴' |
| 296 |
>>> u.encode('ascii', 'backslashreplace') |
| 297 |
b'\\ua000abcd\\u07b4' |
| 298 |
>>> u.encode('ascii', 'namereplace') |
| 299 |
b'\\N{YI SYLLABLE IT}abcd\\u07b4' |
| 300 |
|
| 301 |
The low-level routines for registering and accessing the available |
| 302 |
encodings are found in the :mod:`codecs` module. Implementing new |
| 303 |
encodings also requires understanding the :mod:`codecs` module. |
| 304 |
However, the encoding and decoding functions returned by this module |
| 305 |
are usually more low-level than is comfortable, and writing new encodings |
| 306 |
is a specialized task, so the module won't be covered in this HOWTO. |
| 307 |
|
| 308 |
|
| 309 |
Unicode Literals in Python Source Code |
| 310 |
-------------------------------------- |
| 311 |
|
| 312 |
In Python source code, specific Unicode code points can be written using the |
| 313 |
``\u`` escape sequence, which is followed by four hex digits giving the code |
| 314 |
point. The ``\U`` escape sequence is similar, but expects eight hex digits, |
| 315 |
not four:: |
| 316 |
|
| 317 |
>>> s = "a\xac\u1234\u20ac\U00008000" |
| 318 |
... # ^^^^ two-digit hex escape |
| 319 |
... # ^^^^^^ four-digit Unicode escape |
| 320 |
... # ^^^^^^^^^^ eight-digit Unicode escape |
| 321 |
>>> [ord(c) for c in s] |
| 322 |
[97, 172, 4660, 8364, 32768] |
| 323 |
|
| 324 |
Using escape sequences for code points greater than 127 is fine in small doses, |
| 325 |
but becomes an annoyance if you're using many accented characters, as you would |
| 326 |
in a program with messages in French or some other accent-using language. You |
| 327 |
can also assemble strings using the :func:`chr` built-in function, but this is |
| 328 |
even more tedious. |
| 329 |
|
| 330 |
Ideally, you'd want to be able to write literals in your language's natural |
| 331 |
encoding. You could then edit Python source code with your favorite editor |
| 332 |
which would display the accented characters naturally, and have the right |
| 333 |
characters used at runtime. |
| 334 |
|
| 335 |
Python supports writing source code in UTF-8 by default, but you can use almost |
| 336 |
any encoding if you declare the encoding being used. This is done by including |
| 337 |
a special comment as either the first or second line of the source file:: |
| 338 |
|
| 339 |
#!/usr/bin/env python |
| 340 |
# -*- coding: latin-1 -*- |
| 341 |
|
| 342 |
u = 'abcdé' |
| 343 |
print(ord(u[-1])) |
| 344 |
|
| 345 |
The syntax is inspired by Emacs's notation for specifying variables local to a |
| 346 |
file. Emacs supports many different variables, but Python only supports |
| 347 |
'coding'. The ``-*-`` symbols indicate to Emacs that the comment is special; |
| 348 |
they have no significance to Python but are a convention. Python looks for |
| 349 |
``coding: name`` or ``coding=name`` in the comment. |
| 350 |
|
| 351 |
If you don't include such a comment, the default encoding used will be UTF-8 as |
| 352 |
already mentioned. See also :pep:`263` for more information. |
| 353 |
|
| 354 |
|
| 355 |
.. _unicode-properties: |
| 356 |
|
| 357 |
Unicode Properties |
| 358 |
------------------ |
| 359 |
|
| 360 |
The Unicode specification includes a database of information about |
| 361 |
code points. For each defined code point, the information includes |
| 362 |
the character's name, its category, the numeric value if applicable |
| 363 |
(for characters representing numeric concepts such as the Roman |
| 364 |
numerals, fractions such as one-third and four-fifths, etc.). There |
| 365 |
are also display-related properties, such as how to use the code point |
| 366 |
in bidirectional text. |
| 367 |
|
| 368 |
The following program displays some information about several characters, and |
| 369 |
prints the numeric value of one particular character:: |
| 370 |
|
| 371 |
import unicodedata |
| 372 |
|
| 373 |
u = chr(233) + chr(0x0bf2) + chr(3972) + chr(6000) + chr(13231) |
| 374 |
|
| 375 |
for i, c in enumerate(u): |
| 376 |
print(i, '%04x' % ord(c), unicodedata.category(c), end=" ") |
| 377 |
print(unicodedata.name(c)) |
| 378 |
|
| 379 |
# Get numeric value of second character |
| 380 |
print(unicodedata.numeric(u[1])) |
| 381 |
|
| 382 |
When run, this prints: |
| 383 |
|
| 384 |
.. code-block:: none |
| 385 |
|
| 386 |
0 00e9 Ll LATIN SMALL LETTER E WITH ACUTE |
| 387 |
1 0bf2 No TAMIL NUMBER ONE THOUSAND |
| 388 |
2 0f84 Mn TIBETAN MARK HALANTA |
| 389 |
3 1770 Lo TAGBANWA LETTER SA |
| 390 |
4 33af So SQUARE RAD OVER S SQUARED |
| 391 |
1000.0 |
| 392 |
|
| 393 |
The category codes are abbreviations describing the nature of the character. |
| 394 |
These are grouped into categories such as "Letter", "Number", "Punctuation", or |
| 395 |
"Symbol", which in turn are broken up into subcategories. To take the codes |
| 396 |
from the above output, ``'Ll'`` means 'Letter, lowercase', ``'No'`` means |
| 397 |
"Number, other", ``'Mn'`` is "Mark, nonspacing", and ``'So'`` is "Symbol, |
| 398 |
other". See |
| 399 |
`the General Category Values section of the Unicode Character Database documentation <https://www.unicode.org/reports/tr44/#General_Category_Values>`_ for a |
| 400 |
list of category codes. |
| 401 |
|
| 402 |
|
| 403 |
Comparing Strings |
| 404 |
----------------- |
| 405 |
|
| 406 |
Unicode adds some complication to comparing strings, because the same |
| 407 |
set of characters can be represented by different sequences of code |
| 408 |
points. For example, a letter like 'ê' can be represented as a single |
| 409 |
code point U+00EA, or as U+0065 U+0302, which is the code point for |
| 410 |
'e' followed by a code point for 'COMBINING CIRCUMFLEX ACCENT'. These |
| 411 |
will produce the same output when printed, but one is a string of |
| 412 |
length 1 and the other is of length 2. |
| 413 |
|
| 414 |
One tool for a case-insensitive comparison is the |
| 415 |
:meth:`~str.casefold` string method that converts a string to a |
| 416 |
case-insensitive form following an algorithm described by the Unicode |
| 417 |
Standard. This algorithm has special handling for characters such as |
| 418 |
the German letter 'ß' (code point U+00DF), which becomes the pair of |
| 419 |
lowercase letters 'ss'. |
| 420 |
|
| 421 |
:: |
| 422 |
|
| 423 |
>>> street = 'Gürzenichstraße' |
| 424 |
>>> street.casefold() |
| 425 |
'gürzenichstrasse' |
| 426 |
|
| 427 |
A second tool is the :mod:`unicodedata` module's |
| 428 |
:func:`~unicodedata.normalize` function that converts strings to one |
| 429 |
of several normal forms, where letters followed by a combining character are |
| 430 |
replaced with single characters. :func:`~unicodedata.normalize` can |
| 431 |
be used to perform string comparisons that won't falsely report |
| 432 |
inequality if two strings use combining characters differently: |
| 433 |
|
| 434 |
:: |
| 435 |
|
| 436 |
import unicodedata |
| 437 |
|
| 438 |
def compare_strs(s1, s2): |
| 439 |
def NFD(s): |
| 440 |
return unicodedata.normalize('NFD', s) |
| 441 |
|
| 442 |
return NFD(s1) == NFD(s2) |
| 443 |
|
| 444 |
single_char = 'ê' |
| 445 |
multiple_chars = '\N{LATIN SMALL LETTER E}\N{COMBINING CIRCUMFLEX ACCENT}' |
| 446 |
print('length of first string=', len(single_char)) |
| 447 |
print('length of second string=', len(multiple_chars)) |
| 448 |
print(compare_strs(single_char, multiple_chars)) |
| 449 |
|
| 450 |
When run, this outputs: |
| 451 |
|
| 452 |
.. code-block:: shell-session |
| 453 |
|
| 454 |
$ python compare-strs.py |
| 455 |
length of first string= 1 |
| 456 |
length of second string= 2 |
| 457 |
True |
| 458 |
|
| 459 |
The first argument to the :func:`~unicodedata.normalize` function is a |
| 460 |
string giving the desired normalization form, which can be one of |
| 461 |
'NFC', 'NFKC', 'NFD', and 'NFKD'. |
| 462 |
|
| 463 |
The Unicode Standard also specifies how to do caseless comparisons:: |
| 464 |
|
| 465 |
import unicodedata |
| 466 |
|
| 467 |
def compare_caseless(s1, s2): |
| 468 |
def NFD(s): |
| 469 |
return unicodedata.normalize('NFD', s) |
| 470 |
|
| 471 |
return NFD(NFD(s1).casefold()) == NFD(NFD(s2).casefold()) |
| 472 |
|
| 473 |
# Example usage |
| 474 |
single_char = 'ê' |
| 475 |
multiple_chars = '\N{LATIN CAPITAL LETTER E}\N{COMBINING CIRCUMFLEX ACCENT}' |
| 476 |
|
| 477 |
print(compare_caseless(single_char, multiple_chars)) |
| 478 |
|
| 479 |
This will print ``True``. (Why is :func:`!NFD` invoked twice? Because |
| 480 |
there are a few characters that make :meth:`~str.casefold` return a |
| 481 |
non-normalized string, so the result needs to be normalized again. See |
| 482 |
section 3.13 of the Unicode Standard for a discussion and an example.) |
| 483 |
|
| 484 |
|
| 485 |
Unicode Regular Expressions |
| 486 |
--------------------------- |
| 487 |
|
| 488 |
The regular expressions supported by the :mod:`re` module can be provided |
| 489 |
either as bytes or strings. Some of the special character sequences such as |
| 490 |
``\d`` and ``\w`` have different meanings depending on whether |
| 491 |
the pattern is supplied as bytes or a string. For example, |
| 492 |
``\d`` will match the characters ``[0-9]`` in bytes but |
| 493 |
in strings will match any character that's in the ``'Nd'`` category. |
| 494 |
|
| 495 |
The string in this example has the number 57 written in both Thai and |
| 496 |
Arabic numerals:: |
| 497 |
|
| 498 |
import re |
| 499 |
p = re.compile(r'\d+') |
| 500 |
|
| 501 |
s = "Over \u0e55\u0e57 57 flavours" |
| 502 |
m = p.search(s) |
| 503 |
print(repr(m.group())) |
| 504 |
|
| 505 |
When executed, ``\d+`` will match the Thai numerals and print them |
| 506 |
out. If you supply the :const:`re.ASCII` flag to |
| 507 |
:func:`~re.compile`, ``\d+`` will match the substring "57" instead. |
| 508 |
|
| 509 |
Similarly, ``\w`` matches a wide variety of Unicode characters but |
| 510 |
only ``[a-zA-Z0-9_]`` in bytes or if :const:`re.ASCII` is supplied, |
| 511 |
and ``\s`` will match either Unicode whitespace characters or |
| 512 |
``[ \t\n\r\f\v]``. |
| 513 |
|
| 514 |
|
| 515 |
References |
| 516 |
---------- |
| 517 |
|
| 518 |
.. comment should these be mentioned earlier, e.g. at the start of the "introduction to Unicode" first section? |
| 519 |
|
| 520 |
Some good alternative discussions of Python's Unicode support are: |
| 521 |
|
| 522 |
* `Processing Text Files in Python 3 <https://python-notes.curiousefficiency.org/en/latest/python3/text_file_processing.html>`_, by Nick Coghlan. |
| 523 |
* `Pragmatic Unicode <https://nedbatchelder.com/text/unipain.html>`_, a PyCon 2012 presentation by Ned Batchelder. |
| 524 |
|
| 525 |
The :class:`str` type is described in the Python library reference at |
| 526 |
:ref:`textseq`. |
| 527 |
|
| 528 |
The documentation for the :mod:`unicodedata` module. |
| 529 |
|
| 530 |
The documentation for the :mod:`codecs` module. |
| 531 |
|
| 532 |
Marc-André Lemburg gave `a presentation titled "Python and Unicode" (PDF slides) |
| 533 |
<https://downloads.egenix.com/python/Unicode-EPC2002-Talk.pdf>`_ at |
| 534 |
EuroPython 2002. The slides are an excellent overview of the design of Python |
| 535 |
2's Unicode features (where the Unicode string type is called ``unicode`` and |
| 536 |
literals start with ``u``). |
| 537 |
|
| 538 |
|
| 539 |
Reading and Writing Unicode Data |
| 540 |
================================ |
| 541 |
|
| 542 |
Once you've written some code that works with Unicode data, the next problem is |
| 543 |
input/output. How do you get Unicode strings into your program, and how do you |
| 544 |
convert Unicode into a form suitable for storage or transmission? |
| 545 |
|
| 546 |
It's possible that you may not need to do anything depending on your input |
| 547 |
sources and output destinations; you should check whether the libraries used in |
| 548 |
your application support Unicode natively. XML parsers often return Unicode |
| 549 |
data, for example. Many relational databases also support Unicode-valued |
| 550 |
columns and can return Unicode values from an SQL query. |
| 551 |
|
| 552 |
Unicode data is usually converted to a particular encoding before it gets |
| 553 |
written to disk or sent over a socket. It's possible to do all the work |
| 554 |
yourself: open a file, read an 8-bit bytes object from it, and convert the bytes |
| 555 |
with ``bytes.decode(encoding)``. However, the manual approach is not recommended. |
| 556 |
|
| 557 |
One problem is the multi-byte nature of encodings; one Unicode character can be |
| 558 |
represented by several bytes. If you want to read the file in arbitrary-sized |
| 559 |
chunks (say, 1024 or 4096 bytes), you need to write error-handling code to catch the case |
| 560 |
where only part of the bytes encoding a single Unicode character are read at the |
| 561 |
end of a chunk. One solution would be to read the entire file into memory and |
| 562 |
then perform the decoding, but that prevents you from working with files that |
| 563 |
are extremely large; if you need to read a 2 GiB file, you need 2 GiB of RAM. |
| 564 |
(More, really, since for at least a moment you'd need to have both the encoded |
| 565 |
string and its Unicode version in memory.) |
| 566 |
|
| 567 |
The solution would be to use the low-level decoding interface to catch the case |
| 568 |
of partial coding sequences. The work of implementing this has already been |
| 569 |
done for you: the built-in :func:`open` function can return a file-like object |
| 570 |
that assumes the file's contents are in a specified encoding and accepts Unicode |
| 571 |
parameters for methods such as :meth:`~io.TextIOBase.read` and |
| 572 |
:meth:`~io.TextIOBase.write`. This works through :func:`open`\'s *encoding* and |
| 573 |
*errors* parameters which are interpreted just like those in :meth:`str.encode` |
| 574 |
and :meth:`bytes.decode`. |
| 575 |
|
| 576 |
Reading Unicode from a file is therefore simple:: |
| 577 |
|
| 578 |
with open('unicode.txt', encoding='utf-8') as f: |
| 579 |
for line in f: |
| 580 |
print(repr(line)) |
| 581 |
|
| 582 |
It's also possible to open files in update mode, allowing both reading and |
| 583 |
writing:: |
| 584 |
|
| 585 |
with open('test', encoding='utf-8', mode='w+') as f: |
| 586 |
f.write('\u4500 blah blah blah\n') |
| 587 |
f.seek(0) |
| 588 |
print(repr(f.readline()[:1])) |
| 589 |
|
| 590 |
The Unicode character ``U+FEFF`` is used as a byte-order mark (BOM), and is often |
| 591 |
written as the first character of a file in order to assist with autodetection |
| 592 |
of the file's byte ordering. Some encodings, such as UTF-16, expect a BOM to be |
| 593 |
present at the start of a file; when such an encoding is used, the BOM will be |
| 594 |
automatically written as the first character and will be silently dropped when |
| 595 |
the file is read. There are variants of these encodings, such as 'utf-16-le' |
| 596 |
and 'utf-16-be' for little-endian and big-endian encodings, that specify one |
| 597 |
particular byte ordering and don't skip the BOM. |
| 598 |
|
| 599 |
In some areas, it is also convention to use a "BOM" at the start of UTF-8 |
| 600 |
encoded files; the name is misleading since UTF-8 is not byte-order dependent. |
| 601 |
The mark simply announces that the file is encoded in UTF-8. For reading such |
| 602 |
files, use the 'utf-8-sig' codec to automatically skip the mark if present. |
| 603 |
|
| 604 |
|
| 605 |
Unicode filenames |
| 606 |
----------------- |
| 607 |
|
| 608 |
Most of the operating systems in common use today support filenames |
| 609 |
that contain arbitrary Unicode characters. Usually this is |
| 610 |
implemented by converting the Unicode string into some encoding that |
| 611 |
varies depending on the system. Today Python is converging on using |
| 612 |
UTF-8: Python on MacOS has used UTF-8 for several versions, and Python |
| 613 |
3.6 switched to using UTF-8 on Windows as well. On Unix systems, |
| 614 |
there will only be a :term:`filesystem encoding <filesystem encoding and error |
| 615 |
handler>`. if you've set the ``LANG`` or ``LC_CTYPE`` environment variables; if |
| 616 |
you haven't, the default encoding is again UTF-8. |
| 617 |
|
| 618 |
The :func:`sys.getfilesystemencoding` function returns the encoding to use on |
| 619 |
your current system, in case you want to do the encoding manually, but there's |
| 620 |
not much reason to bother. When opening a file for reading or writing, you can |
| 621 |
usually just provide the Unicode string as the filename, and it will be |
| 622 |
automatically converted to the right encoding for you:: |
| 623 |
|
| 624 |
filename = 'filename\u4500abc' |
| 625 |
with open(filename, 'w') as f: |
| 626 |
f.write('blah\n') |
| 627 |
|
| 628 |
Functions in the :mod:`os` module such as :func:`os.stat` will also accept Unicode |
| 629 |
filenames. |
| 630 |
|
| 631 |
The :func:`os.listdir` function returns filenames, which raises an issue: should it return |
| 632 |
the Unicode version of filenames, or should it return bytes containing |
| 633 |
the encoded versions? :func:`os.listdir` can do both, depending on whether you |
| 634 |
provided the directory path as bytes or a Unicode string. If you pass a |
| 635 |
Unicode string as the path, filenames will be decoded using the filesystem's |
| 636 |
encoding and a list of Unicode strings will be returned, while passing a byte |
| 637 |
path will return the filenames as bytes. For example, |
| 638 |
assuming the default :term:`filesystem encoding <filesystem encoding and error |
| 639 |
handler>` is UTF-8, running the following program:: |
| 640 |
|
| 641 |
fn = 'filename\u4500abc' |
| 642 |
f = open(fn, 'w') |
| 643 |
f.close() |
| 644 |
|
| 645 |
import os |
| 646 |
print(os.listdir(b'.')) |
| 647 |
print(os.listdir('.')) |
| 648 |
|
| 649 |
will produce the following output: |
| 650 |
|
| 651 |
.. code-block:: shell-session |
| 652 |
|
| 653 |
$ python listdir-test.py |
| 654 |
[b'filename\xe4\x94\x80abc', ...] |
| 655 |
['filename\u4500abc', ...] |
| 656 |
|
| 657 |
The first list contains UTF-8-encoded filenames, and the second list contains |
| 658 |
the Unicode versions. |
| 659 |
|
| 660 |
Note that on most occasions, you should can just stick with using |
| 661 |
Unicode with these APIs. The bytes APIs should only be used on |
| 662 |
systems where undecodable file names can be present; that's |
| 663 |
pretty much only Unix systems now. |
| 664 |
|
| 665 |
|
| 666 |
Tips for Writing Unicode-aware Programs |
| 667 |
--------------------------------------- |
| 668 |
|
| 669 |
This section provides some suggestions on writing software that deals with |
| 670 |
Unicode. |
| 671 |
|
| 672 |
The most important tip is: |
| 673 |
|
| 674 |
Software should only work with Unicode strings internally, decoding the input |
| 675 |
data as soon as possible and encoding the output only at the end. |
| 676 |
|
| 677 |
If you attempt to write processing functions that accept both Unicode and byte |
| 678 |
strings, you will find your program vulnerable to bugs wherever you combine the |
| 679 |
two different kinds of strings. There is no automatic encoding or decoding: if |
| 680 |
you do e.g. ``str + bytes``, a :exc:`TypeError` will be raised. |
| 681 |
|
| 682 |
When using data coming from a web browser or some other untrusted source, a |
| 683 |
common technique is to check for illegal characters in a string before using the |
| 684 |
string in a generated command line or storing it in a database. If you're doing |
| 685 |
this, be careful to check the decoded string, not the encoded bytes data; |
| 686 |
some encodings may have interesting properties, such as not being bijective |
| 687 |
or not being fully ASCII-compatible. This is especially true if the input |
| 688 |
data also specifies the encoding, since the attacker can then choose a |
| 689 |
clever way to hide malicious text in the encoded bytestream. |
| 690 |
|
| 691 |
|
| 692 |
Converting Between File Encodings |
| 693 |
''''''''''''''''''''''''''''''''' |
| 694 |
|
| 695 |
The :class:`~codecs.StreamRecoder` class can transparently convert between |
| 696 |
encodings, taking a stream that returns data in encoding #1 |
| 697 |
and behaving like a stream returning data in encoding #2. |
| 698 |
|
| 699 |
For example, if you have an input file *f* that's in Latin-1, you |
| 700 |
can wrap it with a :class:`~codecs.StreamRecoder` to return bytes encoded in |
| 701 |
UTF-8:: |
| 702 |
|
| 703 |
new_f = codecs.StreamRecoder(f, |
| 704 |
# en/decoder: used by read() to encode its results and |
| 705 |
# by write() to decode its input. |
| 706 |
codecs.getencoder('utf-8'), codecs.getdecoder('utf-8'), |
| 707 |
|
| 708 |
# reader/writer: used to read and write to the stream. |
| 709 |
codecs.getreader('latin-1'), codecs.getwriter('latin-1') ) |
| 710 |
|
| 711 |
|
| 712 |
Files in an Unknown Encoding |
| 713 |
'''''''''''''''''''''''''''' |
| 714 |
|
| 715 |
What can you do if you need to make a change to a file, but don't know |
| 716 |
the file's encoding? If you know the encoding is ASCII-compatible and |
| 717 |
only want to examine or modify the ASCII parts, you can open the file |
| 718 |
with the ``surrogateescape`` error handler:: |
| 719 |
|
| 720 |
with open(fname, 'r', encoding="ascii", errors="surrogateescape") as f: |
| 721 |
data = f.read() |
| 722 |
|
| 723 |
# make changes to the string 'data' |
| 724 |
|
| 725 |
with open(fname + '.new', 'w', |
| 726 |
encoding="ascii", errors="surrogateescape") as f: |
| 727 |
f.write(data) |
| 728 |
|
| 729 |
The ``surrogateescape`` error handler will decode any non-ASCII bytes |
| 730 |
as code points in a special range running from U+DC80 to |
| 731 |
U+DCFF. These code points will then turn back into the |
| 732 |
same bytes when the ``surrogateescape`` error handler is used to |
| 733 |
encode the data and write it back out. |
| 734 |
|
| 735 |
|
| 736 |
References |
| 737 |
---------- |
| 738 |
|
| 739 |
One section of `Mastering Python 3 Input/Output |
| 740 |
<https://pyvideo.org/video/289/pycon-2010--mastering-python-3-i-o>`_, |
| 741 |
a PyCon 2010 talk by David Beazley, discusses text processing and binary data handling. |
| 742 |
|
| 743 |
The `PDF slides for Marc-André Lemburg's presentation "Writing Unicode-aware |
| 744 |
Applications in Python" |
| 745 |
<https://downloads.egenix.com/python/LSM2005-Developing-Unicode-aware-applications-in-Python.pdf>`_ |
| 746 |
discuss questions of character encodings as well as how to internationalize |
| 747 |
and localize an application. These slides cover Python 2.x only. |
| 748 |
|
| 749 |
`The Guts of Unicode in Python |
| 750 |
<https://pyvideo.org/video/1768/the-guts-of-unicode-in-python>`_ |
| 751 |
is a PyCon 2013 talk by Benjamin Peterson that discusses the internal Unicode |
| 752 |
representation in Python 3.3. |
| 753 |
|
| 754 |
|
| 755 |
Acknowledgements |
| 756 |
================ |
| 757 |
|
| 758 |
The initial draft of this document was written by Andrew Kuchling. |
| 759 |
It has since been revised further by Alexander Belopolsky, Georg Brandl, |
| 760 |
Andrew Kuchling, and Ezio Melotti. |
| 761 |
|
| 762 |
Thanks to the following people who have noted errors or offered |
| 763 |
suggestions on this article: Éric Araujo, Nicholas Bastin, Nick |
| 764 |
Coghlan, Marius Gedminas, Kent Johnson, Ken Krugler, Marc-André |
| 765 |
Lemburg, Martin von Löwis, Terry J. Reedy, Serhiy Storchaka, |
| 766 |
Eryk Sun, Chad Whitacre, Graham Wideman. |