Debian Codegraph @

Doc/howto/unicode.rst

raw
1 .. _unicode-howto:
2
3 *****************
4 Unicode HOWTO
5 *****************
6
7 :Release: 1.12
8
9 This HOWTO discusses Python's support for the Unicode specification
10 for representing textual data, and explains various problems that
11 people commonly encounter when trying to work with Unicode.
12
13
14 Introduction to Unicode
15 =======================
16
17 Definitions
18 -----------
19
20 Today's programs need to be able to handle a wide variety of
21 characters. Applications are often internationalized to display
22 messages and output in a variety of user-selectable languages; the
23 same program might need to output an error message in English, French,
24 Japanese, Hebrew, or Russian. Web content can be written in any of
25 these languages and can also include a variety of emoji symbols.
26 Python's string type uses the Unicode Standard for representing
27 characters, which lets Python programs work with all these different
28 possible characters.
29
30 Unicode (https://www.unicode.org/) is a specification that aims to
31 list every character used by human languages and give each character
32 its own unique code. The Unicode specifications are continually
33 revised and updated to add new languages and symbols.
34
35 A **character** is the smallest possible component of a text. 'A', 'B', 'C',
36 etc., are all different characters. So are 'È' and 'Í'. Characters vary
37 depending on the language or context you're talking
38 about. For example, there's a character for "Roman Numeral One", 'Ⅰ', that's
39 separate from the uppercase letter 'I'. They'll usually look the same,
40 but these are two different characters that have different meanings.
41
42 The Unicode standard describes how characters are represented by
43 **code points**. A code point value is an integer in the range 0 to
44 0x10FFFF (about 1.1 million values, the
45 `actual number assigned <https://www.unicode.org/versions/latest/#Summary>`_
46 is less than that). In the standard and in this document, a code point is written
47 using the notation ``U+265E`` to mean the character with value
48 ``0x265e`` (9,822 in decimal).
49
50 The Unicode standard contains a lot of tables listing characters and
51 their corresponding code points:
52
53 .. code-block:: none
54
55 0061 'a'; LATIN SMALL LETTER A
56 0062 'b'; LATIN SMALL LETTER B
57 0063 'c'; LATIN SMALL LETTER C
58 ...
59 007B '{'; LEFT CURLY BRACKET
60 ...
61 2167 'Ⅷ'; ROMAN NUMERAL EIGHT
62 2168 'Ⅸ'; ROMAN NUMERAL NINE
63 ...
64 265E '♞'; BLACK CHESS KNIGHT
65 265F '♟'; BLACK CHESS PAWN
66 ...
67 1F600 '😀'; GRINNING FACE
68 1F609 '😉'; WINKING FACE
69 ...
70
71 Strictly, these definitions imply that it's meaningless to say 'this is
72 character ``U+265E``'. ``U+265E`` is a code point, which represents some particular
73 character; in this case, it represents the character 'BLACK CHESS KNIGHT',
74 '♞'. In
75 informal contexts, this distinction between code points and characters will
76 sometimes be forgotten.
77
78 A character is represented on a screen or on paper by a set of graphical
79 elements that's called a **glyph**. The glyph for an uppercase A, for example,
80 is two diagonal strokes and a horizontal stroke, though the exact details will
81 depend on the font being used. Most Python code doesn't need to worry about
82 glyphs; figuring out the correct glyph to display is generally the job of a GUI
83 toolkit or a terminal's font renderer.
84
85
86 Encodings
87 ---------
88
89 To summarize the previous section: a Unicode string is a sequence of
90 code points, which are numbers from 0 through ``0x10FFFF`` (1,114,111
91 decimal). This sequence of code points needs to be represented in
92 memory as a set of **code units**, and **code units** are then mapped
93 to 8-bit bytes. The rules for translating a Unicode string into a
94 sequence of bytes are called a **character encoding**, or just
95 an **encoding**.
96
97 The first encoding you might think of is using 32-bit integers as the
98 code unit, and then using the CPU's representation of 32-bit integers.
99 In this representation, the string "Python" might look like this:
100
101 .. code-block:: none
102
103 P y t h o n
104 0x50 00 00 00 79 00 00 00 74 00 00 00 68 00 00 00 6f 00 00 00 6e 00 00 00
105 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
106
107 This representation is straightforward but using it presents a number of
108 problems.
109
110 1. It's not portable; different processors order the bytes differently.
111
112 2. It's very wasteful of space. In most texts, the majority of the code points
113 are less than 127, or less than 255, so a lot of space is occupied by ``0x00``
114 bytes. The above string takes 24 bytes compared to the 6 bytes needed for an
115 ASCII representation. Increased RAM usage doesn't matter too much (desktop
116 computers have gigabytes of RAM, and strings aren't usually that large), but
117 expanding our usage of disk and network bandwidth by a factor of 4 is
118 intolerable.
119
120 3. It's not compatible with existing C functions such as ``strlen()``, so a new
121 family of wide string functions would need to be used.
122
123 Therefore this encoding isn't used very much, and people instead choose other
124 encodings that are more efficient and convenient, such as UTF-8.
125
126 UTF-8 is one of the most commonly used encodings, and Python often
127 defaults to using it. UTF stands for "Unicode Transformation Format",
128 and the '8' means that 8-bit values are used in the encoding. (There
129 are also UTF-16 and UTF-32 encodings, but they are less frequently
130 used than UTF-8.) UTF-8 uses the following rules:
131
132 1. If the code point is < 128, it's represented by the corresponding byte value.
133 2. If the code point is >= 128, it's turned into a sequence of two, three, or
134 four bytes, where each byte of the sequence is between 128 and 255.
135
136 UTF-8 has several convenient properties:
137
138 1. It can handle any Unicode code point.
139 2. A Unicode string is turned into a sequence of bytes that contains embedded
140 zero bytes only where they represent the null character (U+0000). This means
141 that UTF-8 strings can be processed by C functions such as ``strcpy()`` and sent
142 through protocols that can't handle zero bytes for anything other than
143 end-of-string markers.
144 3. A string of ASCII text is also valid UTF-8 text.
145 4. UTF-8 is fairly compact; the majority of commonly used characters can be
146 represented with one or two bytes.
147 5. If bytes are corrupted or lost, it's possible to determine the start of the
148 next UTF-8-encoded code point and resynchronize. It's also unlikely that
149 random 8-bit data will look like valid UTF-8.
150 6. UTF-8 is a byte oriented encoding. The encoding specifies that each
151 character is represented by a specific sequence of one or more bytes. This
152 avoids the byte-ordering issues that can occur with integer and word oriented
153 encodings, like UTF-16 and UTF-32, where the sequence of bytes varies depending
154 on the hardware on which the string was encoded.
155
156
157 References
158 ----------
159
160 The `Unicode Consortium site <https://www.unicode.org>`_ has character charts, a
161 glossary, and PDF versions of the Unicode specification. Be prepared for some
162 difficult reading. `A chronology <https://www.unicode.org/history/>`_ of the
163 origin and development of Unicode is also available on the site.
164
165 On the Computerphile Youtube channel, Tom Scott briefly
166 `discusses the history of Unicode and UTF-8 <https://www.youtube.com/watch?v=MijmeoH9LT4>`_
167 (9 minutes 36 seconds).
168
169 To help understand the standard, Jukka Korpela has written `an introductory
170 guide <https://jkorpela.fi/unicode/guide.html>`_ to reading the
171 Unicode character tables.
172
173 Another `good introductory article <https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/>`_
174 was written by Joel Spolsky.
175 If this introduction didn't make things clear to you, you should try
176 reading this alternate article before continuing.
177
178 Wikipedia entries are often helpful; see the entries for "`character encoding
179 <https://en.wikipedia.org/wiki/Character_encoding>`_" and `UTF-8
180 <https://en.wikipedia.org/wiki/UTF-8>`_, for example.
181
182
183 Python's Unicode Support
184 ========================
185
186 Now that you've learned the rudiments of Unicode, we can look at Python's
187 Unicode features.
188
189 The String Type
190 ---------------
191
192 Since Python 3.0, the language's :class:`str` type contains Unicode
193 characters, meaning any string created using ``"unicode rocks!"``, ``'unicode
194 rocks!'``, or the triple-quoted string syntax is stored as Unicode.
195
196 The default encoding for Python source code is UTF-8, so you can simply
197 include a Unicode character in a string literal::
198
199 try:
200 with open('/tmp/input.txt', 'r') as f:
201 ...
202 except OSError:
203 # 'File not found' error message.
204 print("Fichier non trouvé")
205
206 Side note: Python 3 also supports using Unicode characters in identifiers::
207
208 répertoire = "/tmp/records.log"
209 with open(répertoire, "w") as f:
210 f.write("test\n")
211
212 If you can't enter a particular character in your editor or want to
213 keep the source code ASCII-only for some reason, you can also use
214 escape sequences in string literals. (Depending on your system,
215 you may see the actual capital-delta glyph instead of a \u escape.) ::
216
217 >>> "\N{GREEK CAPITAL LETTER DELTA}" # Using the character name
218 '\u0394'
219 >>> "\u0394" # Using a 16-bit hex value
220 '\u0394'
221 >>> "\U00000394" # Using a 32-bit hex value
222 '\u0394'
223
224 In addition, one can create a string using the :func:`~bytes.decode` method of
225 :class:`bytes`. This method takes an *encoding* argument, such as ``UTF-8``,
226 and optionally an *errors* argument.
227
228 The *errors* argument specifies the response when the input string can't be
229 converted according to the encoding's rules. Legal values for this argument are
230 ``'strict'`` (raise a :exc:`UnicodeDecodeError` exception), ``'replace'`` (use
231 ``U+FFFD``, ``REPLACEMENT CHARACTER``), ``'ignore'`` (just leave the
232 character out of the Unicode result), or ``'backslashreplace'`` (inserts a
233 ``\xNN`` escape sequence).
234 The following examples show the differences::
235
236 >>> b'\x80abc'.decode("utf-8", "strict") #doctest: +NORMALIZE_WHITESPACE
237 Traceback (most recent call last):
238 ...
239 UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0:
240 invalid start byte
241 >>> b'\x80abc'.decode("utf-8", "replace")
242 '\ufffdabc'
243 >>> b'\x80abc'.decode("utf-8", "backslashreplace")
244 '\\x80abc'
245 >>> b'\x80abc'.decode("utf-8", "ignore")
246 'abc'
247
248 Encodings are specified as strings containing the encoding's name. Python
249 comes with roughly 100 different encodings; see the Python Library Reference at
250 :ref:`standard-encodings` for a list. Some encodings have multiple names; for
251 example, ``'latin-1'``, ``'iso_8859_1'`` and ``'8859``' are all synonyms for
252 the same encoding.
253
254 One-character Unicode strings can also be created with the :func:`chr`
255 built-in function, which takes integers and returns a Unicode string of length 1
256 that contains the corresponding code point. The reverse operation is the
257 built-in :func:`ord` function that takes a one-character Unicode string and
258 returns the code point value::
259
260 >>> chr(57344)
261 '\ue000'
262 >>> ord('\ue000')
263 57344
264
265 Converting to Bytes
266 -------------------
267
268 The opposite method of :meth:`bytes.decode` is :meth:`str.encode`,
269 which returns a :class:`bytes` representation of the Unicode string, encoded in the
270 requested *encoding*.
271
272 The *errors* parameter is the same as the parameter of the
273 :meth:`~bytes.decode` method but supports a few more possible handlers. As well as
274 ``'strict'``, ``'ignore'``, and ``'replace'`` (which in this case
275 inserts a question mark instead of the unencodable character), there is
276 also ``'xmlcharrefreplace'`` (inserts an XML character reference),
277 ``backslashreplace`` (inserts a ``\uNNNN`` escape sequence) and
278 ``namereplace`` (inserts a ``\N{...}`` escape sequence).
279
280 The following example shows the different results::
281
282 >>> u = chr(40960) + 'abcd' + chr(1972)
283 >>> u.encode('utf-8')
284 b'\xea\x80\x80abcd\xde\xb4'
285 >>> u.encode('ascii') #doctest: +NORMALIZE_WHITESPACE
286 Traceback (most recent call last):
287 ...
288 UnicodeEncodeError: 'ascii' codec can't encode character '\ua000' in
289 position 0: ordinal not in range(128)
290 >>> u.encode('ascii', 'ignore')
291 b'abcd'
292 >>> u.encode('ascii', 'replace')
293 b'?abcd?'
294 >>> u.encode('ascii', 'xmlcharrefreplace')
295 b'&#40960;abcd&#1972;'
296 >>> u.encode('ascii', 'backslashreplace')
297 b'\\ua000abcd\\u07b4'
298 >>> u.encode('ascii', 'namereplace')
299 b'\\N{YI SYLLABLE IT}abcd\\u07b4'
300
301 The low-level routines for registering and accessing the available
302 encodings are found in the :mod:`codecs` module. Implementing new
303 encodings also requires understanding the :mod:`codecs` module.
304 However, the encoding and decoding functions returned by this module
305 are usually more low-level than is comfortable, and writing new encodings
306 is a specialized task, so the module won't be covered in this HOWTO.
307
308
309 Unicode Literals in Python Source Code
310 --------------------------------------
311
312 In Python source code, specific Unicode code points can be written using the
313 ``\u`` escape sequence, which is followed by four hex digits giving the code
314 point. The ``\U`` escape sequence is similar, but expects eight hex digits,
315 not four::
316
317 >>> s = "a\xac\u1234\u20ac\U00008000"
318 ... # ^^^^ two-digit hex escape
319 ... # ^^^^^^ four-digit Unicode escape
320 ... # ^^^^^^^^^^ eight-digit Unicode escape
321 >>> [ord(c) for c in s]
322 [97, 172, 4660, 8364, 32768]
323
324 Using escape sequences for code points greater than 127 is fine in small doses,
325 but becomes an annoyance if you're using many accented characters, as you would
326 in a program with messages in French or some other accent-using language. You
327 can also assemble strings using the :func:`chr` built-in function, but this is
328 even more tedious.
329
330 Ideally, you'd want to be able to write literals in your language's natural
331 encoding. You could then edit Python source code with your favorite editor
332 which would display the accented characters naturally, and have the right
333 characters used at runtime.
334
335 Python supports writing source code in UTF-8 by default, but you can use almost
336 any encoding if you declare the encoding being used. This is done by including
337 a special comment as either the first or second line of the source file::
338
339 #!/usr/bin/env python
340 # -*- coding: latin-1 -*-
341
342 u = 'abcdé'
343 print(ord(u[-1]))
344
345 The syntax is inspired by Emacs's notation for specifying variables local to a
346 file. Emacs supports many different variables, but Python only supports
347 'coding'. The ``-*-`` symbols indicate to Emacs that the comment is special;
348 they have no significance to Python but are a convention. Python looks for
349 ``coding: name`` or ``coding=name`` in the comment.
350
351 If you don't include such a comment, the default encoding used will be UTF-8 as
352 already mentioned. See also :pep:`263` for more information.
353
354
355 .. _unicode-properties:
356
357 Unicode Properties
358 ------------------
359
360 The Unicode specification includes a database of information about
361 code points. For each defined code point, the information includes
362 the character's name, its category, the numeric value if applicable
363 (for characters representing numeric concepts such as the Roman
364 numerals, fractions such as one-third and four-fifths, etc.). There
365 are also display-related properties, such as how to use the code point
366 in bidirectional text.
367
368 The following program displays some information about several characters, and
369 prints the numeric value of one particular character::
370
371 import unicodedata
372
373 u = chr(233) + chr(0x0bf2) + chr(3972) + chr(6000) + chr(13231)
374
375 for i, c in enumerate(u):
376 print(i, '%04x' % ord(c), unicodedata.category(c), end=" ")
377 print(unicodedata.name(c))
378
379 # Get numeric value of second character
380 print(unicodedata.numeric(u[1]))
381
382 When run, this prints:
383
384 .. code-block:: none
385
386 0 00e9 Ll LATIN SMALL LETTER E WITH ACUTE
387 1 0bf2 No TAMIL NUMBER ONE THOUSAND
388 2 0f84 Mn TIBETAN MARK HALANTA
389 3 1770 Lo TAGBANWA LETTER SA
390 4 33af So SQUARE RAD OVER S SQUARED
391 1000.0
392
393 The category codes are abbreviations describing the nature of the character.
394 These are grouped into categories such as "Letter", "Number", "Punctuation", or
395 "Symbol", which in turn are broken up into subcategories. To take the codes
396 from the above output, ``'Ll'`` means 'Letter, lowercase', ``'No'`` means
397 "Number, other", ``'Mn'`` is "Mark, nonspacing", and ``'So'`` is "Symbol,
398 other". See
399 `the General Category Values section of the Unicode Character Database documentation <https://www.unicode.org/reports/tr44/#General_Category_Values>`_ for a
400 list of category codes.
401
402
403 Comparing Strings
404 -----------------
405
406 Unicode adds some complication to comparing strings, because the same
407 set of characters can be represented by different sequences of code
408 points. For example, a letter like 'ê' can be represented as a single
409 code point U+00EA, or as U+0065 U+0302, which is the code point for
410 'e' followed by a code point for 'COMBINING CIRCUMFLEX ACCENT'. These
411 will produce the same output when printed, but one is a string of
412 length 1 and the other is of length 2.
413
414 One tool for a case-insensitive comparison is the
415 :meth:`~str.casefold` string method that converts a string to a
416 case-insensitive form following an algorithm described by the Unicode
417 Standard. This algorithm has special handling for characters such as
418 the German letter 'ß' (code point U+00DF), which becomes the pair of
419 lowercase letters 'ss'.
420
421 ::
422
423 >>> street = 'Gürzenichstraße'
424 >>> street.casefold()
425 'gürzenichstrasse'
426
427 A second tool is the :mod:`unicodedata` module's
428 :func:`~unicodedata.normalize` function that converts strings to one
429 of several normal forms, where letters followed by a combining character are
430 replaced with single characters. :func:`~unicodedata.normalize` can
431 be used to perform string comparisons that won't falsely report
432 inequality if two strings use combining characters differently:
433
434 ::
435
436 import unicodedata
437
438 def compare_strs(s1, s2):
439 def NFD(s):
440 return unicodedata.normalize('NFD', s)
441
442 return NFD(s1) == NFD(s2)
443
444 single_char = 'ê'
445 multiple_chars = '\N{LATIN SMALL LETTER E}\N{COMBINING CIRCUMFLEX ACCENT}'
446 print('length of first string=', len(single_char))
447 print('length of second string=', len(multiple_chars))
448 print(compare_strs(single_char, multiple_chars))
449
450 When run, this outputs:
451
452 .. code-block:: shell-session
453
454 $ python compare-strs.py
455 length of first string= 1
456 length of second string= 2
457 True
458
459 The first argument to the :func:`~unicodedata.normalize` function is a
460 string giving the desired normalization form, which can be one of
461 'NFC', 'NFKC', 'NFD', and 'NFKD'.
462
463 The Unicode Standard also specifies how to do caseless comparisons::
464
465 import unicodedata
466
467 def compare_caseless(s1, s2):
468 def NFD(s):
469 return unicodedata.normalize('NFD', s)
470
471 return NFD(NFD(s1).casefold()) == NFD(NFD(s2).casefold())
472
473 # Example usage
474 single_char = 'ê'
475 multiple_chars = '\N{LATIN CAPITAL LETTER E}\N{COMBINING CIRCUMFLEX ACCENT}'
476
477 print(compare_caseless(single_char, multiple_chars))
478
479 This will print ``True``. (Why is :func:`!NFD` invoked twice? Because
480 there are a few characters that make :meth:`~str.casefold` return a
481 non-normalized string, so the result needs to be normalized again. See
482 section 3.13 of the Unicode Standard for a discussion and an example.)
483
484
485 Unicode Regular Expressions
486 ---------------------------
487
488 The regular expressions supported by the :mod:`re` module can be provided
489 either as bytes or strings. Some of the special character sequences such as
490 ``\d`` and ``\w`` have different meanings depending on whether
491 the pattern is supplied as bytes or a string. For example,
492 ``\d`` will match the characters ``[0-9]`` in bytes but
493 in strings will match any character that's in the ``'Nd'`` category.
494
495 The string in this example has the number 57 written in both Thai and
496 Arabic numerals::
497
498 import re
499 p = re.compile(r'\d+')
500
501 s = "Over \u0e55\u0e57 57 flavours"
502 m = p.search(s)
503 print(repr(m.group()))
504
505 When executed, ``\d+`` will match the Thai numerals and print them
506 out. If you supply the :const:`re.ASCII` flag to
507 :func:`~re.compile`, ``\d+`` will match the substring "57" instead.
508
509 Similarly, ``\w`` matches a wide variety of Unicode characters but
510 only ``[a-zA-Z0-9_]`` in bytes or if :const:`re.ASCII` is supplied,
511 and ``\s`` will match either Unicode whitespace characters or
512 ``[ \t\n\r\f\v]``.
513
514
515 References
516 ----------
517
518 .. comment should these be mentioned earlier, e.g. at the start of the "introduction to Unicode" first section?
519
520 Some good alternative discussions of Python's Unicode support are:
521
522 * `Processing Text Files in Python 3 <https://python-notes.curiousefficiency.org/en/latest/python3/text_file_processing.html>`_, by Nick Coghlan.
523 * `Pragmatic Unicode <https://nedbatchelder.com/text/unipain.html>`_, a PyCon 2012 presentation by Ned Batchelder.
524
525 The :class:`str` type is described in the Python library reference at
526 :ref:`textseq`.
527
528 The documentation for the :mod:`unicodedata` module.
529
530 The documentation for the :mod:`codecs` module.
531
532 Marc-André Lemburg gave `a presentation titled "Python and Unicode" (PDF slides)
533 <https://downloads.egenix.com/python/Unicode-EPC2002-Talk.pdf>`_ at
534 EuroPython 2002. The slides are an excellent overview of the design of Python
535 2's Unicode features (where the Unicode string type is called ``unicode`` and
536 literals start with ``u``).
537
538
539 Reading and Writing Unicode Data
540 ================================
541
542 Once you've written some code that works with Unicode data, the next problem is
543 input/output. How do you get Unicode strings into your program, and how do you
544 convert Unicode into a form suitable for storage or transmission?
545
546 It's possible that you may not need to do anything depending on your input
547 sources and output destinations; you should check whether the libraries used in
548 your application support Unicode natively. XML parsers often return Unicode
549 data, for example. Many relational databases also support Unicode-valued
550 columns and can return Unicode values from an SQL query.
551
552 Unicode data is usually converted to a particular encoding before it gets
553 written to disk or sent over a socket. It's possible to do all the work
554 yourself: open a file, read an 8-bit bytes object from it, and convert the bytes
555 with ``bytes.decode(encoding)``. However, the manual approach is not recommended.
556
557 One problem is the multi-byte nature of encodings; one Unicode character can be
558 represented by several bytes. If you want to read the file in arbitrary-sized
559 chunks (say, 1024 or 4096 bytes), you need to write error-handling code to catch the case
560 where only part of the bytes encoding a single Unicode character are read at the
561 end of a chunk. One solution would be to read the entire file into memory and
562 then perform the decoding, but that prevents you from working with files that
563 are extremely large; if you need to read a 2 GiB file, you need 2 GiB of RAM.
564 (More, really, since for at least a moment you'd need to have both the encoded
565 string and its Unicode version in memory.)
566
567 The solution would be to use the low-level decoding interface to catch the case
568 of partial coding sequences. The work of implementing this has already been
569 done for you: the built-in :func:`open` function can return a file-like object
570 that assumes the file's contents are in a specified encoding and accepts Unicode
571 parameters for methods such as :meth:`~io.TextIOBase.read` and
572 :meth:`~io.TextIOBase.write`. This works through :func:`open`\'s *encoding* and
573 *errors* parameters which are interpreted just like those in :meth:`str.encode`
574 and :meth:`bytes.decode`.
575
576 Reading Unicode from a file is therefore simple::
577
578 with open('unicode.txt', encoding='utf-8') as f:
579 for line in f:
580 print(repr(line))
581
582 It's also possible to open files in update mode, allowing both reading and
583 writing::
584
585 with open('test', encoding='utf-8', mode='w+') as f:
586 f.write('\u4500 blah blah blah\n')
587 f.seek(0)
588 print(repr(f.readline()[:1]))
589
590 The Unicode character ``U+FEFF`` is used as a byte-order mark (BOM), and is often
591 written as the first character of a file in order to assist with autodetection
592 of the file's byte ordering. Some encodings, such as UTF-16, expect a BOM to be
593 present at the start of a file; when such an encoding is used, the BOM will be
594 automatically written as the first character and will be silently dropped when
595 the file is read. There are variants of these encodings, such as 'utf-16-le'
596 and 'utf-16-be' for little-endian and big-endian encodings, that specify one
597 particular byte ordering and don't skip the BOM.
598
599 In some areas, it is also convention to use a "BOM" at the start of UTF-8
600 encoded files; the name is misleading since UTF-8 is not byte-order dependent.
601 The mark simply announces that the file is encoded in UTF-8. For reading such
602 files, use the 'utf-8-sig' codec to automatically skip the mark if present.
603
604
605 Unicode filenames
606 -----------------
607
608 Most of the operating systems in common use today support filenames
609 that contain arbitrary Unicode characters. Usually this is
610 implemented by converting the Unicode string into some encoding that
611 varies depending on the system. Today Python is converging on using
612 UTF-8: Python on MacOS has used UTF-8 for several versions, and Python
613 3.6 switched to using UTF-8 on Windows as well. On Unix systems,
614 there will only be a :term:`filesystem encoding <filesystem encoding and error
615 handler>`. if you've set the ``LANG`` or ``LC_CTYPE`` environment variables; if
616 you haven't, the default encoding is again UTF-8.
617
618 The :func:`sys.getfilesystemencoding` function returns the encoding to use on
619 your current system, in case you want to do the encoding manually, but there's
620 not much reason to bother. When opening a file for reading or writing, you can
621 usually just provide the Unicode string as the filename, and it will be
622 automatically converted to the right encoding for you::
623
624 filename = 'filename\u4500abc'
625 with open(filename, 'w') as f:
626 f.write('blah\n')
627
628 Functions in the :mod:`os` module such as :func:`os.stat` will also accept Unicode
629 filenames.
630
631 The :func:`os.listdir` function returns filenames, which raises an issue: should it return
632 the Unicode version of filenames, or should it return bytes containing
633 the encoded versions? :func:`os.listdir` can do both, depending on whether you
634 provided the directory path as bytes or a Unicode string. If you pass a
635 Unicode string as the path, filenames will be decoded using the filesystem's
636 encoding and a list of Unicode strings will be returned, while passing a byte
637 path will return the filenames as bytes. For example,
638 assuming the default :term:`filesystem encoding <filesystem encoding and error
639 handler>` is UTF-8, running the following program::
640
641 fn = 'filename\u4500abc'
642 f = open(fn, 'w')
643 f.close()
644
645 import os
646 print(os.listdir(b'.'))
647 print(os.listdir('.'))
648
649 will produce the following output:
650
651 .. code-block:: shell-session
652
653 $ python listdir-test.py
654 [b'filename\xe4\x94\x80abc', ...]
655 ['filename\u4500abc', ...]
656
657 The first list contains UTF-8-encoded filenames, and the second list contains
658 the Unicode versions.
659
660 Note that on most occasions, you should can just stick with using
661 Unicode with these APIs. The bytes APIs should only be used on
662 systems where undecodable file names can be present; that's
663 pretty much only Unix systems now.
664
665
666 Tips for Writing Unicode-aware Programs
667 ---------------------------------------
668
669 This section provides some suggestions on writing software that deals with
670 Unicode.
671
672 The most important tip is:
673
674 Software should only work with Unicode strings internally, decoding the input
675 data as soon as possible and encoding the output only at the end.
676
677 If you attempt to write processing functions that accept both Unicode and byte
678 strings, you will find your program vulnerable to bugs wherever you combine the
679 two different kinds of strings. There is no automatic encoding or decoding: if
680 you do e.g. ``str + bytes``, a :exc:`TypeError` will be raised.
681
682 When using data coming from a web browser or some other untrusted source, a
683 common technique is to check for illegal characters in a string before using the
684 string in a generated command line or storing it in a database. If you're doing
685 this, be careful to check the decoded string, not the encoded bytes data;
686 some encodings may have interesting properties, such as not being bijective
687 or not being fully ASCII-compatible. This is especially true if the input
688 data also specifies the encoding, since the attacker can then choose a
689 clever way to hide malicious text in the encoded bytestream.
690
691
692 Converting Between File Encodings
693 '''''''''''''''''''''''''''''''''
694
695 The :class:`~codecs.StreamRecoder` class can transparently convert between
696 encodings, taking a stream that returns data in encoding #1
697 and behaving like a stream returning data in encoding #2.
698
699 For example, if you have an input file *f* that's in Latin-1, you
700 can wrap it with a :class:`~codecs.StreamRecoder` to return bytes encoded in
701 UTF-8::
702
703 new_f = codecs.StreamRecoder(f,
704 # en/decoder: used by read() to encode its results and
705 # by write() to decode its input.
706 codecs.getencoder('utf-8'), codecs.getdecoder('utf-8'),
707
708 # reader/writer: used to read and write to the stream.
709 codecs.getreader('latin-1'), codecs.getwriter('latin-1') )
710
711
712 Files in an Unknown Encoding
713 ''''''''''''''''''''''''''''
714
715 What can you do if you need to make a change to a file, but don't know
716 the file's encoding? If you know the encoding is ASCII-compatible and
717 only want to examine or modify the ASCII parts, you can open the file
718 with the ``surrogateescape`` error handler::
719
720 with open(fname, 'r', encoding="ascii", errors="surrogateescape") as f:
721 data = f.read()
722
723 # make changes to the string 'data'
724
725 with open(fname + '.new', 'w',
726 encoding="ascii", errors="surrogateescape") as f:
727 f.write(data)
728
729 The ``surrogateescape`` error handler will decode any non-ASCII bytes
730 as code points in a special range running from U+DC80 to
731 U+DCFF. These code points will then turn back into the
732 same bytes when the ``surrogateescape`` error handler is used to
733 encode the data and write it back out.
734
735
736 References
737 ----------
738
739 One section of `Mastering Python 3 Input/Output
740 <https://pyvideo.org/video/289/pycon-2010--mastering-python-3-i-o>`_,
741 a PyCon 2010 talk by David Beazley, discusses text processing and binary data handling.
742
743 The `PDF slides for Marc-André Lemburg's presentation "Writing Unicode-aware
744 Applications in Python"
745 <https://downloads.egenix.com/python/LSM2005-Developing-Unicode-aware-applications-in-Python.pdf>`_
746 discuss questions of character encodings as well as how to internationalize
747 and localize an application. These slides cover Python 2.x only.
748
749 `The Guts of Unicode in Python
750 <https://pyvideo.org/video/1768/the-guts-of-unicode-in-python>`_
751 is a PyCon 2013 talk by Benjamin Peterson that discusses the internal Unicode
752 representation in Python 3.3.
753
754
755 Acknowledgements
756 ================
757
758 The initial draft of this document was written by Andrew Kuchling.
759 It has since been revised further by Alexander Belopolsky, Georg Brandl,
760 Andrew Kuchling, and Ezio Melotti.
761
762 Thanks to the following people who have noted errors or offered
763 suggestions on this article: Éric Araujo, Nicholas Bastin, Nick
764 Coghlan, Marius Gedminas, Kent Johnson, Ken Krugler, Marc-André
765 Lemburg, Martin von Löwis, Terry J. Reedy, Serhiy Storchaka,
766 Eryk Sun, Chad Whitacre, Graham Wideman.