Debian Codegraph @

Doc/library/codecs.rst

raw
1 :mod:`!codecs` --- Codec registry and base classes
2 ==================================================
3
4 .. module:: codecs
5 :synopsis: Encode and decode data and streams.
6
7 .. moduleauthor:: Marc-André Lemburg <[email protected]>
8 .. sectionauthor:: Marc-André Lemburg <[email protected]>
9 .. sectionauthor:: Martin v. Löwis <[email protected]>
10
11 **Source code:** :source:`Lib/codecs.py`
12
13 .. index::
14 single: Unicode
15 single: Codecs
16 pair: Codecs; encode
17 pair: Codecs; decode
18 single: streams
19 pair: stackable; streams
20
21 --------------
22
23 This module defines base classes for standard Python codecs (encoders and
24 decoders) and provides access to the internal Python codec registry, which
25 manages the codec and error handling lookup process. Most standard codecs
26 are :term:`text encodings <text encoding>`, which encode text to bytes (and
27 decode bytes to text), but there are also codecs provided that encode text to
28 text, and bytes to bytes. Custom codecs may encode and decode between arbitrary
29 types, but some module features are restricted to be used specifically with
30 :term:`text encodings <text encoding>` or with codecs that encode to
31 :class:`bytes`.
32
33 The module defines the following functions for encoding and decoding with
34 any codec:
35
36 .. function:: encode(obj, encoding='utf-8', errors='strict')
37
38 Encodes *obj* using the codec registered for *encoding*.
39
40 *Errors* may be given to set the desired error handling scheme. The
41 default error handler is ``'strict'`` meaning that encoding errors raise
42 :exc:`ValueError` (or a more codec specific subclass, such as
43 :exc:`UnicodeEncodeError`). Refer to :ref:`codec-base-classes` for more
44 information on codec error handling.
45
46 .. function:: decode(obj, encoding='utf-8', errors='strict')
47
48 Decodes *obj* using the codec registered for *encoding*.
49
50 *Errors* may be given to set the desired error handling scheme. The
51 default error handler is ``'strict'`` meaning that decoding errors raise
52 :exc:`ValueError` (or a more codec specific subclass, such as
53 :exc:`UnicodeDecodeError`). Refer to :ref:`codec-base-classes` for more
54 information on codec error handling.
55
56 .. function:: charmap_build(string)
57
58 Return a mapping suitable for encoding with a custom single-byte encoding.
59 Given a :class:`str` *string* of up to 256 characters representing a
60 decoding table, returns either a compact internal mapping object
61 ``EncodingMap`` or a :class:`dictionary <dict>` mapping character ordinals
62 to byte values. Raises a :exc:`TypeError` on invalid input.
63
64 The full details for each codec can also be looked up directly:
65
66 .. function:: lookup(encoding, /)
67
68 Looks up the codec info in the Python codec registry and returns a
69 :class:`CodecInfo` object as defined below.
70
71 Encodings are first looked up in the registry's cache. If not found, the list of
72 registered search functions is scanned. If no :class:`CodecInfo` object is
73 found, a :exc:`LookupError` is raised. Otherwise, the :class:`CodecInfo` object
74 is stored in the cache and returned to the caller.
75
76 .. class:: CodecInfo(encode, decode, streamreader=None, streamwriter=None, incrementalencoder=None, incrementaldecoder=None, name=None)
77
78 Codec details when looking up the codec registry. The constructor
79 arguments are stored in attributes of the same name:
80
81
82 .. attribute:: name
83
84 The name of the encoding.
85
86
87 .. attribute:: encode
88 decode
89
90 The stateless encoding and decoding functions. These must be
91 functions or methods which have the same interface as
92 the :meth:`~Codec.encode` and :meth:`~Codec.decode` methods of Codec
93 instances (see :ref:`Codec Interface <codec-objects>`).
94 The functions or methods are expected to work in a stateless mode.
95
96
97 .. attribute:: incrementalencoder
98 incrementaldecoder
99
100 Incremental encoder and decoder classes or factory functions.
101 These have to provide the interface defined by the base classes
102 :class:`IncrementalEncoder` and :class:`IncrementalDecoder`,
103 respectively. Incremental codecs can maintain state.
104
105
106 .. attribute:: streamwriter
107 streamreader
108
109 Stream writer and reader classes or factory functions. These have to
110 provide the interface defined by the base classes
111 :class:`StreamWriter` and :class:`StreamReader`, respectively.
112 Stream codecs can maintain state.
113
114 To simplify access to the various codec components, the module provides
115 these additional functions which use :func:`lookup` for the codec lookup:
116
117 .. function:: getencoder(encoding)
118
119 Look up the codec for the given encoding and return its encoder function.
120
121 Raises a :exc:`LookupError` in case the encoding cannot be found.
122
123
124 .. function:: getdecoder(encoding)
125
126 Look up the codec for the given encoding and return its decoder function.
127
128 Raises a :exc:`LookupError` in case the encoding cannot be found.
129
130
131 .. function:: getincrementalencoder(encoding)
132
133 Look up the codec for the given encoding and return its incremental encoder
134 class or factory function.
135
136 Raises a :exc:`LookupError` in case the encoding cannot be found or the codec
137 doesn't support an incremental encoder.
138
139
140 .. function:: getincrementaldecoder(encoding)
141
142 Look up the codec for the given encoding and return its incremental decoder
143 class or factory function.
144
145 Raises a :exc:`LookupError` in case the encoding cannot be found or the codec
146 doesn't support an incremental decoder.
147
148
149 .. function:: getreader(encoding)
150
151 Look up the codec for the given encoding and return its :class:`StreamReader`
152 class or factory function.
153
154 Raises a :exc:`LookupError` in case the encoding cannot be found.
155
156
157 .. function:: getwriter(encoding)
158
159 Look up the codec for the given encoding and return its :class:`StreamWriter`
160 class or factory function.
161
162 Raises a :exc:`LookupError` in case the encoding cannot be found.
163
164 Custom codecs are made available by registering a suitable codec search
165 function:
166
167 .. function:: register(search_function, /)
168
169 Register a codec search function. Search functions are expected to take one
170 argument, being the encoding name in all lower case letters with hyphens
171 and spaces converted to underscores, and return a :class:`CodecInfo` object.
172 In case a search function cannot find a given encoding, it should return
173 ``None``.
174
175 .. versionchanged:: 3.9
176 Hyphens and spaces are converted to underscore.
177
178
179 .. function:: unregister(search_function, /)
180
181 Unregister a codec search function and clear the registry's cache.
182 If the search function is not registered, do nothing.
183
184 .. versionadded:: 3.10
185
186
187 While the builtin :func:`open` and the associated :mod:`io` module are the
188 recommended approach for working with encoded text files, this module
189 provides additional utility functions and classes that allow the use of a
190 wider range of codecs when working with binary files:
191
192 .. function:: open(filename, mode='r', encoding=None, errors='strict', buffering=-1)
193
194 Open an encoded file using the given *mode* and return an instance of
195 :class:`StreamReaderWriter`, providing transparent encoding/decoding.
196 The default file mode is ``'r'``, meaning to open the file in read mode.
197
198 .. note::
199
200 If *encoding* is not ``None``, then the
201 underlying encoded files are always opened in binary mode.
202 No automatic conversion of ``'\n'`` is done on reading and writing.
203 The *mode* argument may be any binary mode acceptable to the built-in
204 :func:`open` function; the ``'b'`` is automatically added.
205
206 *encoding* specifies the encoding which is to be used for the file.
207 Any encoding that encodes to and decodes from bytes is allowed, and
208 the data types supported by the file methods depend on the codec used.
209
210 *errors* may be given to define the error handling. It defaults to ``'strict'``
211 which causes a :exc:`ValueError` to be raised in case an encoding error occurs.
212
213 *buffering* has the same meaning as for the built-in :func:`open` function.
214 It defaults to -1 which means that the default buffer size will be used.
215
216 .. versionchanged:: 3.11
217 The ``'U'`` mode has been removed.
218
219 .. deprecated:: 3.14
220
221 :func:`codecs.open` has been superseded by :func:`open`.
222
223
224 .. function:: EncodedFile(file, data_encoding, file_encoding=None, errors='strict')
225
226 Return a :class:`StreamRecoder` instance, a wrapped version of *file*
227 which provides transparent transcoding. The original file is closed
228 when the wrapped version is closed.
229
230 Data written to the wrapped file is decoded according to the given
231 *data_encoding* and then written to the original file as bytes using
232 *file_encoding*. Bytes read from the original file are decoded
233 according to *file_encoding*, and the result is encoded
234 using *data_encoding*.
235
236 If *file_encoding* is not given, it defaults to *data_encoding*.
237
238 *errors* may be given to define the error handling. It defaults to
239 ``'strict'``, which causes :exc:`ValueError` to be raised in case an encoding
240 error occurs.
241
242
243 .. function:: iterencode(iterator, encoding, errors='strict', **kwargs)
244
245 Uses an incremental encoder to iteratively encode the input provided by
246 *iterator*. *iterator* must yield :class:`str` objects.
247 This function is a :term:`generator`. The *errors* argument (as well as any
248 other keyword argument) is passed through to the incremental encoder.
249
250 This function requires that the codec accept text :class:`str` objects
251 to encode. Therefore it does not support bytes-to-bytes encoders such as
252 ``base64_codec``.
253
254
255 .. function:: iterdecode(iterator, encoding, errors='strict', **kwargs)
256
257 Uses an incremental decoder to iteratively decode the input provided by
258 *iterator*. *iterator* must yield :class:`bytes` objects.
259 This function is a :term:`generator`. The *errors* argument (as well as any
260 other keyword argument) is passed through to the incremental decoder.
261
262 This function requires that the codec accept :class:`bytes` objects
263 to decode. Therefore it does not support text-to-text encoders such as
264 ``rot_13``, although ``rot_13`` may be used equivalently with
265 :func:`iterencode`.
266
267
268 .. function:: readbuffer_encode(buffer, errors=None, /)
269
270 Return a :class:`tuple` containing the raw bytes of *buffer*, a
271 :ref:`buffer-compatible object <bufferobjects>` or :class:`str`
272 (encoded to UTF-8 before processing), and their length in bytes.
273
274 The *errors* argument is ignored.
275
276 .. code-block:: pycon
277
278 >>> codecs.readbuffer_encode(b"Zito")
279 (b'Zito', 4)
280
281
282 The module also provides the following constants which are useful for reading
283 and writing to platform dependent files:
284
285
286 .. data:: BOM
287 BOM_BE
288 BOM_LE
289 BOM_UTF8
290 BOM_UTF16
291 BOM_UTF16_BE
292 BOM_UTF16_LE
293 BOM_UTF32
294 BOM_UTF32_BE
295 BOM_UTF32_LE
296
297 These constants define various byte sequences,
298 being Unicode byte order marks (BOMs) for several encodings. They are
299 used in UTF-16 and UTF-32 data streams to indicate the byte order used,
300 and in UTF-8 as a Unicode signature. :const:`BOM_UTF16` is either
301 :const:`BOM_UTF16_BE` or :const:`BOM_UTF16_LE` depending on the platform's
302 native byte order, :const:`BOM` is an alias for :const:`BOM_UTF16`,
303 :const:`BOM_LE` for :const:`BOM_UTF16_LE` and :const:`BOM_BE` for
304 :const:`BOM_UTF16_BE`. The others represent the BOM in UTF-8 and UTF-32
305 encodings.
306
307
308 .. _codec-base-classes:
309
310 Codec Base Classes
311 ------------------
312
313 The :mod:`!codecs` module defines a set of base classes which define the
314 interfaces for working with codec objects, and can also be used as the basis
315 for custom codec implementations.
316
317 Each codec has to define four interfaces to make it usable as codec in Python:
318 stateless encoder, stateless decoder, stream reader and stream writer. The
319 stream reader and writers typically reuse the stateless encoder/decoder to
320 implement the file protocols. Codec authors also need to define how the
321 codec will handle encoding and decoding errors.
322
323
324 .. _surrogateescape:
325 .. _error-handlers:
326
327 Error Handlers
328 ^^^^^^^^^^^^^^
329
330 To simplify and standardize error handling, codecs may implement different
331 error handling schemes by accepting the *errors* string argument:
332
333 >>> 'German ß, ♬'.encode(encoding='ascii', errors='backslashreplace')
334 b'German \\xdf, \\u266c'
335 >>> 'German ß, ♬'.encode(encoding='ascii', errors='xmlcharrefreplace')
336 b'German &#223;, &#9836;'
337
338 .. index::
339 pair: strict; error handler's name
340 pair: ignore; error handler's name
341 pair: replace; error handler's name
342 pair: backslashreplace; error handler's name
343 pair: surrogateescape; error handler's name
344 single: ? (question mark); replacement character
345 single: \ (backslash); escape sequence
346 single: \x; escape sequence
347 single: \u; escape sequence
348 single: \U; escape sequence
349
350 The following error handlers can be used with all Python
351 :ref:`standard-encodings` codecs:
352
353 .. tabularcolumns:: |l|L|
354
355 +-------------------------+-----------------------------------------------+
356 | Value | Meaning |
357 +=========================+===============================================+
358 | ``'strict'`` | Raise :exc:`UnicodeError` (or a subclass), |
359 | | this is the default. Implemented in |
360 | | :func:`strict_errors`. |
361 +-------------------------+-----------------------------------------------+
362 | ``'ignore'`` | Ignore the malformed data and continue without|
363 | | further notice. Implemented in |
364 | | :func:`ignore_errors`. |
365 +-------------------------+-----------------------------------------------+
366 | ``'replace'`` | Replace with a replacement marker. On |
367 | | encoding, use ``?`` (ASCII character). On |
368 | | decoding, use ``�`` (U+FFFD, the official |
369 | | REPLACEMENT CHARACTER). Implemented in |
370 | | :func:`replace_errors`. |
371 +-------------------------+-----------------------------------------------+
372 | ``'backslashreplace'`` | Replace with backslashed escape sequences. |
373 | | On encoding, use hexadecimal form of Unicode |
374 | | code point with formats :samp:`\\x{hh}` |
375 | | :samp:`\\u{xxxx}` :samp:`\\U{xxxxxxxx}`. |
376 | | On decoding, use hexadecimal form of byte |
377 | | value with format :samp:`\\x{hh}`. |
378 | | Implemented in |
379 | | :func:`backslashreplace_errors`. |
380 +-------------------------+-----------------------------------------------+
381 | ``'surrogateescape'`` | On decoding, replace byte with individual |
382 | | surrogate code ranging from ``U+DC80`` to |
383 | | ``U+DCFF``. This code will then be turned |
384 | | back into the same byte when the |
385 | | ``'surrogateescape'`` error handler is used |
386 | | when encoding the data. (See :pep:`383` for |
387 | | more.) |
388 +-------------------------+-----------------------------------------------+
389
390 .. index::
391 pair: xmlcharrefreplace; error handler's name
392 pair: namereplace; error handler's name
393 single: \N; escape sequence
394
395 The following error handlers are only applicable to encoding (within
396 :term:`text encodings <text encoding>`):
397
398 +-------------------------+-----------------------------------------------+
399 | Value | Meaning |
400 +=========================+===============================================+
401 | ``'xmlcharrefreplace'`` | Replace with XML/HTML numeric character |
402 | | reference, which is a decimal form of Unicode |
403 | | code point with format :samp:`&#{num};`. |
404 | | Implemented in |
405 | | :func:`xmlcharrefreplace_errors`. |
406 +-------------------------+-----------------------------------------------+
407 | ``'namereplace'`` | Replace with ``\N{...}`` escape sequences, |
408 | | what appears in the braces is the Name |
409 | | property from Unicode Character Database. |
410 | | Implemented in :func:`namereplace_errors`. |
411 +-------------------------+-----------------------------------------------+
412
413 .. index::
414 pair: surrogatepass; error handler's name
415
416 In addition, the following error handler is specific to the given codecs:
417
418 +-------------------+------------------------+-------------------------------------------+
419 | Value | Codecs | Meaning |
420 +===================+========================+===========================================+
421 |``'surrogatepass'``| utf-8, utf-16, utf-32, | Allow encoding and decoding surrogate code|
422 | | utf-16-be, utf-16-le, | point (``U+D800`` - ``U+DFFF``) as normal |
423 | | utf-32-be, utf-32-le | code point. Otherwise these codecs treat |
424 | | | the presence of surrogate code point in |
425 | | | :class:`str` as an error. |
426 +-------------------+------------------------+-------------------------------------------+
427
428 .. versionadded:: 3.1
429 The ``'surrogateescape'`` and ``'surrogatepass'`` error handlers.
430
431 .. versionchanged:: 3.4
432 The ``'surrogatepass'`` error handler now works with utf-16\* and utf-32\*
433 codecs.
434
435 .. versionadded:: 3.5
436 The ``'namereplace'`` error handler.
437
438 .. versionchanged:: 3.5
439 The ``'backslashreplace'`` error handler now works with decoding and
440 translating.
441
442 The set of allowed values can be extended by registering a new named error
443 handler:
444
445 .. function:: register_error(name, error_handler, /)
446
447 Register the error handling function *error_handler* under the name *name*.
448 The *error_handler* argument will be called during encoding and decoding
449 in case of an error, when *name* is specified as the errors parameter.
450
451 For encoding, *error_handler* will be called with a :exc:`UnicodeEncodeError`
452 instance, which contains information about the location of the error. The
453 error handler must either raise this or a different exception, or return a
454 tuple with a replacement for the unencodable part of the input and a position
455 where encoding should continue. The replacement may be either :class:`str` or
456 :class:`bytes`. If the replacement is bytes, the encoder will simply copy
457 them into the output buffer. If the replacement is a string, the encoder will
458 encode the replacement. Encoding continues on original input at the
459 specified position. Negative position values will be treated as being
460 relative to the end of the input string. If the resulting position is out of
461 bound an :exc:`IndexError` will be raised.
462
463 Decoding and translating works similarly, except :exc:`UnicodeDecodeError` or
464 :exc:`UnicodeTranslateError` will be passed to the handler and that the
465 replacement from the error handler will be put into the output directly.
466
467
468 Previously registered error handlers (including the standard error handlers)
469 can be looked up by name:
470
471 .. function:: lookup_error(name, /)
472
473 Return the error handler previously registered under the name *name*.
474
475 Raises a :exc:`LookupError` in case the handler cannot be found.
476
477 The following standard error handlers are also made available as module level
478 functions:
479
480 .. function:: strict_errors(exception)
481
482 Implements the ``'strict'`` error handling.
483
484 Each encoding or decoding error raises a :exc:`UnicodeError`.
485
486
487 .. function:: ignore_errors(exception)
488
489 Implements the ``'ignore'`` error handling.
490
491 Malformed data is ignored; encoding or decoding is continued without
492 further notice.
493
494
495 .. function:: replace_errors(exception)
496
497 Implements the ``'replace'`` error handling.
498
499 Substitutes ``?`` (ASCII character) for encoding errors or ``�`` (U+FFFD,
500 the official REPLACEMENT CHARACTER) for decoding errors.
501
502
503 .. function:: backslashreplace_errors(exception)
504
505 Implements the ``'backslashreplace'`` error handling.
506
507 Malformed data is replaced by a backslashed escape sequence.
508 On encoding, use the hexadecimal form of Unicode code point with formats
509 :samp:`\\x{hh}` :samp:`\\u{xxxx}` :samp:`\\U{xxxxxxxx}`.
510 On decoding, use the hexadecimal form of
511 byte value with format :samp:`\\x{hh}`.
512
513 .. versionchanged:: 3.5
514 Works with decoding and translating.
515
516
517 .. function:: xmlcharrefreplace_errors(exception)
518
519 Implements the ``'xmlcharrefreplace'`` error handling (for encoding within
520 :term:`text encoding` only).
521
522 The unencodable character is replaced by an appropriate XML/HTML numeric
523 character reference, which is a decimal form of Unicode code point with
524 format :samp:`&#{num};` .
525
526
527 .. function:: namereplace_errors(exception)
528
529 Implements the ``'namereplace'`` error handling (for encoding within
530 :term:`text encoding` only).
531
532 The unencodable character is replaced by a ``\N{...}`` escape sequence. The
533 set of characters that appear in the braces is the Name property from
534 Unicode Character Database. For example, the German lowercase letter ``'ß'``
535 will be converted to byte sequence ``\N{LATIN SMALL LETTER SHARP S}`` .
536
537 .. versionadded:: 3.5
538
539
540 .. _codec-objects:
541
542 Stateless Encoding and Decoding
543 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
544
545 The base :class:`Codec` class defines these methods which also define the
546 function interfaces of the stateless encoder and decoder:
547
548
549 .. class:: Codec
550
551 .. method:: encode(input, errors='strict')
552
553 Encodes the object *input* and returns a tuple (output object, length consumed).
554 For instance, :term:`text encoding` converts
555 a string object to a bytes object using a particular
556 character set encoding (e.g., ``cp1252`` or ``iso-8859-1``).
557
558 The *errors* argument defines the error handling to apply.
559 It defaults to ``'strict'`` handling.
560
561 The method may not store state in the :class:`Codec` instance. Use
562 :class:`StreamWriter` for codecs which have to keep state in order to make
563 encoding efficient.
564
565 The encoder must be able to handle zero length input and return an empty object
566 of the output object type in this situation.
567
568
569 .. method:: decode(input, errors='strict')
570
571 Decodes the object *input* and returns a tuple (output object, length
572 consumed). For instance, for a :term:`text encoding`, decoding converts
573 a bytes object encoded using a particular
574 character set encoding to a string object.
575
576 For text encodings and bytes-to-bytes codecs,
577 *input* must be a bytes object or one which provides the read-only
578 buffer interface -- for example, buffer objects and memory mapped files.
579
580 The *errors* argument defines the error handling to apply.
581 It defaults to ``'strict'`` handling.
582
583 The method may not store state in the :class:`Codec` instance. Use
584 :class:`StreamReader` for codecs which have to keep state in order to make
585 decoding efficient.
586
587 The decoder must be able to handle zero length input and return an empty object
588 of the output object type in this situation.
589
590
591 Incremental Encoding and Decoding
592 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
593
594 The :class:`IncrementalEncoder` and :class:`IncrementalDecoder` classes provide
595 the basic interface for incremental encoding and decoding. Encoding/decoding the
596 input isn't done with one call to the stateless encoder/decoder function, but
597 with multiple calls to the
598 :meth:`~IncrementalEncoder.encode`/:meth:`~IncrementalDecoder.decode` method of
599 the incremental encoder/decoder. The incremental encoder/decoder keeps track of
600 the encoding/decoding process during method calls.
601
602 The joined output of calls to the
603 :meth:`~IncrementalEncoder.encode`/:meth:`~IncrementalDecoder.decode` method is
604 the same as if all the single inputs were joined into one, and this input was
605 encoded/decoded with the stateless encoder/decoder.
606
607
608 .. _incremental-encoder-objects:
609
610 IncrementalEncoder Objects
611 ~~~~~~~~~~~~~~~~~~~~~~~~~~
612
613 The :class:`IncrementalEncoder` class is used for encoding an input in multiple
614 steps. It defines the following methods which every incremental encoder must
615 define in order to be compatible with the Python codec registry.
616
617
618 .. class:: IncrementalEncoder(errors='strict')
619
620 Constructor for an :class:`IncrementalEncoder` instance.
621
622 All incremental encoders must provide this constructor interface. They are free
623 to add additional keyword arguments, but only the ones defined here are used by
624 the Python codec registry.
625
626 The :class:`IncrementalEncoder` may implement different error handling schemes
627 by providing the *errors* keyword argument. See :ref:`error-handlers` for
628 possible values.
629
630 The *errors* argument will be assigned to an attribute of the same name.
631 Assigning to this attribute makes it possible to switch between different error
632 handling strategies during the lifetime of the :class:`IncrementalEncoder`
633 object.
634
635
636 .. method:: encode(object, final=False)
637
638 Encodes *object* (taking the current state of the encoder into account)
639 and returns the resulting encoded object. If this is the last call to
640 :meth:`encode` *final* must be true (the default is false).
641
642
643 .. method:: reset()
644
645 Reset the encoder to the initial state. The output is discarded: call
646 ``.encode(object, final=True)``, passing an empty byte or text string
647 if necessary, to reset the encoder and to get the output.
648
649
650 .. method:: getstate()
651
652 Return the current state of the encoder which must be an integer. The
653 implementation should make sure that ``0`` is the most common
654 state. (States that are more complicated than integers can be converted
655 into an integer by marshaling/pickling the state and encoding the bytes
656 of the resulting string into an integer.)
657
658
659 .. method:: setstate(state)
660
661 Set the state of the encoder to *state*. *state* must be an encoder state
662 returned by :meth:`getstate`.
663
664
665 .. _incremental-decoder-objects:
666
667 IncrementalDecoder Objects
668 ~~~~~~~~~~~~~~~~~~~~~~~~~~
669
670 The :class:`IncrementalDecoder` class is used for decoding an input in multiple
671 steps. It defines the following methods which every incremental decoder must
672 define in order to be compatible with the Python codec registry.
673
674
675 .. class:: IncrementalDecoder(errors='strict')
676
677 Constructor for an :class:`IncrementalDecoder` instance.
678
679 All incremental decoders must provide this constructor interface. They are free
680 to add additional keyword arguments, but only the ones defined here are used by
681 the Python codec registry.
682
683 The :class:`IncrementalDecoder` may implement different error handling schemes
684 by providing the *errors* keyword argument. See :ref:`error-handlers` for
685 possible values.
686
687 The *errors* argument will be assigned to an attribute of the same name.
688 Assigning to this attribute makes it possible to switch between different error
689 handling strategies during the lifetime of the :class:`IncrementalDecoder`
690 object.
691
692
693 .. method:: decode(object, final=False)
694
695 Decodes *object* (taking the current state of the decoder into account)
696 and returns the resulting decoded object. If this is the last call to
697 :meth:`decode` *final* must be true (the default is false). If *final* is
698 true the decoder must decode the input completely and must flush all
699 buffers. If this isn't possible (e.g. because of incomplete byte sequences
700 at the end of the input) it must initiate error handling just like in the
701 stateless case (which might raise an exception).
702
703
704 .. method:: reset()
705
706 Reset the decoder to the initial state.
707
708
709 .. method:: getstate()
710
711 Return the current state of the decoder. This must be a tuple with two
712 items, the first must be the buffer containing the still undecoded
713 input. The second must be an integer and can be additional state
714 info. (The implementation should make sure that ``0`` is the most common
715 additional state info.) If this additional state info is ``0`` it must be
716 possible to set the decoder to the state which has no input buffered and
717 ``0`` as the additional state info, so that feeding the previously
718 buffered input to the decoder returns it to the previous state without
719 producing any output. (Additional state info that is more complicated than
720 integers can be converted into an integer by marshaling/pickling the info
721 and encoding the bytes of the resulting string into an integer.)
722
723
724 .. method:: setstate(state)
725
726 Set the state of the decoder to *state*. *state* must be a decoder state
727 returned by :meth:`getstate`.
728
729
730 Stream Encoding and Decoding
731 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
732
733
734 The :class:`StreamWriter` and :class:`StreamReader` classes provide generic
735 working interfaces which can be used to implement new encoding submodules very
736 easily. See :mod:`!encodings.utf_8` for an example of how this is done.
737
738
739 .. _stream-writer-objects:
740
741 StreamWriter Objects
742 ~~~~~~~~~~~~~~~~~~~~
743
744 The :class:`StreamWriter` class is a subclass of :class:`Codec` and defines the
745 following methods which every stream writer must define in order to be
746 compatible with the Python codec registry.
747
748
749 .. class:: StreamWriter(stream, errors='strict')
750
751 Constructor for a :class:`StreamWriter` instance.
752
753 All stream writers must provide this constructor interface. They are free to add
754 additional keyword arguments, but only the ones defined here are used by the
755 Python codec registry.
756
757 The *stream* argument must be a file-like object open for writing
758 text or binary data, as appropriate for the specific codec.
759
760 The :class:`StreamWriter` may implement different error handling schemes by
761 providing the *errors* keyword argument. See :ref:`error-handlers` for
762 the standard error handlers the underlying stream codec may support.
763
764 The *errors* argument will be assigned to an attribute of the same name.
765 Assigning to this attribute makes it possible to switch between different error
766 handling strategies during the lifetime of the :class:`StreamWriter` object.
767
768 .. method:: write(object)
769
770 Writes the object's contents encoded to the stream.
771
772
773 .. method:: writelines(list)
774
775 Writes the concatenated iterable of strings to the stream (possibly by reusing
776 the :meth:`write` method). Infinite or
777 very large iterables are not supported. The standard bytes-to-bytes codecs
778 do not support this method.
779
780
781 .. method:: reset()
782
783 Resets the codec buffers used for keeping internal state.
784
785 Calling this method should ensure that the data on the output is put into
786 a clean state that allows appending of new fresh data without having to
787 rescan the whole stream to recover state.
788
789
790 In addition to the above methods, the :class:`StreamWriter` must also inherit
791 all other methods and attributes from the underlying stream.
792
793
794 .. _stream-reader-objects:
795
796 StreamReader Objects
797 ~~~~~~~~~~~~~~~~~~~~
798
799 The :class:`StreamReader` class is a subclass of :class:`Codec` and defines the
800 following methods which every stream reader must define in order to be
801 compatible with the Python codec registry.
802
803
804 .. class:: StreamReader(stream, errors='strict')
805
806 Constructor for a :class:`StreamReader` instance.
807
808 All stream readers must provide this constructor interface. They are free to add
809 additional keyword arguments, but only the ones defined here are used by the
810 Python codec registry.
811
812 The *stream* argument must be a file-like object open for reading
813 text or binary data, as appropriate for the specific codec.
814
815 The :class:`StreamReader` may implement different error handling schemes by
816 providing the *errors* keyword argument. See :ref:`error-handlers` for
817 the standard error handlers the underlying stream codec may support.
818
819 The *errors* argument will be assigned to an attribute of the same name.
820 Assigning to this attribute makes it possible to switch between different error
821 handling strategies during the lifetime of the :class:`StreamReader` object.
822
823 The set of allowed values for the *errors* argument can be extended with
824 :func:`register_error`.
825
826
827 .. method:: read(size=-1, chars=-1, firstline=False)
828
829 Decodes data from the stream and returns the resulting object.
830
831 The *chars* argument indicates the number of decoded
832 code points or bytes to return. The :func:`read` method will
833 never return more data than requested, but it might return less,
834 if there is not enough available.
835
836 The *size* argument indicates the approximate maximum
837 number of encoded bytes or code points to read
838 for decoding. The decoder can modify this setting as
839 appropriate. The default value -1 indicates to read and decode as much as
840 possible. This parameter is intended to
841 prevent having to decode huge files in one step.
842
843 The *firstline* flag indicates that
844 it would be sufficient to only return the first
845 line, if there are decoding errors on later lines.
846
847 The method should use a greedy read strategy meaning that it should read
848 as much data as is allowed within the definition of the encoding and the
849 given size, e.g. if optional encoding endings or state markers are
850 available on the stream, these should be read too.
851
852
853 .. method:: readline(size=None, keepends=True)
854
855 Read one line from the input stream and return the decoded data.
856
857 *size*, if given, is passed as size argument to the stream's
858 :meth:`read` method.
859
860 If *keepends* is false line-endings will be stripped from the lines
861 returned.
862
863
864 .. method:: readlines(sizehint=None, keepends=True)
865
866 Read all lines available on the input stream and return them as a list of
867 lines.
868
869 Line-endings are implemented using the codec's :meth:`decode` method and
870 are included in the list entries if *keepends* is true.
871
872 *sizehint*, if given, is passed as the *size* argument to the stream's
873 :meth:`read` method.
874
875
876 .. method:: reset()
877
878 Resets the codec buffers used for keeping internal state.
879
880 Note that no stream repositioning should take place. This method is
881 primarily intended to be able to recover from decoding errors.
882
883
884 In addition to the above methods, the :class:`StreamReader` must also inherit
885 all other methods and attributes from the underlying stream.
886
887 .. _stream-reader-writer:
888
889 StreamReaderWriter Objects
890 ~~~~~~~~~~~~~~~~~~~~~~~~~~
891
892 The :class:`StreamReaderWriter` is a convenience class that allows wrapping
893 streams which work in both read and write modes.
894
895 The design is such that one can use the factory functions returned by the
896 :func:`lookup` function to construct the instance.
897
898
899 .. class:: StreamReaderWriter(stream, Reader, Writer, errors='strict')
900
901 Creates a :class:`StreamReaderWriter` instance. *stream* must be a file-like
902 object. *Reader* and *Writer* must be factory functions or classes providing the
903 :class:`StreamReader` and :class:`StreamWriter` interface resp. Error handling
904 is done in the same way as defined for the stream readers and writers.
905
906 :class:`StreamReaderWriter` instances define the combined interfaces of
907 :class:`StreamReader` and :class:`StreamWriter` classes. They inherit all other
908 methods and attributes from the underlying stream.
909
910
911 .. _stream-recoder-objects:
912
913 StreamRecoder Objects
914 ~~~~~~~~~~~~~~~~~~~~~
915
916 The :class:`StreamRecoder` translates data from one encoding to another,
917 which is sometimes useful when dealing with different encoding environments.
918
919 The design is such that one can use the factory functions returned by the
920 :func:`lookup` function to construct the instance.
921
922
923 .. class:: StreamRecoder(stream, encode, decode, Reader, Writer, errors='strict')
924
925 Creates a :class:`StreamRecoder` instance which implements a two-way conversion:
926 *encode* and *decode* work on the frontend — the data visible to
927 code calling :meth:`~StreamReader.read` and :meth:`~StreamWriter.write`,
928 while *Reader* and *Writer*
929 work on the backend — the data in *stream*.
930
931 You can use these objects to do transparent transcodings, e.g., from Latin-1
932 to UTF-8 and back.
933
934 The *stream* argument must be a file-like object.
935
936 The *encode* and *decode* arguments must
937 adhere to the :class:`Codec` interface. *Reader* and
938 *Writer* must be factory functions or classes providing objects of the
939 :class:`StreamReader` and :class:`StreamWriter` interface respectively.
940
941 Error handling is done in the same way as defined for the stream readers and
942 writers.
943
944
945 :class:`StreamRecoder` instances define the combined interfaces of
946 :class:`StreamReader` and :class:`StreamWriter` classes. They inherit all other
947 methods and attributes from the underlying stream.
948
949
950 .. _encodings-overview:
951
952 Encodings and Unicode
953 ---------------------
954
955 Strings are stored internally as sequences of code points in
956 range ``U+0000``--``U+10FFFF``. (See :pep:`393` for
957 more details about the implementation.)
958 Once a string object is used outside of CPU and memory, endianness
959 and how these arrays are stored as bytes become an issue. As with other
960 codecs, serialising a string into a sequence of bytes is known as *encoding*,
961 and recreating the string from the sequence of bytes is known as *decoding*.
962 There are a variety of different text serialisation codecs, which are
963 collectivity referred to as :term:`text encodings <text encoding>`.
964
965 The simplest text encoding (called ``'latin-1'`` or ``'iso-8859-1'``) maps
966 the code points 0--255 to the bytes ``0x0``--``0xff``, which means that a string
967 object that contains code points above ``U+00FF`` can't be encoded with this
968 codec. Doing so will raise a :exc:`UnicodeEncodeError` that looks
969 like the following (although the details of the error message may differ):
970 ``UnicodeEncodeError: 'latin-1' codec can't encode character '\u1234' in
971 position 3: ordinal not in range(256)``.
972
973 There's another group of encodings (the so called charmap encodings) that choose
974 a different subset of all Unicode code points and how these code points are
975 mapped to the bytes ``0x0``--``0xff``. To see how this is done simply open
976 e.g. :file:`encodings/cp1252.py` (which is an encoding that is used primarily on
977 Windows). There's a string constant with 256 characters that shows you which
978 character is mapped to which byte value.
979
980 All of these encodings can only encode 256 of the 1114112 code points
981 defined in Unicode. A simple and straightforward way that can store each Unicode
982 code point, is to store each code point as four consecutive bytes. There are two
983 possibilities: store the bytes in big endian or in little endian order. These
984 two encodings are called ``UTF-32-BE`` and ``UTF-32-LE`` respectively. Their
985 disadvantage is that if, for example, you use ``UTF-32-BE`` on a little endian
986 machine you will always have to swap bytes on encoding and decoding.
987 Python's ``UTF-16`` and ``UTF-32`` codecs avoid this problem by using the
988 platform's native byte order when no BOM is present.
989 Python follows prevailing platform
990 practice, so native-endian data round-trips without redundant byte swapping,
991 even though the Unicode Standard defaults to big-endian when the byte order is
992 unspecified. When these bytes are read by a CPU with a different endianness,
993 the bytes have to be swapped. To be able to detect the endianness of a
994 ``UTF-16`` or ``UTF-32`` byte sequence, a BOM ("Byte Order Mark") is used.
995 This is the Unicode character ``U+FEFF``. This character can be prepended to every
996 ``UTF-16`` or ``UTF-32`` byte sequence. The byte swapped version of this character
997 (``0xFFFE``) is an illegal character that may not appear in a Unicode text.
998 When the first character of a ``UTF-16`` or ``UTF-32`` byte sequence is
999 ``U+FFFE``, the bytes have to be swapped on decoding.
1000
1001 Unfortunately the character ``U+FEFF`` had a second purpose as
1002 a ``ZERO WIDTH NO-BREAK SPACE``: a character that has no width and doesn't allow
1003 a word to be split. It can e.g. be used to give hints to a ligature algorithm.
1004 With Unicode 4.0 using ``U+FEFF`` as a ``ZERO WIDTH NO-BREAK SPACE`` has been
1005 deprecated (with ``U+2060`` (``WORD JOINER``) assuming this role). Nevertheless
1006 Unicode software still must be able to handle ``U+FEFF`` in both roles: as a BOM
1007 it's a device to determine the storage layout of the encoded bytes, and vanishes
1008 once the byte sequence has been decoded into a string; as a ``ZERO WIDTH
1009 NO-BREAK SPACE`` it's a normal character that will be decoded like any other.
1010
1011 There's another encoding that is able to encode the full range of Unicode
1012 characters: UTF-8. UTF-8 is an 8-bit encoding, which means there are no issues
1013 with byte order in UTF-8. Each byte in a UTF-8 byte sequence consists of two
1014 parts: marker bits (the most significant bits) and payload bits. The marker bits
1015 are a sequence of zero to four ``1`` bits followed by a ``0`` bit. Unicode characters are
1016 encoded like this (with x being payload bits, which when concatenated give the
1017 Unicode character):
1018
1019 +-----------------------------------+----------------------------------------------+
1020 | Range | Encoding |
1021 +===================================+==============================================+
1022 | ``U-00000000`` ... ``U-0000007F`` | 0xxxxxxx |
1023 +-----------------------------------+----------------------------------------------+
1024 | ``U-00000080`` ... ``U-000007FF`` | 110xxxxx 10xxxxxx |
1025 +-----------------------------------+----------------------------------------------+
1026 | ``U-00000800`` ... ``U-0000FFFF`` | 1110xxxx 10xxxxxx 10xxxxxx |
1027 +-----------------------------------+----------------------------------------------+
1028 | ``U-00010000`` ... ``U-0010FFFF`` | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx |
1029 +-----------------------------------+----------------------------------------------+
1030
1031 The least significant bit of the Unicode character is the rightmost x bit.
1032
1033 As UTF-8 is an 8-bit encoding no BOM is required and any ``U+FEFF`` character in
1034 the decoded string (even if it's the first character) is treated as a ``ZERO
1035 WIDTH NO-BREAK SPACE``.
1036
1037 Without external information it's impossible to reliably determine which
1038 encoding was used for encoding a string. Each charmap encoding can
1039 decode any random byte sequence. However that's not possible with UTF-8, as
1040 UTF-8 byte sequences have a structure that doesn't allow arbitrary byte
1041 sequences. To increase the reliability with which a UTF-8 encoding can be
1042 detected, Microsoft invented a variant of UTF-8 (that Python calls
1043 ``"utf-8-sig"``) for its Notepad program: Before any of the Unicode characters
1044 is written to the file, a UTF-8 encoded BOM (which looks like this as a byte
1045 sequence: ``0xef``, ``0xbb``, ``0xbf``) is written. As it's rather improbable
1046 that any charmap encoded file starts with these byte values (which would e.g.
1047 map to
1048
1049 | LATIN SMALL LETTER I WITH DIAERESIS
1050 | RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK
1051 | INVERTED QUESTION MARK
1052
1053 in iso-8859-1), this increases the probability that a ``utf-8-sig`` encoding can be
1054 correctly guessed from the byte sequence. So here the BOM is not used to be able
1055 to determine the byte order used for generating the byte sequence, but as a
1056 signature that helps in guessing the encoding. On encoding the utf-8-sig codec
1057 will write ``0xef``, ``0xbb``, ``0xbf`` as the first three bytes to the file. On
1058 decoding ``utf-8-sig`` will skip those three bytes if they appear as the first
1059 three bytes in the file. In UTF-8, the use of the BOM is discouraged and
1060 should generally be avoided.
1061
1062
1063 .. _standard-encodings:
1064
1065 Standard Encodings
1066 ------------------
1067
1068 Python comes with a number of codecs built-in, either implemented as C functions
1069 or with dictionaries as mapping tables. The following table lists the codecs by
1070 name, together with a few common aliases, and the languages for which the
1071 encoding is likely used. Neither the list of aliases nor the list of languages
1072 is meant to be exhaustive. Notice that spelling alternatives that only differ in
1073 case or use a hyphen instead of an underscore are also valid aliases
1074 because they are equivalent when normalized by
1075 :func:`~encodings.normalize_encoding`. For example, ``'utf-8'`` is a valid
1076 alias for the ``'utf_8'`` codec.
1077
1078 .. note::
1079
1080 The below table lists the most common aliases, for a complete list
1081 refer to the source :source:`aliases.py <Lib/encodings/aliases.py>` file.
1082
1083 On Windows, ``cpXXX`` codecs are available for all code pages.
1084 But only codecs listed in the following table are guarantead to exist on
1085 other platforms.
1086
1087 .. impl-detail::
1088
1089 Some common encodings can bypass the codecs lookup machinery to
1090 improve performance. These optimization opportunities are only
1091 recognized by CPython for a limited set of (case insensitive)
1092 aliases: utf-8, utf8, latin-1, latin1, iso-8859-1, iso8859-1, mbcs
1093 (Windows only), ascii, us-ascii, utf-16, utf16, utf-32, utf32, and
1094 the same using underscores instead of dashes. Using alternative
1095 aliases for these encodings may result in slower execution.
1096
1097 .. versionchanged:: 3.6
1098 Optimization opportunity recognized for us-ascii.
1099
1100 Many of the character sets support the same languages. They vary in individual
1101 characters (e.g. whether the EURO SIGN is supported or not), and in the
1102 assignment of characters to code positions. For the European languages in
1103 particular, the following variants typically exist:
1104
1105 * an ISO 8859 codeset
1106
1107 * a Microsoft Windows code page, which is typically derived from an 8859 codeset,
1108 but replaces control characters with additional graphic characters
1109
1110 * an IBM EBCDIC code page
1111
1112 * an IBM PC code page, which is ASCII compatible
1113
1114 .. tabularcolumns:: |l|p{0.3\linewidth}|p{0.3\linewidth}|
1115
1116 +-----------------+--------------------------------+--------------------------------+
1117 | Codec | Aliases | Languages |
1118 +=================+================================+================================+
1119 | ascii | 646, us-ascii | English |
1120 +-----------------+--------------------------------+--------------------------------+
1121 | big5 | big5-tw, csbig5 | Traditional Chinese |
1122 +-----------------+--------------------------------+--------------------------------+
1123 | big5hkscs | big5-hkscs, hkscs | Traditional Chinese |
1124 +-----------------+--------------------------------+--------------------------------+
1125 | cp037 | IBM037, IBM039 | English |
1126 +-----------------+--------------------------------+--------------------------------+
1127 | cp273 | 273, IBM273, csIBM273 | German |
1128 | | | |
1129 | | | .. versionadded:: 3.4 |
1130 +-----------------+--------------------------------+--------------------------------+
1131 | cp424 | EBCDIC-CP-HE, IBM424 | Hebrew |
1132 +-----------------+--------------------------------+--------------------------------+
1133 | cp437 | 437, IBM437 | English |
1134 +-----------------+--------------------------------+--------------------------------+
1135 | cp500 | EBCDIC-CP-BE, EBCDIC-CP-CH, | Western Europe |
1136 | | IBM500 | |
1137 +-----------------+--------------------------------+--------------------------------+
1138 | cp720 | | Arabic |
1139 +-----------------+--------------------------------+--------------------------------+
1140 | cp737 | | Greek |
1141 +-----------------+--------------------------------+--------------------------------+
1142 | cp775 | IBM775 | Baltic languages |
1143 +-----------------+--------------------------------+--------------------------------+
1144 | cp850 | 850, IBM850 | Western Europe |
1145 +-----------------+--------------------------------+--------------------------------+
1146 | cp852 | 852, IBM852 | Central and Eastern Europe |
1147 +-----------------+--------------------------------+--------------------------------+
1148 | cp855 | 855, IBM855 | Belarusian, Bulgarian, |
1149 | | | Macedonian, Russian, Serbian |
1150 +-----------------+--------------------------------+--------------------------------+
1151 | cp856 | | Hebrew |
1152 +-----------------+--------------------------------+--------------------------------+
1153 | cp857 | 857, IBM857 | Turkish |
1154 +-----------------+--------------------------------+--------------------------------+
1155 | cp858 | 858, IBM858 | Western Europe |
1156 +-----------------+--------------------------------+--------------------------------+
1157 | cp860 | 860, IBM860 | Portuguese |
1158 +-----------------+--------------------------------+--------------------------------+
1159 | cp861 | 861, CP-IS, IBM861 | Icelandic |
1160 +-----------------+--------------------------------+--------------------------------+
1161 | cp862 | 862, IBM862 | Hebrew |
1162 +-----------------+--------------------------------+--------------------------------+
1163 | cp863 | 863, IBM863 | Canadian |
1164 +-----------------+--------------------------------+--------------------------------+
1165 | cp864 | IBM864 | Arabic |
1166 +-----------------+--------------------------------+--------------------------------+
1167 | cp865 | 865, IBM865 | Danish, Norwegian |
1168 +-----------------+--------------------------------+--------------------------------+
1169 | cp866 | 866, IBM866 | Russian |
1170 +-----------------+--------------------------------+--------------------------------+
1171 | cp869 | 869, CP-GR, IBM869 | Greek |
1172 +-----------------+--------------------------------+--------------------------------+
1173 | cp874 | | Thai |
1174 +-----------------+--------------------------------+--------------------------------+
1175 | cp875 | | Greek |
1176 +-----------------+--------------------------------+--------------------------------+
1177 | cp932 | 932, ms932, mskanji, ms-kanji, | Japanese |
1178 | | windows-31j | |
1179 +-----------------+--------------------------------+--------------------------------+
1180 | cp949 | 949, ms949, uhc | Korean |
1181 +-----------------+--------------------------------+--------------------------------+
1182 | cp950 | 950, ms950 | Traditional Chinese |
1183 +-----------------+--------------------------------+--------------------------------+
1184 | cp1006 | | Urdu |
1185 +-----------------+--------------------------------+--------------------------------+
1186 | cp1026 | ibm1026 | Turkish |
1187 +-----------------+--------------------------------+--------------------------------+
1188 | cp1125 | 1125, ibm1125, cp866u, ruscii | Ukrainian |
1189 | | | |
1190 | | | .. versionadded:: 3.4 |
1191 +-----------------+--------------------------------+--------------------------------+
1192 | cp1140 | ibm1140 | Western Europe |
1193 +-----------------+--------------------------------+--------------------------------+
1194 | cp1250 | windows-1250 | Central and Eastern Europe |
1195 +-----------------+--------------------------------+--------------------------------+
1196 | cp1251 | windows-1251 | Belarusian, Bulgarian, |
1197 | | | Macedonian, Russian, Serbian |
1198 +-----------------+--------------------------------+--------------------------------+
1199 | cp1252 | windows-1252 | Western Europe |
1200 +-----------------+--------------------------------+--------------------------------+
1201 | cp1253 | windows-1253 | Greek |
1202 +-----------------+--------------------------------+--------------------------------+
1203 | cp1254 | windows-1254 | Turkish |
1204 +-----------------+--------------------------------+--------------------------------+
1205 | cp1255 | windows-1255 | Hebrew |
1206 +-----------------+--------------------------------+--------------------------------+
1207 | cp1256 | windows-1256 | Arabic |
1208 +-----------------+--------------------------------+--------------------------------+
1209 | cp1257 | windows-1257 | Baltic languages |
1210 +-----------------+--------------------------------+--------------------------------+
1211 | cp1258 | windows-1258 | Vietnamese |
1212 +-----------------+--------------------------------+--------------------------------+
1213 | euc_jp | eucjp, ujis, u-jis | Japanese |
1214 +-----------------+--------------------------------+--------------------------------+
1215 | euc_jis_2004 | jisx0213, eucjis2004 | Japanese |
1216 +-----------------+--------------------------------+--------------------------------+
1217 | euc_jisx0213 | eucjisx0213 | Japanese |
1218 +-----------------+--------------------------------+--------------------------------+
1219 | euc_kr | euckr, korean, ksc5601, | Korean |
1220 | | ks_c-5601, ks_c-5601-1987, | |
1221 | | ksx1001, ks_x-1001 | |
1222 +-----------------+--------------------------------+--------------------------------+
1223 | gb2312 | chinese, csiso58gb231280, | Simplified Chinese |
1224 | | euc-cn, euccn, eucgb2312-cn, | |
1225 | | gb2312-1980, gb2312-80, | |
1226 | | iso-ir-58 | |
1227 +-----------------+--------------------------------+--------------------------------+
1228 | gbk | 936, cp936, ms936 | Unified Chinese |
1229 +-----------------+--------------------------------+--------------------------------+
1230 | gb18030 | gb18030-2000 | Unified Chinese |
1231 +-----------------+--------------------------------+--------------------------------+
1232 | hz | hzgb, hz-gb, hz-gb-2312 | Simplified Chinese |
1233 +-----------------+--------------------------------+--------------------------------+
1234 | iso2022_jp | csiso2022jp, iso2022jp, | Japanese |
1235 | | iso-2022-jp | |
1236 +-----------------+--------------------------------+--------------------------------+
1237 | iso2022_jp_1 | iso2022jp-1, iso-2022-jp-1 | Japanese |
1238 +-----------------+--------------------------------+--------------------------------+
1239 | iso2022_jp_2 | iso2022jp-2, iso-2022-jp-2 | Japanese, Korean, Simplified |
1240 | | | Chinese, Western Europe, Greek |
1241 +-----------------+--------------------------------+--------------------------------+
1242 | iso2022_jp_2004 | iso2022jp-2004, | Japanese |
1243 | | iso-2022-jp-2004 | |
1244 +-----------------+--------------------------------+--------------------------------+
1245 | iso2022_jp_3 | iso2022jp-3, iso-2022-jp-3 | Japanese |
1246 +-----------------+--------------------------------+--------------------------------+
1247 | iso2022_jp_ext | iso2022jp-ext, iso-2022-jp-ext | Japanese |
1248 +-----------------+--------------------------------+--------------------------------+
1249 | iso2022_kr | csiso2022kr, iso2022kr, | Korean |
1250 | | iso-2022-kr | |
1251 +-----------------+--------------------------------+--------------------------------+
1252 | latin_1 | iso-8859-1, iso8859-1, 8859, | Western Europe |
1253 | | cp819, latin, latin1, L1 | |
1254 +-----------------+--------------------------------+--------------------------------+
1255 | iso8859_2 | iso-8859-2, latin2, L2 | Central and Eastern Europe |
1256 +-----------------+--------------------------------+--------------------------------+
1257 | iso8859_3 | iso-8859-3, latin3, L3 | Esperanto, Maltese |
1258 +-----------------+--------------------------------+--------------------------------+
1259 | iso8859_4 | iso-8859-4, latin4, L4 | Northern Europe |
1260 +-----------------+--------------------------------+--------------------------------+
1261 | iso8859_5 | iso-8859-5, cyrillic | Belarusian, Bulgarian, |
1262 | | | Macedonian, Russian, Serbian |
1263 +-----------------+--------------------------------+--------------------------------+
1264 | iso8859_6 | iso-8859-6, arabic | Arabic |
1265 +-----------------+--------------------------------+--------------------------------+
1266 | iso8859_7 | iso-8859-7, greek, greek8 | Greek |
1267 +-----------------+--------------------------------+--------------------------------+
1268 | iso8859_8 | iso-8859-8, hebrew | Hebrew |
1269 +-----------------+--------------------------------+--------------------------------+
1270 | iso8859_9 | iso-8859-9, latin5, L5 | Turkish |
1271 +-----------------+--------------------------------+--------------------------------+
1272 | iso8859_10 | iso-8859-10, latin6, L6 | Nordic languages |
1273 +-----------------+--------------------------------+--------------------------------+
1274 | iso8859_11 | iso-8859-11, thai | Thai languages |
1275 +-----------------+--------------------------------+--------------------------------+
1276 | iso8859_13 | iso-8859-13, latin7, L7 | Baltic languages |
1277 +-----------------+--------------------------------+--------------------------------+
1278 | iso8859_14 | iso-8859-14, latin8, L8 | Celtic languages |
1279 +-----------------+--------------------------------+--------------------------------+
1280 | iso8859_15 | iso-8859-15, latin9, L9 | Western Europe |
1281 +-----------------+--------------------------------+--------------------------------+
1282 | iso8859_16 | iso-8859-16, latin10, L10 | South-Eastern Europe |
1283 +-----------------+--------------------------------+--------------------------------+
1284 | johab | cp1361, ms1361 | Korean |
1285 +-----------------+--------------------------------+--------------------------------+
1286 | koi8_r | | Russian |
1287 +-----------------+--------------------------------+--------------------------------+
1288 | koi8_t | | Tajik |
1289 | | | |
1290 | | | .. versionadded:: 3.5 |
1291 +-----------------+--------------------------------+--------------------------------+
1292 | koi8_u | | Ukrainian |
1293 +-----------------+--------------------------------+--------------------------------+
1294 | kz1048 | kz_1048, strk1048_2002, rk1048 | Kazakh |
1295 | | | |
1296 | | | .. versionadded:: 3.5 |
1297 +-----------------+--------------------------------+--------------------------------+
1298 | mac_cyrillic | maccyrillic | Belarusian, Bulgarian, |
1299 | | | Macedonian, Russian, Serbian |
1300 +-----------------+--------------------------------+--------------------------------+
1301 | mac_greek | macgreek | Greek |
1302 +-----------------+--------------------------------+--------------------------------+
1303 | mac_iceland | maciceland | Icelandic |
1304 +-----------------+--------------------------------+--------------------------------+
1305 | mac_latin2 | maclatin2, maccentraleurope, | Central and Eastern Europe |
1306 | | mac_centeuro | |
1307 +-----------------+--------------------------------+--------------------------------+
1308 | mac_roman | macroman, macintosh | Western Europe |
1309 +-----------------+--------------------------------+--------------------------------+
1310 | mac_turkish | macturkish | Turkish |
1311 +-----------------+--------------------------------+--------------------------------+
1312 | ptcp154 | csptcp154, pt154, cp154, | Kazakh |
1313 | | cyrillic-asian | |
1314 +-----------------+--------------------------------+--------------------------------+
1315 | shift_jis | csshiftjis, shiftjis, sjis, | Japanese |
1316 | | s_jis | |
1317 +-----------------+--------------------------------+--------------------------------+
1318 | shift_jis_2004 | shiftjis2004, sjis_2004, | Japanese |
1319 | | sjis2004 | |
1320 +-----------------+--------------------------------+--------------------------------+
1321 | shift_jisx0213 | shiftjisx0213, sjisx0213, | Japanese |
1322 | | s_jisx0213 | |
1323 +-----------------+--------------------------------+--------------------------------+
1324 | utf_32 | U32, utf32 | all languages |
1325 +-----------------+--------------------------------+--------------------------------+
1326 | utf_32_be | UTF-32BE | all languages |
1327 +-----------------+--------------------------------+--------------------------------+
1328 | utf_32_le | UTF-32LE | all languages |
1329 +-----------------+--------------------------------+--------------------------------+
1330 | utf_16 | U16, utf16 | all languages |
1331 +-----------------+--------------------------------+--------------------------------+
1332 | utf_16_be | UTF-16BE | all languages |
1333 +-----------------+--------------------------------+--------------------------------+
1334 | utf_16_le | UTF-16LE | all languages |
1335 +-----------------+--------------------------------+--------------------------------+
1336 | utf_7 | U7, unicode-1-1-utf-7 | all languages |
1337 +-----------------+--------------------------------+--------------------------------+
1338 | utf_8 | U8, UTF, utf8, cp65001 | all languages |
1339 +-----------------+--------------------------------+--------------------------------+
1340 | utf_8_sig | | all languages |
1341 +-----------------+--------------------------------+--------------------------------+
1342
1343 .. versionchanged:: 3.4
1344 The utf-16\* and utf-32\* encoders no longer allow surrogate code points
1345 (``U+D800``--``U+DFFF``) to be encoded.
1346 The utf-32\* decoders no longer decode
1347 byte sequences that correspond to surrogate code points.
1348
1349 .. versionchanged:: 3.8
1350 ``cp65001`` is now an alias to ``utf_8``.
1351
1352 .. versionchanged:: 3.14
1353 On Windows, ``cpXXX`` codecs are now available for all code pages.
1354
1355
1356 Python Specific Encodings
1357 -------------------------
1358
1359 A number of predefined codecs are specific to Python, so their codec names have
1360 no meaning outside Python. These are listed in the tables below based on the
1361 expected input and output types (note that while text encodings are the most
1362 common use case for codecs, the underlying codec infrastructure supports
1363 arbitrary data transforms rather than just text encodings). For asymmetric
1364 codecs, the stated meaning describes the encoding direction.
1365
1366 Text Encodings
1367 ^^^^^^^^^^^^^^
1368
1369 The following codecs provide :class:`str` to :class:`bytes` encoding and
1370 :term:`bytes-like object` to :class:`str` decoding, similar to the Unicode text
1371 encodings.
1372
1373 .. tabularcolumns:: |l|p{0.3\linewidth}|p{0.3\linewidth}|
1374
1375 +--------------------+---------+---------------------------+
1376 | Codec | Aliases | Meaning |
1377 +====================+=========+===========================+
1378 | idna | | Implement :rfc:`3490`, |
1379 | | | see also |
1380 | | | :mod:`encodings.idna`. |
1381 | | | Only ``errors='strict'`` |
1382 | | | is supported. |
1383 +--------------------+---------+---------------------------+
1384 | mbcs | ansi, | Windows only: Encode the |
1385 | | dbcs | operand according to the |
1386 | | | ANSI codepage (CP_ACP). |
1387 +--------------------+---------+---------------------------+
1388 | oem | | Windows only: Encode the |
1389 | | | operand according to the |
1390 | | | OEM codepage (CP_OEMCP). |
1391 | | | |
1392 | | | .. versionadded:: 3.6 |
1393 +--------------------+---------+---------------------------+
1394 | palmos | | Encoding of PalmOS 3.5. |
1395 +--------------------+---------+---------------------------+
1396 | punycode | | Implement :rfc:`3492`. |
1397 | | | Stateful codecs are not |
1398 | | | supported. |
1399 +--------------------+---------+---------------------------+
1400 | raw_unicode_escape | | Latin-1 encoding with |
1401 | | | :samp:`\\u{XXXX}` and |
1402 | | | :samp:`\\U{XXXXXXXX}` |
1403 | | | for other code points. |
1404 | | | Existing |
1405 | | | backslashes are not |
1406 | | | escaped in any way. |
1407 | | | It is used in the Python |
1408 | | | pickle protocol. |
1409 +--------------------+---------+---------------------------+
1410 | undefined | | This Codec should only |
1411 | | | be used for testing |
1412 | | | purposes. |
1413 | | | |
1414 | | | Raise an exception for |
1415 | | | all conversions, even |
1416 | | | empty strings. The error |
1417 | | | handler is ignored. |
1418 +--------------------+---------+---------------------------+
1419 | unicode_escape | | Encoding suitable as the |
1420 | | | contents of a Unicode |
1421 | | | literal in ASCII-encoded |
1422 | | | Python source code, |
1423 | | | except that quotes are |
1424 | | | not escaped. Decode |
1425 | | | from Latin-1 source code. |
1426 | | | Beware that Python source |
1427 | | | code actually uses UTF-8 |
1428 | | | by default. |
1429 +--------------------+---------+---------------------------+
1430
1431 .. versionchanged:: 3.8
1432 "unicode_internal" codec is removed.
1433
1434
1435 .. _binary-transforms:
1436
1437 Binary Transforms
1438 ^^^^^^^^^^^^^^^^^
1439
1440 The following codecs provide binary transforms: :term:`bytes-like object`
1441 to :class:`bytes` mappings. They are not supported by :meth:`bytes.decode`
1442 (which only produces :class:`str` output).
1443
1444
1445 .. tabularcolumns:: |l|L|L|L|
1446
1447 +----------------------+------------------+------------------------------+------------------------------+
1448 | Codec | Aliases | Meaning | Encoder / decoder |
1449 +======================+==================+==============================+==============================+
1450 | base64_codec [#b64]_ | base64, base_64 | Convert the operand to | :meth:`base64.encodebytes` / |
1451 | | | multiline MIME base64 (the | :meth:`base64.decodebytes` |
1452 | | | result always includes a | |
1453 | | | trailing ``'\n'``). | |
1454 | | | | |
1455 | | | .. versionchanged:: 3.4 | |
1456 | | | accepts any | |
1457 | | | :term:`bytes-like object` | |
1458 | | | as input for encoding and | |
1459 | | | decoding | |
1460 +----------------------+------------------+------------------------------+------------------------------+
1461 | bz2_codec | bz2 | Compress the operand using | :meth:`bz2.compress` / |
1462 | | | bz2. | :meth:`bz2.decompress` |
1463 +----------------------+------------------+------------------------------+------------------------------+
1464 | hex_codec | hex | Convert the operand to | :meth:`binascii.b2a_hex` / |
1465 | | | hexadecimal | :meth:`binascii.a2b_hex` |
1466 | | | representation, with two | |
1467 | | | digits per byte. | |
1468 +----------------------+------------------+------------------------------+------------------------------+
1469 | quopri_codec | quopri, | Convert the operand to MIME | :meth:`quopri.encode` with |
1470 | | quotedprintable, | quoted printable. | ``quotetabs=True`` / |
1471 | | quoted_printable | | :meth:`quopri.decode` |
1472 +----------------------+------------------+------------------------------+------------------------------+
1473 | uu_codec | uu | Convert the operand using | |
1474 | | | uuencode. | |
1475 +----------------------+------------------+------------------------------+------------------------------+
1476 | zlib_codec | zip, zlib | Compress the operand using | :meth:`zlib.compress` / |
1477 | | | gzip. | :meth:`zlib.decompress` |
1478 +----------------------+------------------+------------------------------+------------------------------+
1479
1480 .. [#b64] In addition to :term:`bytes-like objects <bytes-like object>`,
1481 ``'base64_codec'`` also accepts ASCII-only instances of :class:`str` for
1482 decoding
1483
1484 .. versionadded:: 3.2
1485 Restoration of the binary transforms.
1486
1487 .. versionchanged:: 3.4
1488 Restoration of the aliases for the binary transforms.
1489
1490
1491 .. _standalone-codec-functions:
1492
1493 Standalone Codec Functions
1494 ^^^^^^^^^^^^^^^^^^^^^^^^^^
1495
1496 The following functions provide encoding and decoding functionality similar to codecs,
1497 but are not available as named codecs through :func:`codecs.encode` or :func:`codecs.decode`.
1498 They are used internally (for example, by :mod:`pickle`) and behave similarly to the
1499 ``string_escape`` codec that was removed in Python 3.
1500
1501 .. function:: codecs.escape_encode(input, errors=None)
1502
1503 Encode *input* using escape sequences. Similar to how :func:`repr` on bytes
1504 produces escaped byte values.
1505
1506 *input* must be a :class:`bytes` object.
1507
1508 Returns a tuple ``(output, length)`` where *output* is a :class:`bytes`
1509 object and *length* is the number of bytes consumed.
1510
1511 .. function:: codecs.escape_decode(input, errors=None)
1512
1513 Decode *input* from escape sequences back to the original bytes.
1514
1515 *input* must be a :term:`bytes-like object`.
1516
1517 Returns a tuple ``(output, length)`` where *output* is a :class:`bytes`
1518 object and *length* is the number of bytes consumed.
1519
1520
1521 .. _text-transforms:
1522
1523 Text Transforms
1524 ^^^^^^^^^^^^^^^
1525
1526 The following codec provides a text transform: a :class:`str` to :class:`str`
1527 mapping. It is not supported by :meth:`str.encode` (which only produces
1528 :class:`bytes` output).
1529
1530 .. tabularcolumns:: |l|l|L|
1531
1532 +--------------------+---------+---------------------------+
1533 | Codec | Aliases | Meaning |
1534 +====================+=========+===========================+
1535 | rot_13 | rot13 | Return the Caesar-cypher |
1536 | | | encryption of the |
1537 | | | operand. |
1538 +--------------------+---------+---------------------------+
1539
1540 .. versionadded:: 3.2
1541 Restoration of the ``rot_13`` text transform.
1542
1543 .. versionchanged:: 3.4
1544 Restoration of the ``rot13`` alias.
1545
1546
1547 :mod:`!encodings` --- Encodings package
1548 ---------------------------------------
1549
1550 .. module:: encodings
1551 :synopsis: Encodings package
1552
1553 This module implements the following functions:
1554
1555 .. function:: normalize_encoding(encoding)
1556
1557 Normalize encoding name *encoding*.
1558
1559 Normalization works as follows: all non-alphanumeric characters except the
1560 dot used for Python package names are collapsed and replaced with a single
1561 underscore, leading and trailing underscores are removed.
1562 For example, ``' -;#'`` becomes ``'_'``.
1563
1564 Note that *encoding* should be ASCII only.
1565
1566
1567 .. note::
1568 The following functions should not be used directly, except for testing
1569 purposes; :func:`codecs.lookup` should be used instead.
1570
1571
1572 .. function:: search_function(encoding)
1573
1574 Search for the codec module corresponding to the given encoding name
1575 *encoding*.
1576
1577 This function first normalizes the *encoding* using
1578 :func:`normalize_encoding`, then looks for a corresponding alias.
1579 It attempts to import a codec module from the encodings package using either
1580 the alias or the normalized name. If the module is found and defines a valid
1581 ``getregentry()`` function that returns a :class:`codecs.CodecInfo` object,
1582 the codec is cached and returned.
1583
1584 If the codec module defines a ``getaliases()`` function any returned aliases
1585 are registered for future use.
1586
1587
1588 .. function:: win32_code_page_search_function(encoding)
1589
1590 Search for a Windows code page encoding *encoding* of the form ``cpXXXX``.
1591
1592 If the code page is valid and supported, return a :class:`codecs.CodecInfo`
1593 object for it.
1594
1595 .. availability:: Windows.
1596
1597 .. versionadded:: 3.14
1598
1599
1600 This module implements the following exception:
1601
1602 .. exception:: CodecRegistryError
1603
1604 Raised when a codec is invalid or incompatible.
1605
1606
1607 :mod:`!encodings.idna` --- Internationalized Domain Names in Applications
1608 -------------------------------------------------------------------------
1609
1610 .. module:: encodings.idna
1611 :synopsis: Internationalized Domain Names implementation
1612 .. moduleauthor:: Martin v. Löwis
1613
1614 This module implements :rfc:`3490` (Internationalized Domain Names in
1615 Applications) and :rfc:`3492` (Nameprep: A Stringprep Profile for
1616 Internationalized Domain Names (IDN)). It builds upon the ``punycode`` encoding
1617 and :mod:`stringprep`.
1618
1619 If you need the IDNA 2008 standard from :rfc:`5891` and :rfc:`5895`, use the
1620 third-party :pypi:`idna` module.
1621
1622 These RFCs together define a protocol to support non-ASCII characters in domain
1623 names. A domain name containing non-ASCII characters (such as
1624 ``www.Alliancefrançaise.nu``) is converted into an ASCII-compatible encoding
1625 (ACE, such as ``www.xn--alliancefranaise-npb.nu``). The ACE form of the domain
1626 name is then used in all places where arbitrary characters are not allowed by
1627 the protocol, such as DNS queries, HTTP :mailheader:`Host` fields, and so
1628 on. This conversion is carried out in the application; if possible invisible to
1629 the user: The application should transparently convert Unicode domain labels to
1630 IDNA on the wire, and convert back ACE labels to Unicode before presenting them
1631 to the user.
1632
1633 Python supports this conversion in several ways: the ``idna`` codec performs
1634 conversion between Unicode and ACE, separating an input string into labels
1635 based on the separator characters defined in :rfc:`section 3.1 of RFC 3490 <3490#section-3.1>`
1636 and converting each label to ACE as required, and conversely separating an input
1637 byte string into labels based on the ``.`` separator and converting any ACE
1638 labels found into unicode. Furthermore, the :mod:`socket` module
1639 transparently converts Unicode host names to ACE, so that applications need not
1640 be concerned about converting host names themselves when they pass them to the
1641 socket module. On top of that, modules that have host names as function
1642 parameters, such as :mod:`http.client` and :mod:`ftplib`, accept Unicode host
1643 names (:mod:`http.client` then also transparently sends an IDNA hostname in the
1644 :mailheader:`Host` field if it sends that field at all).
1645
1646 When receiving host names from the wire (such as in reverse name lookup), no
1647 automatic conversion to Unicode is performed: applications wishing to present
1648 such host names to the user should decode them to Unicode.
1649
1650 The module :mod:`!encodings.idna` also implements the nameprep procedure, which
1651 performs certain normalizations on host names, to achieve case-insensitivity of
1652 international domain names, and to unify similar characters. The nameprep
1653 functions can be used directly if desired.
1654
1655
1656 .. function:: nameprep(label)
1657
1658 Return the nameprepped version of *label*. The implementation currently assumes
1659 query strings, so ``AllowUnassigned`` is true.
1660
1661
1662 .. function:: ToASCII(label)
1663
1664 Convert a label to ASCII, as specified in :rfc:`3490`. ``UseSTD3ASCIIRules`` is
1665 assumed to be false.
1666
1667
1668 .. function:: ToUnicode(label)
1669
1670 Convert a label to Unicode, as specified in :rfc:`3490`.
1671
1672
1673 :mod:`!encodings.mbcs` --- Windows ANSI codepage
1674 ------------------------------------------------
1675
1676 .. module:: encodings.mbcs
1677 :synopsis: Windows ANSI codepage
1678
1679 This module implements the ANSI codepage (CP_ACP).
1680
1681 .. availability:: Windows.
1682
1683 .. versionchanged:: 3.2
1684 Before 3.2, the *errors* argument was ignored; ``'replace'`` was always used
1685 to encode, and ``'ignore'`` to decode.
1686
1687 .. versionchanged:: 3.3
1688 Support any error handler.
1689
1690
1691 :mod:`!encodings.utf_8_sig` --- UTF-8 codec with BOM signature
1692 --------------------------------------------------------------
1693
1694 .. module:: encodings.utf_8_sig
1695 :synopsis: UTF-8 codec with BOM signature
1696 .. moduleauthor:: Walter Dörwald
1697
1698 This module implements a variant of the UTF-8 codec. On encoding, a UTF-8 encoded
1699 BOM will be prepended to the UTF-8 encoded bytes. For the stateful encoder this
1700 is only done once (on the first write to the byte stream). On decoding, an
1701 optional UTF-8 encoded BOM at the start of the data will be skipped.