| 1 |
:mod:`!codecs` --- Codec registry and base classes |
| 2 |
================================================== |
| 3 |
|
| 4 |
.. module:: codecs |
| 5 |
:synopsis: Encode and decode data and streams. |
| 6 |
|
| 7 |
.. moduleauthor:: Marc-André Lemburg <[email protected]> |
| 8 |
.. sectionauthor:: Marc-André Lemburg <[email protected]> |
| 9 |
.. sectionauthor:: Martin v. Löwis <[email protected]> |
| 10 |
|
| 11 |
**Source code:** :source:`Lib/codecs.py` |
| 12 |
|
| 13 |
.. index:: |
| 14 |
single: Unicode |
| 15 |
single: Codecs |
| 16 |
pair: Codecs; encode |
| 17 |
pair: Codecs; decode |
| 18 |
single: streams |
| 19 |
pair: stackable; streams |
| 20 |
|
| 21 |
-------------- |
| 22 |
|
| 23 |
This module defines base classes for standard Python codecs (encoders and |
| 24 |
decoders) and provides access to the internal Python codec registry, which |
| 25 |
manages the codec and error handling lookup process. Most standard codecs |
| 26 |
are :term:`text encodings <text encoding>`, which encode text to bytes (and |
| 27 |
decode bytes to text), but there are also codecs provided that encode text to |
| 28 |
text, and bytes to bytes. Custom codecs may encode and decode between arbitrary |
| 29 |
types, but some module features are restricted to be used specifically with |
| 30 |
:term:`text encodings <text encoding>` or with codecs that encode to |
| 31 |
:class:`bytes`. |
| 32 |
|
| 33 |
The module defines the following functions for encoding and decoding with |
| 34 |
any codec: |
| 35 |
|
| 36 |
.. function:: encode(obj, encoding='utf-8', errors='strict') |
| 37 |
|
| 38 |
Encodes *obj* using the codec registered for *encoding*. |
| 39 |
|
| 40 |
*Errors* may be given to set the desired error handling scheme. The |
| 41 |
default error handler is ``'strict'`` meaning that encoding errors raise |
| 42 |
:exc:`ValueError` (or a more codec specific subclass, such as |
| 43 |
:exc:`UnicodeEncodeError`). Refer to :ref:`codec-base-classes` for more |
| 44 |
information on codec error handling. |
| 45 |
|
| 46 |
.. function:: decode(obj, encoding='utf-8', errors='strict') |
| 47 |
|
| 48 |
Decodes *obj* using the codec registered for *encoding*. |
| 49 |
|
| 50 |
*Errors* may be given to set the desired error handling scheme. The |
| 51 |
default error handler is ``'strict'`` meaning that decoding errors raise |
| 52 |
:exc:`ValueError` (or a more codec specific subclass, such as |
| 53 |
:exc:`UnicodeDecodeError`). Refer to :ref:`codec-base-classes` for more |
| 54 |
information on codec error handling. |
| 55 |
|
| 56 |
.. function:: charmap_build(string) |
| 57 |
|
| 58 |
Return a mapping suitable for encoding with a custom single-byte encoding. |
| 59 |
Given a :class:`str` *string* of up to 256 characters representing a |
| 60 |
decoding table, returns either a compact internal mapping object |
| 61 |
``EncodingMap`` or a :class:`dictionary <dict>` mapping character ordinals |
| 62 |
to byte values. Raises a :exc:`TypeError` on invalid input. |
| 63 |
|
| 64 |
The full details for each codec can also be looked up directly: |
| 65 |
|
| 66 |
.. function:: lookup(encoding, /) |
| 67 |
|
| 68 |
Looks up the codec info in the Python codec registry and returns a |
| 69 |
:class:`CodecInfo` object as defined below. |
| 70 |
|
| 71 |
Encodings are first looked up in the registry's cache. If not found, the list of |
| 72 |
registered search functions is scanned. If no :class:`CodecInfo` object is |
| 73 |
found, a :exc:`LookupError` is raised. Otherwise, the :class:`CodecInfo` object |
| 74 |
is stored in the cache and returned to the caller. |
| 75 |
|
| 76 |
.. class:: CodecInfo(encode, decode, streamreader=None, streamwriter=None, incrementalencoder=None, incrementaldecoder=None, name=None) |
| 77 |
|
| 78 |
Codec details when looking up the codec registry. The constructor |
| 79 |
arguments are stored in attributes of the same name: |
| 80 |
|
| 81 |
|
| 82 |
.. attribute:: name |
| 83 |
|
| 84 |
The name of the encoding. |
| 85 |
|
| 86 |
|
| 87 |
.. attribute:: encode |
| 88 |
decode |
| 89 |
|
| 90 |
The stateless encoding and decoding functions. These must be |
| 91 |
functions or methods which have the same interface as |
| 92 |
the :meth:`~Codec.encode` and :meth:`~Codec.decode` methods of Codec |
| 93 |
instances (see :ref:`Codec Interface <codec-objects>`). |
| 94 |
The functions or methods are expected to work in a stateless mode. |
| 95 |
|
| 96 |
|
| 97 |
.. attribute:: incrementalencoder |
| 98 |
incrementaldecoder |
| 99 |
|
| 100 |
Incremental encoder and decoder classes or factory functions. |
| 101 |
These have to provide the interface defined by the base classes |
| 102 |
:class:`IncrementalEncoder` and :class:`IncrementalDecoder`, |
| 103 |
respectively. Incremental codecs can maintain state. |
| 104 |
|
| 105 |
|
| 106 |
.. attribute:: streamwriter |
| 107 |
streamreader |
| 108 |
|
| 109 |
Stream writer and reader classes or factory functions. These have to |
| 110 |
provide the interface defined by the base classes |
| 111 |
:class:`StreamWriter` and :class:`StreamReader`, respectively. |
| 112 |
Stream codecs can maintain state. |
| 113 |
|
| 114 |
To simplify access to the various codec components, the module provides |
| 115 |
these additional functions which use :func:`lookup` for the codec lookup: |
| 116 |
|
| 117 |
.. function:: getencoder(encoding) |
| 118 |
|
| 119 |
Look up the codec for the given encoding and return its encoder function. |
| 120 |
|
| 121 |
Raises a :exc:`LookupError` in case the encoding cannot be found. |
| 122 |
|
| 123 |
|
| 124 |
.. function:: getdecoder(encoding) |
| 125 |
|
| 126 |
Look up the codec for the given encoding and return its decoder function. |
| 127 |
|
| 128 |
Raises a :exc:`LookupError` in case the encoding cannot be found. |
| 129 |
|
| 130 |
|
| 131 |
.. function:: getincrementalencoder(encoding) |
| 132 |
|
| 133 |
Look up the codec for the given encoding and return its incremental encoder |
| 134 |
class or factory function. |
| 135 |
|
| 136 |
Raises a :exc:`LookupError` in case the encoding cannot be found or the codec |
| 137 |
doesn't support an incremental encoder. |
| 138 |
|
| 139 |
|
| 140 |
.. function:: getincrementaldecoder(encoding) |
| 141 |
|
| 142 |
Look up the codec for the given encoding and return its incremental decoder |
| 143 |
class or factory function. |
| 144 |
|
| 145 |
Raises a :exc:`LookupError` in case the encoding cannot be found or the codec |
| 146 |
doesn't support an incremental decoder. |
| 147 |
|
| 148 |
|
| 149 |
.. function:: getreader(encoding) |
| 150 |
|
| 151 |
Look up the codec for the given encoding and return its :class:`StreamReader` |
| 152 |
class or factory function. |
| 153 |
|
| 154 |
Raises a :exc:`LookupError` in case the encoding cannot be found. |
| 155 |
|
| 156 |
|
| 157 |
.. function:: getwriter(encoding) |
| 158 |
|
| 159 |
Look up the codec for the given encoding and return its :class:`StreamWriter` |
| 160 |
class or factory function. |
| 161 |
|
| 162 |
Raises a :exc:`LookupError` in case the encoding cannot be found. |
| 163 |
|
| 164 |
Custom codecs are made available by registering a suitable codec search |
| 165 |
function: |
| 166 |
|
| 167 |
.. function:: register(search_function, /) |
| 168 |
|
| 169 |
Register a codec search function. Search functions are expected to take one |
| 170 |
argument, being the encoding name in all lower case letters with hyphens |
| 171 |
and spaces converted to underscores, and return a :class:`CodecInfo` object. |
| 172 |
In case a search function cannot find a given encoding, it should return |
| 173 |
``None``. |
| 174 |
|
| 175 |
.. versionchanged:: 3.9 |
| 176 |
Hyphens and spaces are converted to underscore. |
| 177 |
|
| 178 |
|
| 179 |
.. function:: unregister(search_function, /) |
| 180 |
|
| 181 |
Unregister a codec search function and clear the registry's cache. |
| 182 |
If the search function is not registered, do nothing. |
| 183 |
|
| 184 |
.. versionadded:: 3.10 |
| 185 |
|
| 186 |
|
| 187 |
While the builtin :func:`open` and the associated :mod:`io` module are the |
| 188 |
recommended approach for working with encoded text files, this module |
| 189 |
provides additional utility functions and classes that allow the use of a |
| 190 |
wider range of codecs when working with binary files: |
| 191 |
|
| 192 |
.. function:: open(filename, mode='r', encoding=None, errors='strict', buffering=-1) |
| 193 |
|
| 194 |
Open an encoded file using the given *mode* and return an instance of |
| 195 |
:class:`StreamReaderWriter`, providing transparent encoding/decoding. |
| 196 |
The default file mode is ``'r'``, meaning to open the file in read mode. |
| 197 |
|
| 198 |
.. note:: |
| 199 |
|
| 200 |
If *encoding* is not ``None``, then the |
| 201 |
underlying encoded files are always opened in binary mode. |
| 202 |
No automatic conversion of ``'\n'`` is done on reading and writing. |
| 203 |
The *mode* argument may be any binary mode acceptable to the built-in |
| 204 |
:func:`open` function; the ``'b'`` is automatically added. |
| 205 |
|
| 206 |
*encoding* specifies the encoding which is to be used for the file. |
| 207 |
Any encoding that encodes to and decodes from bytes is allowed, and |
| 208 |
the data types supported by the file methods depend on the codec used. |
| 209 |
|
| 210 |
*errors* may be given to define the error handling. It defaults to ``'strict'`` |
| 211 |
which causes a :exc:`ValueError` to be raised in case an encoding error occurs. |
| 212 |
|
| 213 |
*buffering* has the same meaning as for the built-in :func:`open` function. |
| 214 |
It defaults to -1 which means that the default buffer size will be used. |
| 215 |
|
| 216 |
.. versionchanged:: 3.11 |
| 217 |
The ``'U'`` mode has been removed. |
| 218 |
|
| 219 |
.. deprecated:: 3.14 |
| 220 |
|
| 221 |
:func:`codecs.open` has been superseded by :func:`open`. |
| 222 |
|
| 223 |
|
| 224 |
.. function:: EncodedFile(file, data_encoding, file_encoding=None, errors='strict') |
| 225 |
|
| 226 |
Return a :class:`StreamRecoder` instance, a wrapped version of *file* |
| 227 |
which provides transparent transcoding. The original file is closed |
| 228 |
when the wrapped version is closed. |
| 229 |
|
| 230 |
Data written to the wrapped file is decoded according to the given |
| 231 |
*data_encoding* and then written to the original file as bytes using |
| 232 |
*file_encoding*. Bytes read from the original file are decoded |
| 233 |
according to *file_encoding*, and the result is encoded |
| 234 |
using *data_encoding*. |
| 235 |
|
| 236 |
If *file_encoding* is not given, it defaults to *data_encoding*. |
| 237 |
|
| 238 |
*errors* may be given to define the error handling. It defaults to |
| 239 |
``'strict'``, which causes :exc:`ValueError` to be raised in case an encoding |
| 240 |
error occurs. |
| 241 |
|
| 242 |
|
| 243 |
.. function:: iterencode(iterator, encoding, errors='strict', **kwargs) |
| 244 |
|
| 245 |
Uses an incremental encoder to iteratively encode the input provided by |
| 246 |
*iterator*. *iterator* must yield :class:`str` objects. |
| 247 |
This function is a :term:`generator`. The *errors* argument (as well as any |
| 248 |
other keyword argument) is passed through to the incremental encoder. |
| 249 |
|
| 250 |
This function requires that the codec accept text :class:`str` objects |
| 251 |
to encode. Therefore it does not support bytes-to-bytes encoders such as |
| 252 |
``base64_codec``. |
| 253 |
|
| 254 |
|
| 255 |
.. function:: iterdecode(iterator, encoding, errors='strict', **kwargs) |
| 256 |
|
| 257 |
Uses an incremental decoder to iteratively decode the input provided by |
| 258 |
*iterator*. *iterator* must yield :class:`bytes` objects. |
| 259 |
This function is a :term:`generator`. The *errors* argument (as well as any |
| 260 |
other keyword argument) is passed through to the incremental decoder. |
| 261 |
|
| 262 |
This function requires that the codec accept :class:`bytes` objects |
| 263 |
to decode. Therefore it does not support text-to-text encoders such as |
| 264 |
``rot_13``, although ``rot_13`` may be used equivalently with |
| 265 |
:func:`iterencode`. |
| 266 |
|
| 267 |
|
| 268 |
.. function:: readbuffer_encode(buffer, errors=None, /) |
| 269 |
|
| 270 |
Return a :class:`tuple` containing the raw bytes of *buffer*, a |
| 271 |
:ref:`buffer-compatible object <bufferobjects>` or :class:`str` |
| 272 |
(encoded to UTF-8 before processing), and their length in bytes. |
| 273 |
|
| 274 |
The *errors* argument is ignored. |
| 275 |
|
| 276 |
.. code-block:: pycon |
| 277 |
|
| 278 |
>>> codecs.readbuffer_encode(b"Zito") |
| 279 |
(b'Zito', 4) |
| 280 |
|
| 281 |
|
| 282 |
The module also provides the following constants which are useful for reading |
| 283 |
and writing to platform dependent files: |
| 284 |
|
| 285 |
|
| 286 |
.. data:: BOM |
| 287 |
BOM_BE |
| 288 |
BOM_LE |
| 289 |
BOM_UTF8 |
| 290 |
BOM_UTF16 |
| 291 |
BOM_UTF16_BE |
| 292 |
BOM_UTF16_LE |
| 293 |
BOM_UTF32 |
| 294 |
BOM_UTF32_BE |
| 295 |
BOM_UTF32_LE |
| 296 |
|
| 297 |
These constants define various byte sequences, |
| 298 |
being Unicode byte order marks (BOMs) for several encodings. They are |
| 299 |
used in UTF-16 and UTF-32 data streams to indicate the byte order used, |
| 300 |
and in UTF-8 as a Unicode signature. :const:`BOM_UTF16` is either |
| 301 |
:const:`BOM_UTF16_BE` or :const:`BOM_UTF16_LE` depending on the platform's |
| 302 |
native byte order, :const:`BOM` is an alias for :const:`BOM_UTF16`, |
| 303 |
:const:`BOM_LE` for :const:`BOM_UTF16_LE` and :const:`BOM_BE` for |
| 304 |
:const:`BOM_UTF16_BE`. The others represent the BOM in UTF-8 and UTF-32 |
| 305 |
encodings. |
| 306 |
|
| 307 |
|
| 308 |
.. _codec-base-classes: |
| 309 |
|
| 310 |
Codec Base Classes |
| 311 |
------------------ |
| 312 |
|
| 313 |
The :mod:`!codecs` module defines a set of base classes which define the |
| 314 |
interfaces for working with codec objects, and can also be used as the basis |
| 315 |
for custom codec implementations. |
| 316 |
|
| 317 |
Each codec has to define four interfaces to make it usable as codec in Python: |
| 318 |
stateless encoder, stateless decoder, stream reader and stream writer. The |
| 319 |
stream reader and writers typically reuse the stateless encoder/decoder to |
| 320 |
implement the file protocols. Codec authors also need to define how the |
| 321 |
codec will handle encoding and decoding errors. |
| 322 |
|
| 323 |
|
| 324 |
.. _surrogateescape: |
| 325 |
.. _error-handlers: |
| 326 |
|
| 327 |
Error Handlers |
| 328 |
^^^^^^^^^^^^^^ |
| 329 |
|
| 330 |
To simplify and standardize error handling, codecs may implement different |
| 331 |
error handling schemes by accepting the *errors* string argument: |
| 332 |
|
| 333 |
>>> 'German ß, ♬'.encode(encoding='ascii', errors='backslashreplace') |
| 334 |
b'German \\xdf, \\u266c' |
| 335 |
>>> 'German ß, ♬'.encode(encoding='ascii', errors='xmlcharrefreplace') |
| 336 |
b'German ß, ♬' |
| 337 |
|
| 338 |
.. index:: |
| 339 |
pair: strict; error handler's name |
| 340 |
pair: ignore; error handler's name |
| 341 |
pair: replace; error handler's name |
| 342 |
pair: backslashreplace; error handler's name |
| 343 |
pair: surrogateescape; error handler's name |
| 344 |
single: ? (question mark); replacement character |
| 345 |
single: \ (backslash); escape sequence |
| 346 |
single: \x; escape sequence |
| 347 |
single: \u; escape sequence |
| 348 |
single: \U; escape sequence |
| 349 |
|
| 350 |
The following error handlers can be used with all Python |
| 351 |
:ref:`standard-encodings` codecs: |
| 352 |
|
| 353 |
.. tabularcolumns:: |l|L| |
| 354 |
|
| 355 |
+-------------------------+-----------------------------------------------+ |
| 356 |
| Value | Meaning | |
| 357 |
+=========================+===============================================+ |
| 358 |
| ``'strict'`` | Raise :exc:`UnicodeError` (or a subclass), | |
| 359 |
| | this is the default. Implemented in | |
| 360 |
| | :func:`strict_errors`. | |
| 361 |
+-------------------------+-----------------------------------------------+ |
| 362 |
| ``'ignore'`` | Ignore the malformed data and continue without| |
| 363 |
| | further notice. Implemented in | |
| 364 |
| | :func:`ignore_errors`. | |
| 365 |
+-------------------------+-----------------------------------------------+ |
| 366 |
| ``'replace'`` | Replace with a replacement marker. On | |
| 367 |
| | encoding, use ``?`` (ASCII character). On | |
| 368 |
| | decoding, use ``�`` (U+FFFD, the official | |
| 369 |
| | REPLACEMENT CHARACTER). Implemented in | |
| 370 |
| | :func:`replace_errors`. | |
| 371 |
+-------------------------+-----------------------------------------------+ |
| 372 |
| ``'backslashreplace'`` | Replace with backslashed escape sequences. | |
| 373 |
| | On encoding, use hexadecimal form of Unicode | |
| 374 |
| | code point with formats :samp:`\\x{hh}` | |
| 375 |
| | :samp:`\\u{xxxx}` :samp:`\\U{xxxxxxxx}`. | |
| 376 |
| | On decoding, use hexadecimal form of byte | |
| 377 |
| | value with format :samp:`\\x{hh}`. | |
| 378 |
| | Implemented in | |
| 379 |
| | :func:`backslashreplace_errors`. | |
| 380 |
+-------------------------+-----------------------------------------------+ |
| 381 |
| ``'surrogateescape'`` | On decoding, replace byte with individual | |
| 382 |
| | surrogate code ranging from ``U+DC80`` to | |
| 383 |
| | ``U+DCFF``. This code will then be turned | |
| 384 |
| | back into the same byte when the | |
| 385 |
| | ``'surrogateescape'`` error handler is used | |
| 386 |
| | when encoding the data. (See :pep:`383` for | |
| 387 |
| | more.) | |
| 388 |
+-------------------------+-----------------------------------------------+ |
| 389 |
|
| 390 |
.. index:: |
| 391 |
pair: xmlcharrefreplace; error handler's name |
| 392 |
pair: namereplace; error handler's name |
| 393 |
single: \N; escape sequence |
| 394 |
|
| 395 |
The following error handlers are only applicable to encoding (within |
| 396 |
:term:`text encodings <text encoding>`): |
| 397 |
|
| 398 |
+-------------------------+-----------------------------------------------+ |
| 399 |
| Value | Meaning | |
| 400 |
+=========================+===============================================+ |
| 401 |
| ``'xmlcharrefreplace'`` | Replace with XML/HTML numeric character | |
| 402 |
| | reference, which is a decimal form of Unicode | |
| 403 |
| | code point with format :samp:`&#{num};`. | |
| 404 |
| | Implemented in | |
| 405 |
| | :func:`xmlcharrefreplace_errors`. | |
| 406 |
+-------------------------+-----------------------------------------------+ |
| 407 |
| ``'namereplace'`` | Replace with ``\N{...}`` escape sequences, | |
| 408 |
| | what appears in the braces is the Name | |
| 409 |
| | property from Unicode Character Database. | |
| 410 |
| | Implemented in :func:`namereplace_errors`. | |
| 411 |
+-------------------------+-----------------------------------------------+ |
| 412 |
|
| 413 |
.. index:: |
| 414 |
pair: surrogatepass; error handler's name |
| 415 |
|
| 416 |
In addition, the following error handler is specific to the given codecs: |
| 417 |
|
| 418 |
+-------------------+------------------------+-------------------------------------------+ |
| 419 |
| Value | Codecs | Meaning | |
| 420 |
+===================+========================+===========================================+ |
| 421 |
|``'surrogatepass'``| utf-8, utf-16, utf-32, | Allow encoding and decoding surrogate code| |
| 422 |
| | utf-16-be, utf-16-le, | point (``U+D800`` - ``U+DFFF``) as normal | |
| 423 |
| | utf-32-be, utf-32-le | code point. Otherwise these codecs treat | |
| 424 |
| | | the presence of surrogate code point in | |
| 425 |
| | | :class:`str` as an error. | |
| 426 |
+-------------------+------------------------+-------------------------------------------+ |
| 427 |
|
| 428 |
.. versionadded:: 3.1 |
| 429 |
The ``'surrogateescape'`` and ``'surrogatepass'`` error handlers. |
| 430 |
|
| 431 |
.. versionchanged:: 3.4 |
| 432 |
The ``'surrogatepass'`` error handler now works with utf-16\* and utf-32\* |
| 433 |
codecs. |
| 434 |
|
| 435 |
.. versionadded:: 3.5 |
| 436 |
The ``'namereplace'`` error handler. |
| 437 |
|
| 438 |
.. versionchanged:: 3.5 |
| 439 |
The ``'backslashreplace'`` error handler now works with decoding and |
| 440 |
translating. |
| 441 |
|
| 442 |
The set of allowed values can be extended by registering a new named error |
| 443 |
handler: |
| 444 |
|
| 445 |
.. function:: register_error(name, error_handler, /) |
| 446 |
|
| 447 |
Register the error handling function *error_handler* under the name *name*. |
| 448 |
The *error_handler* argument will be called during encoding and decoding |
| 449 |
in case of an error, when *name* is specified as the errors parameter. |
| 450 |
|
| 451 |
For encoding, *error_handler* will be called with a :exc:`UnicodeEncodeError` |
| 452 |
instance, which contains information about the location of the error. The |
| 453 |
error handler must either raise this or a different exception, or return a |
| 454 |
tuple with a replacement for the unencodable part of the input and a position |
| 455 |
where encoding should continue. The replacement may be either :class:`str` or |
| 456 |
:class:`bytes`. If the replacement is bytes, the encoder will simply copy |
| 457 |
them into the output buffer. If the replacement is a string, the encoder will |
| 458 |
encode the replacement. Encoding continues on original input at the |
| 459 |
specified position. Negative position values will be treated as being |
| 460 |
relative to the end of the input string. If the resulting position is out of |
| 461 |
bound an :exc:`IndexError` will be raised. |
| 462 |
|
| 463 |
Decoding and translating works similarly, except :exc:`UnicodeDecodeError` or |
| 464 |
:exc:`UnicodeTranslateError` will be passed to the handler and that the |
| 465 |
replacement from the error handler will be put into the output directly. |
| 466 |
|
| 467 |
|
| 468 |
Previously registered error handlers (including the standard error handlers) |
| 469 |
can be looked up by name: |
| 470 |
|
| 471 |
.. function:: lookup_error(name, /) |
| 472 |
|
| 473 |
Return the error handler previously registered under the name *name*. |
| 474 |
|
| 475 |
Raises a :exc:`LookupError` in case the handler cannot be found. |
| 476 |
|
| 477 |
The following standard error handlers are also made available as module level |
| 478 |
functions: |
| 479 |
|
| 480 |
.. function:: strict_errors(exception) |
| 481 |
|
| 482 |
Implements the ``'strict'`` error handling. |
| 483 |
|
| 484 |
Each encoding or decoding error raises a :exc:`UnicodeError`. |
| 485 |
|
| 486 |
|
| 487 |
.. function:: ignore_errors(exception) |
| 488 |
|
| 489 |
Implements the ``'ignore'`` error handling. |
| 490 |
|
| 491 |
Malformed data is ignored; encoding or decoding is continued without |
| 492 |
further notice. |
| 493 |
|
| 494 |
|
| 495 |
.. function:: replace_errors(exception) |
| 496 |
|
| 497 |
Implements the ``'replace'`` error handling. |
| 498 |
|
| 499 |
Substitutes ``?`` (ASCII character) for encoding errors or ``�`` (U+FFFD, |
| 500 |
the official REPLACEMENT CHARACTER) for decoding errors. |
| 501 |
|
| 502 |
|
| 503 |
.. function:: backslashreplace_errors(exception) |
| 504 |
|
| 505 |
Implements the ``'backslashreplace'`` error handling. |
| 506 |
|
| 507 |
Malformed data is replaced by a backslashed escape sequence. |
| 508 |
On encoding, use the hexadecimal form of Unicode code point with formats |
| 509 |
:samp:`\\x{hh}` :samp:`\\u{xxxx}` :samp:`\\U{xxxxxxxx}`. |
| 510 |
On decoding, use the hexadecimal form of |
| 511 |
byte value with format :samp:`\\x{hh}`. |
| 512 |
|
| 513 |
.. versionchanged:: 3.5 |
| 514 |
Works with decoding and translating. |
| 515 |
|
| 516 |
|
| 517 |
.. function:: xmlcharrefreplace_errors(exception) |
| 518 |
|
| 519 |
Implements the ``'xmlcharrefreplace'`` error handling (for encoding within |
| 520 |
:term:`text encoding` only). |
| 521 |
|
| 522 |
The unencodable character is replaced by an appropriate XML/HTML numeric |
| 523 |
character reference, which is a decimal form of Unicode code point with |
| 524 |
format :samp:`&#{num};` . |
| 525 |
|
| 526 |
|
| 527 |
.. function:: namereplace_errors(exception) |
| 528 |
|
| 529 |
Implements the ``'namereplace'`` error handling (for encoding within |
| 530 |
:term:`text encoding` only). |
| 531 |
|
| 532 |
The unencodable character is replaced by a ``\N{...}`` escape sequence. The |
| 533 |
set of characters that appear in the braces is the Name property from |
| 534 |
Unicode Character Database. For example, the German lowercase letter ``'ß'`` |
| 535 |
will be converted to byte sequence ``\N{LATIN SMALL LETTER SHARP S}`` . |
| 536 |
|
| 537 |
.. versionadded:: 3.5 |
| 538 |
|
| 539 |
|
| 540 |
.. _codec-objects: |
| 541 |
|
| 542 |
Stateless Encoding and Decoding |
| 543 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 544 |
|
| 545 |
The base :class:`Codec` class defines these methods which also define the |
| 546 |
function interfaces of the stateless encoder and decoder: |
| 547 |
|
| 548 |
|
| 549 |
.. class:: Codec |
| 550 |
|
| 551 |
.. method:: encode(input, errors='strict') |
| 552 |
|
| 553 |
Encodes the object *input* and returns a tuple (output object, length consumed). |
| 554 |
For instance, :term:`text encoding` converts |
| 555 |
a string object to a bytes object using a particular |
| 556 |
character set encoding (e.g., ``cp1252`` or ``iso-8859-1``). |
| 557 |
|
| 558 |
The *errors* argument defines the error handling to apply. |
| 559 |
It defaults to ``'strict'`` handling. |
| 560 |
|
| 561 |
The method may not store state in the :class:`Codec` instance. Use |
| 562 |
:class:`StreamWriter` for codecs which have to keep state in order to make |
| 563 |
encoding efficient. |
| 564 |
|
| 565 |
The encoder must be able to handle zero length input and return an empty object |
| 566 |
of the output object type in this situation. |
| 567 |
|
| 568 |
|
| 569 |
.. method:: decode(input, errors='strict') |
| 570 |
|
| 571 |
Decodes the object *input* and returns a tuple (output object, length |
| 572 |
consumed). For instance, for a :term:`text encoding`, decoding converts |
| 573 |
a bytes object encoded using a particular |
| 574 |
character set encoding to a string object. |
| 575 |
|
| 576 |
For text encodings and bytes-to-bytes codecs, |
| 577 |
*input* must be a bytes object or one which provides the read-only |
| 578 |
buffer interface -- for example, buffer objects and memory mapped files. |
| 579 |
|
| 580 |
The *errors* argument defines the error handling to apply. |
| 581 |
It defaults to ``'strict'`` handling. |
| 582 |
|
| 583 |
The method may not store state in the :class:`Codec` instance. Use |
| 584 |
:class:`StreamReader` for codecs which have to keep state in order to make |
| 585 |
decoding efficient. |
| 586 |
|
| 587 |
The decoder must be able to handle zero length input and return an empty object |
| 588 |
of the output object type in this situation. |
| 589 |
|
| 590 |
|
| 591 |
Incremental Encoding and Decoding |
| 592 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 593 |
|
| 594 |
The :class:`IncrementalEncoder` and :class:`IncrementalDecoder` classes provide |
| 595 |
the basic interface for incremental encoding and decoding. Encoding/decoding the |
| 596 |
input isn't done with one call to the stateless encoder/decoder function, but |
| 597 |
with multiple calls to the |
| 598 |
:meth:`~IncrementalEncoder.encode`/:meth:`~IncrementalDecoder.decode` method of |
| 599 |
the incremental encoder/decoder. The incremental encoder/decoder keeps track of |
| 600 |
the encoding/decoding process during method calls. |
| 601 |
|
| 602 |
The joined output of calls to the |
| 603 |
:meth:`~IncrementalEncoder.encode`/:meth:`~IncrementalDecoder.decode` method is |
| 604 |
the same as if all the single inputs were joined into one, and this input was |
| 605 |
encoded/decoded with the stateless encoder/decoder. |
| 606 |
|
| 607 |
|
| 608 |
.. _incremental-encoder-objects: |
| 609 |
|
| 610 |
IncrementalEncoder Objects |
| 611 |
~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 612 |
|
| 613 |
The :class:`IncrementalEncoder` class is used for encoding an input in multiple |
| 614 |
steps. It defines the following methods which every incremental encoder must |
| 615 |
define in order to be compatible with the Python codec registry. |
| 616 |
|
| 617 |
|
| 618 |
.. class:: IncrementalEncoder(errors='strict') |
| 619 |
|
| 620 |
Constructor for an :class:`IncrementalEncoder` instance. |
| 621 |
|
| 622 |
All incremental encoders must provide this constructor interface. They are free |
| 623 |
to add additional keyword arguments, but only the ones defined here are used by |
| 624 |
the Python codec registry. |
| 625 |
|
| 626 |
The :class:`IncrementalEncoder` may implement different error handling schemes |
| 627 |
by providing the *errors* keyword argument. See :ref:`error-handlers` for |
| 628 |
possible values. |
| 629 |
|
| 630 |
The *errors* argument will be assigned to an attribute of the same name. |
| 631 |
Assigning to this attribute makes it possible to switch between different error |
| 632 |
handling strategies during the lifetime of the :class:`IncrementalEncoder` |
| 633 |
object. |
| 634 |
|
| 635 |
|
| 636 |
.. method:: encode(object, final=False) |
| 637 |
|
| 638 |
Encodes *object* (taking the current state of the encoder into account) |
| 639 |
and returns the resulting encoded object. If this is the last call to |
| 640 |
:meth:`encode` *final* must be true (the default is false). |
| 641 |
|
| 642 |
|
| 643 |
.. method:: reset() |
| 644 |
|
| 645 |
Reset the encoder to the initial state. The output is discarded: call |
| 646 |
``.encode(object, final=True)``, passing an empty byte or text string |
| 647 |
if necessary, to reset the encoder and to get the output. |
| 648 |
|
| 649 |
|
| 650 |
.. method:: getstate() |
| 651 |
|
| 652 |
Return the current state of the encoder which must be an integer. The |
| 653 |
implementation should make sure that ``0`` is the most common |
| 654 |
state. (States that are more complicated than integers can be converted |
| 655 |
into an integer by marshaling/pickling the state and encoding the bytes |
| 656 |
of the resulting string into an integer.) |
| 657 |
|
| 658 |
|
| 659 |
.. method:: setstate(state) |
| 660 |
|
| 661 |
Set the state of the encoder to *state*. *state* must be an encoder state |
| 662 |
returned by :meth:`getstate`. |
| 663 |
|
| 664 |
|
| 665 |
.. _incremental-decoder-objects: |
| 666 |
|
| 667 |
IncrementalDecoder Objects |
| 668 |
~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 669 |
|
| 670 |
The :class:`IncrementalDecoder` class is used for decoding an input in multiple |
| 671 |
steps. It defines the following methods which every incremental decoder must |
| 672 |
define in order to be compatible with the Python codec registry. |
| 673 |
|
| 674 |
|
| 675 |
.. class:: IncrementalDecoder(errors='strict') |
| 676 |
|
| 677 |
Constructor for an :class:`IncrementalDecoder` instance. |
| 678 |
|
| 679 |
All incremental decoders must provide this constructor interface. They are free |
| 680 |
to add additional keyword arguments, but only the ones defined here are used by |
| 681 |
the Python codec registry. |
| 682 |
|
| 683 |
The :class:`IncrementalDecoder` may implement different error handling schemes |
| 684 |
by providing the *errors* keyword argument. See :ref:`error-handlers` for |
| 685 |
possible values. |
| 686 |
|
| 687 |
The *errors* argument will be assigned to an attribute of the same name. |
| 688 |
Assigning to this attribute makes it possible to switch between different error |
| 689 |
handling strategies during the lifetime of the :class:`IncrementalDecoder` |
| 690 |
object. |
| 691 |
|
| 692 |
|
| 693 |
.. method:: decode(object, final=False) |
| 694 |
|
| 695 |
Decodes *object* (taking the current state of the decoder into account) |
| 696 |
and returns the resulting decoded object. If this is the last call to |
| 697 |
:meth:`decode` *final* must be true (the default is false). If *final* is |
| 698 |
true the decoder must decode the input completely and must flush all |
| 699 |
buffers. If this isn't possible (e.g. because of incomplete byte sequences |
| 700 |
at the end of the input) it must initiate error handling just like in the |
| 701 |
stateless case (which might raise an exception). |
| 702 |
|
| 703 |
|
| 704 |
.. method:: reset() |
| 705 |
|
| 706 |
Reset the decoder to the initial state. |
| 707 |
|
| 708 |
|
| 709 |
.. method:: getstate() |
| 710 |
|
| 711 |
Return the current state of the decoder. This must be a tuple with two |
| 712 |
items, the first must be the buffer containing the still undecoded |
| 713 |
input. The second must be an integer and can be additional state |
| 714 |
info. (The implementation should make sure that ``0`` is the most common |
| 715 |
additional state info.) If this additional state info is ``0`` it must be |
| 716 |
possible to set the decoder to the state which has no input buffered and |
| 717 |
``0`` as the additional state info, so that feeding the previously |
| 718 |
buffered input to the decoder returns it to the previous state without |
| 719 |
producing any output. (Additional state info that is more complicated than |
| 720 |
integers can be converted into an integer by marshaling/pickling the info |
| 721 |
and encoding the bytes of the resulting string into an integer.) |
| 722 |
|
| 723 |
|
| 724 |
.. method:: setstate(state) |
| 725 |
|
| 726 |
Set the state of the decoder to *state*. *state* must be a decoder state |
| 727 |
returned by :meth:`getstate`. |
| 728 |
|
| 729 |
|
| 730 |
Stream Encoding and Decoding |
| 731 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 732 |
|
| 733 |
|
| 734 |
The :class:`StreamWriter` and :class:`StreamReader` classes provide generic |
| 735 |
working interfaces which can be used to implement new encoding submodules very |
| 736 |
easily. See :mod:`!encodings.utf_8` for an example of how this is done. |
| 737 |
|
| 738 |
|
| 739 |
.. _stream-writer-objects: |
| 740 |
|
| 741 |
StreamWriter Objects |
| 742 |
~~~~~~~~~~~~~~~~~~~~ |
| 743 |
|
| 744 |
The :class:`StreamWriter` class is a subclass of :class:`Codec` and defines the |
| 745 |
following methods which every stream writer must define in order to be |
| 746 |
compatible with the Python codec registry. |
| 747 |
|
| 748 |
|
| 749 |
.. class:: StreamWriter(stream, errors='strict') |
| 750 |
|
| 751 |
Constructor for a :class:`StreamWriter` instance. |
| 752 |
|
| 753 |
All stream writers must provide this constructor interface. They are free to add |
| 754 |
additional keyword arguments, but only the ones defined here are used by the |
| 755 |
Python codec registry. |
| 756 |
|
| 757 |
The *stream* argument must be a file-like object open for writing |
| 758 |
text or binary data, as appropriate for the specific codec. |
| 759 |
|
| 760 |
The :class:`StreamWriter` may implement different error handling schemes by |
| 761 |
providing the *errors* keyword argument. See :ref:`error-handlers` for |
| 762 |
the standard error handlers the underlying stream codec may support. |
| 763 |
|
| 764 |
The *errors* argument will be assigned to an attribute of the same name. |
| 765 |
Assigning to this attribute makes it possible to switch between different error |
| 766 |
handling strategies during the lifetime of the :class:`StreamWriter` object. |
| 767 |
|
| 768 |
.. method:: write(object) |
| 769 |
|
| 770 |
Writes the object's contents encoded to the stream. |
| 771 |
|
| 772 |
|
| 773 |
.. method:: writelines(list) |
| 774 |
|
| 775 |
Writes the concatenated iterable of strings to the stream (possibly by reusing |
| 776 |
the :meth:`write` method). Infinite or |
| 777 |
very large iterables are not supported. The standard bytes-to-bytes codecs |
| 778 |
do not support this method. |
| 779 |
|
| 780 |
|
| 781 |
.. method:: reset() |
| 782 |
|
| 783 |
Resets the codec buffers used for keeping internal state. |
| 784 |
|
| 785 |
Calling this method should ensure that the data on the output is put into |
| 786 |
a clean state that allows appending of new fresh data without having to |
| 787 |
rescan the whole stream to recover state. |
| 788 |
|
| 789 |
|
| 790 |
In addition to the above methods, the :class:`StreamWriter` must also inherit |
| 791 |
all other methods and attributes from the underlying stream. |
| 792 |
|
| 793 |
|
| 794 |
.. _stream-reader-objects: |
| 795 |
|
| 796 |
StreamReader Objects |
| 797 |
~~~~~~~~~~~~~~~~~~~~ |
| 798 |
|
| 799 |
The :class:`StreamReader` class is a subclass of :class:`Codec` and defines the |
| 800 |
following methods which every stream reader must define in order to be |
| 801 |
compatible with the Python codec registry. |
| 802 |
|
| 803 |
|
| 804 |
.. class:: StreamReader(stream, errors='strict') |
| 805 |
|
| 806 |
Constructor for a :class:`StreamReader` instance. |
| 807 |
|
| 808 |
All stream readers must provide this constructor interface. They are free to add |
| 809 |
additional keyword arguments, but only the ones defined here are used by the |
| 810 |
Python codec registry. |
| 811 |
|
| 812 |
The *stream* argument must be a file-like object open for reading |
| 813 |
text or binary data, as appropriate for the specific codec. |
| 814 |
|
| 815 |
The :class:`StreamReader` may implement different error handling schemes by |
| 816 |
providing the *errors* keyword argument. See :ref:`error-handlers` for |
| 817 |
the standard error handlers the underlying stream codec may support. |
| 818 |
|
| 819 |
The *errors* argument will be assigned to an attribute of the same name. |
| 820 |
Assigning to this attribute makes it possible to switch between different error |
| 821 |
handling strategies during the lifetime of the :class:`StreamReader` object. |
| 822 |
|
| 823 |
The set of allowed values for the *errors* argument can be extended with |
| 824 |
:func:`register_error`. |
| 825 |
|
| 826 |
|
| 827 |
.. method:: read(size=-1, chars=-1, firstline=False) |
| 828 |
|
| 829 |
Decodes data from the stream and returns the resulting object. |
| 830 |
|
| 831 |
The *chars* argument indicates the number of decoded |
| 832 |
code points or bytes to return. The :func:`read` method will |
| 833 |
never return more data than requested, but it might return less, |
| 834 |
if there is not enough available. |
| 835 |
|
| 836 |
The *size* argument indicates the approximate maximum |
| 837 |
number of encoded bytes or code points to read |
| 838 |
for decoding. The decoder can modify this setting as |
| 839 |
appropriate. The default value -1 indicates to read and decode as much as |
| 840 |
possible. This parameter is intended to |
| 841 |
prevent having to decode huge files in one step. |
| 842 |
|
| 843 |
The *firstline* flag indicates that |
| 844 |
it would be sufficient to only return the first |
| 845 |
line, if there are decoding errors on later lines. |
| 846 |
|
| 847 |
The method should use a greedy read strategy meaning that it should read |
| 848 |
as much data as is allowed within the definition of the encoding and the |
| 849 |
given size, e.g. if optional encoding endings or state markers are |
| 850 |
available on the stream, these should be read too. |
| 851 |
|
| 852 |
|
| 853 |
.. method:: readline(size=None, keepends=True) |
| 854 |
|
| 855 |
Read one line from the input stream and return the decoded data. |
| 856 |
|
| 857 |
*size*, if given, is passed as size argument to the stream's |
| 858 |
:meth:`read` method. |
| 859 |
|
| 860 |
If *keepends* is false line-endings will be stripped from the lines |
| 861 |
returned. |
| 862 |
|
| 863 |
|
| 864 |
.. method:: readlines(sizehint=None, keepends=True) |
| 865 |
|
| 866 |
Read all lines available on the input stream and return them as a list of |
| 867 |
lines. |
| 868 |
|
| 869 |
Line-endings are implemented using the codec's :meth:`decode` method and |
| 870 |
are included in the list entries if *keepends* is true. |
| 871 |
|
| 872 |
*sizehint*, if given, is passed as the *size* argument to the stream's |
| 873 |
:meth:`read` method. |
| 874 |
|
| 875 |
|
| 876 |
.. method:: reset() |
| 877 |
|
| 878 |
Resets the codec buffers used for keeping internal state. |
| 879 |
|
| 880 |
Note that no stream repositioning should take place. This method is |
| 881 |
primarily intended to be able to recover from decoding errors. |
| 882 |
|
| 883 |
|
| 884 |
In addition to the above methods, the :class:`StreamReader` must also inherit |
| 885 |
all other methods and attributes from the underlying stream. |
| 886 |
|
| 887 |
.. _stream-reader-writer: |
| 888 |
|
| 889 |
StreamReaderWriter Objects |
| 890 |
~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 891 |
|
| 892 |
The :class:`StreamReaderWriter` is a convenience class that allows wrapping |
| 893 |
streams which work in both read and write modes. |
| 894 |
|
| 895 |
The design is such that one can use the factory functions returned by the |
| 896 |
:func:`lookup` function to construct the instance. |
| 897 |
|
| 898 |
|
| 899 |
.. class:: StreamReaderWriter(stream, Reader, Writer, errors='strict') |
| 900 |
|
| 901 |
Creates a :class:`StreamReaderWriter` instance. *stream* must be a file-like |
| 902 |
object. *Reader* and *Writer* must be factory functions or classes providing the |
| 903 |
:class:`StreamReader` and :class:`StreamWriter` interface resp. Error handling |
| 904 |
is done in the same way as defined for the stream readers and writers. |
| 905 |
|
| 906 |
:class:`StreamReaderWriter` instances define the combined interfaces of |
| 907 |
:class:`StreamReader` and :class:`StreamWriter` classes. They inherit all other |
| 908 |
methods and attributes from the underlying stream. |
| 909 |
|
| 910 |
|
| 911 |
.. _stream-recoder-objects: |
| 912 |
|
| 913 |
StreamRecoder Objects |
| 914 |
~~~~~~~~~~~~~~~~~~~~~ |
| 915 |
|
| 916 |
The :class:`StreamRecoder` translates data from one encoding to another, |
| 917 |
which is sometimes useful when dealing with different encoding environments. |
| 918 |
|
| 919 |
The design is such that one can use the factory functions returned by the |
| 920 |
:func:`lookup` function to construct the instance. |
| 921 |
|
| 922 |
|
| 923 |
.. class:: StreamRecoder(stream, encode, decode, Reader, Writer, errors='strict') |
| 924 |
|
| 925 |
Creates a :class:`StreamRecoder` instance which implements a two-way conversion: |
| 926 |
*encode* and *decode* work on the frontend — the data visible to |
| 927 |
code calling :meth:`~StreamReader.read` and :meth:`~StreamWriter.write`, |
| 928 |
while *Reader* and *Writer* |
| 929 |
work on the backend — the data in *stream*. |
| 930 |
|
| 931 |
You can use these objects to do transparent transcodings, e.g., from Latin-1 |
| 932 |
to UTF-8 and back. |
| 933 |
|
| 934 |
The *stream* argument must be a file-like object. |
| 935 |
|
| 936 |
The *encode* and *decode* arguments must |
| 937 |
adhere to the :class:`Codec` interface. *Reader* and |
| 938 |
*Writer* must be factory functions or classes providing objects of the |
| 939 |
:class:`StreamReader` and :class:`StreamWriter` interface respectively. |
| 940 |
|
| 941 |
Error handling is done in the same way as defined for the stream readers and |
| 942 |
writers. |
| 943 |
|
| 944 |
|
| 945 |
:class:`StreamRecoder` instances define the combined interfaces of |
| 946 |
:class:`StreamReader` and :class:`StreamWriter` classes. They inherit all other |
| 947 |
methods and attributes from the underlying stream. |
| 948 |
|
| 949 |
|
| 950 |
.. _encodings-overview: |
| 951 |
|
| 952 |
Encodings and Unicode |
| 953 |
--------------------- |
| 954 |
|
| 955 |
Strings are stored internally as sequences of code points in |
| 956 |
range ``U+0000``--``U+10FFFF``. (See :pep:`393` for |
| 957 |
more details about the implementation.) |
| 958 |
Once a string object is used outside of CPU and memory, endianness |
| 959 |
and how these arrays are stored as bytes become an issue. As with other |
| 960 |
codecs, serialising a string into a sequence of bytes is known as *encoding*, |
| 961 |
and recreating the string from the sequence of bytes is known as *decoding*. |
| 962 |
There are a variety of different text serialisation codecs, which are |
| 963 |
collectivity referred to as :term:`text encodings <text encoding>`. |
| 964 |
|
| 965 |
The simplest text encoding (called ``'latin-1'`` or ``'iso-8859-1'``) maps |
| 966 |
the code points 0--255 to the bytes ``0x0``--``0xff``, which means that a string |
| 967 |
object that contains code points above ``U+00FF`` can't be encoded with this |
| 968 |
codec. Doing so will raise a :exc:`UnicodeEncodeError` that looks |
| 969 |
like the following (although the details of the error message may differ): |
| 970 |
``UnicodeEncodeError: 'latin-1' codec can't encode character '\u1234' in |
| 971 |
position 3: ordinal not in range(256)``. |
| 972 |
|
| 973 |
There's another group of encodings (the so called charmap encodings) that choose |
| 974 |
a different subset of all Unicode code points and how these code points are |
| 975 |
mapped to the bytes ``0x0``--``0xff``. To see how this is done simply open |
| 976 |
e.g. :file:`encodings/cp1252.py` (which is an encoding that is used primarily on |
| 977 |
Windows). There's a string constant with 256 characters that shows you which |
| 978 |
character is mapped to which byte value. |
| 979 |
|
| 980 |
All of these encodings can only encode 256 of the 1114112 code points |
| 981 |
defined in Unicode. A simple and straightforward way that can store each Unicode |
| 982 |
code point, is to store each code point as four consecutive bytes. There are two |
| 983 |
possibilities: store the bytes in big endian or in little endian order. These |
| 984 |
two encodings are called ``UTF-32-BE`` and ``UTF-32-LE`` respectively. Their |
| 985 |
disadvantage is that if, for example, you use ``UTF-32-BE`` on a little endian |
| 986 |
machine you will always have to swap bytes on encoding and decoding. |
| 987 |
Python's ``UTF-16`` and ``UTF-32`` codecs avoid this problem by using the |
| 988 |
platform's native byte order when no BOM is present. |
| 989 |
Python follows prevailing platform |
| 990 |
practice, so native-endian data round-trips without redundant byte swapping, |
| 991 |
even though the Unicode Standard defaults to big-endian when the byte order is |
| 992 |
unspecified. When these bytes are read by a CPU with a different endianness, |
| 993 |
the bytes have to be swapped. To be able to detect the endianness of a |
| 994 |
``UTF-16`` or ``UTF-32`` byte sequence, a BOM ("Byte Order Mark") is used. |
| 995 |
This is the Unicode character ``U+FEFF``. This character can be prepended to every |
| 996 |
``UTF-16`` or ``UTF-32`` byte sequence. The byte swapped version of this character |
| 997 |
(``0xFFFE``) is an illegal character that may not appear in a Unicode text. |
| 998 |
When the first character of a ``UTF-16`` or ``UTF-32`` byte sequence is |
| 999 |
``U+FFFE``, the bytes have to be swapped on decoding. |
| 1000 |
|
| 1001 |
Unfortunately the character ``U+FEFF`` had a second purpose as |
| 1002 |
a ``ZERO WIDTH NO-BREAK SPACE``: a character that has no width and doesn't allow |
| 1003 |
a word to be split. It can e.g. be used to give hints to a ligature algorithm. |
| 1004 |
With Unicode 4.0 using ``U+FEFF`` as a ``ZERO WIDTH NO-BREAK SPACE`` has been |
| 1005 |
deprecated (with ``U+2060`` (``WORD JOINER``) assuming this role). Nevertheless |
| 1006 |
Unicode software still must be able to handle ``U+FEFF`` in both roles: as a BOM |
| 1007 |
it's a device to determine the storage layout of the encoded bytes, and vanishes |
| 1008 |
once the byte sequence has been decoded into a string; as a ``ZERO WIDTH |
| 1009 |
NO-BREAK SPACE`` it's a normal character that will be decoded like any other. |
| 1010 |
|
| 1011 |
There's another encoding that is able to encode the full range of Unicode |
| 1012 |
characters: UTF-8. UTF-8 is an 8-bit encoding, which means there are no issues |
| 1013 |
with byte order in UTF-8. Each byte in a UTF-8 byte sequence consists of two |
| 1014 |
parts: marker bits (the most significant bits) and payload bits. The marker bits |
| 1015 |
are a sequence of zero to four ``1`` bits followed by a ``0`` bit. Unicode characters are |
| 1016 |
encoded like this (with x being payload bits, which when concatenated give the |
| 1017 |
Unicode character): |
| 1018 |
|
| 1019 |
+-----------------------------------+----------------------------------------------+ |
| 1020 |
| Range | Encoding | |
| 1021 |
+===================================+==============================================+ |
| 1022 |
| ``U-00000000`` ... ``U-0000007F`` | 0xxxxxxx | |
| 1023 |
+-----------------------------------+----------------------------------------------+ |
| 1024 |
| ``U-00000080`` ... ``U-000007FF`` | 110xxxxx 10xxxxxx | |
| 1025 |
+-----------------------------------+----------------------------------------------+ |
| 1026 |
| ``U-00000800`` ... ``U-0000FFFF`` | 1110xxxx 10xxxxxx 10xxxxxx | |
| 1027 |
+-----------------------------------+----------------------------------------------+ |
| 1028 |
| ``U-00010000`` ... ``U-0010FFFF`` | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx | |
| 1029 |
+-----------------------------------+----------------------------------------------+ |
| 1030 |
|
| 1031 |
The least significant bit of the Unicode character is the rightmost x bit. |
| 1032 |
|
| 1033 |
As UTF-8 is an 8-bit encoding no BOM is required and any ``U+FEFF`` character in |
| 1034 |
the decoded string (even if it's the first character) is treated as a ``ZERO |
| 1035 |
WIDTH NO-BREAK SPACE``. |
| 1036 |
|
| 1037 |
Without external information it's impossible to reliably determine which |
| 1038 |
encoding was used for encoding a string. Each charmap encoding can |
| 1039 |
decode any random byte sequence. However that's not possible with UTF-8, as |
| 1040 |
UTF-8 byte sequences have a structure that doesn't allow arbitrary byte |
| 1041 |
sequences. To increase the reliability with which a UTF-8 encoding can be |
| 1042 |
detected, Microsoft invented a variant of UTF-8 (that Python calls |
| 1043 |
``"utf-8-sig"``) for its Notepad program: Before any of the Unicode characters |
| 1044 |
is written to the file, a UTF-8 encoded BOM (which looks like this as a byte |
| 1045 |
sequence: ``0xef``, ``0xbb``, ``0xbf``) is written. As it's rather improbable |
| 1046 |
that any charmap encoded file starts with these byte values (which would e.g. |
| 1047 |
map to |
| 1048 |
|
| 1049 |
| LATIN SMALL LETTER I WITH DIAERESIS |
| 1050 |
| RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK |
| 1051 |
| INVERTED QUESTION MARK |
| 1052 |
|
| 1053 |
in iso-8859-1), this increases the probability that a ``utf-8-sig`` encoding can be |
| 1054 |
correctly guessed from the byte sequence. So here the BOM is not used to be able |
| 1055 |
to determine the byte order used for generating the byte sequence, but as a |
| 1056 |
signature that helps in guessing the encoding. On encoding the utf-8-sig codec |
| 1057 |
will write ``0xef``, ``0xbb``, ``0xbf`` as the first three bytes to the file. On |
| 1058 |
decoding ``utf-8-sig`` will skip those three bytes if they appear as the first |
| 1059 |
three bytes in the file. In UTF-8, the use of the BOM is discouraged and |
| 1060 |
should generally be avoided. |
| 1061 |
|
| 1062 |
|
| 1063 |
.. _standard-encodings: |
| 1064 |
|
| 1065 |
Standard Encodings |
| 1066 |
------------------ |
| 1067 |
|
| 1068 |
Python comes with a number of codecs built-in, either implemented as C functions |
| 1069 |
or with dictionaries as mapping tables. The following table lists the codecs by |
| 1070 |
name, together with a few common aliases, and the languages for which the |
| 1071 |
encoding is likely used. Neither the list of aliases nor the list of languages |
| 1072 |
is meant to be exhaustive. Notice that spelling alternatives that only differ in |
| 1073 |
case or use a hyphen instead of an underscore are also valid aliases |
| 1074 |
because they are equivalent when normalized by |
| 1075 |
:func:`~encodings.normalize_encoding`. For example, ``'utf-8'`` is a valid |
| 1076 |
alias for the ``'utf_8'`` codec. |
| 1077 |
|
| 1078 |
.. note:: |
| 1079 |
|
| 1080 |
The below table lists the most common aliases, for a complete list |
| 1081 |
refer to the source :source:`aliases.py <Lib/encodings/aliases.py>` file. |
| 1082 |
|
| 1083 |
On Windows, ``cpXXX`` codecs are available for all code pages. |
| 1084 |
But only codecs listed in the following table are guarantead to exist on |
| 1085 |
other platforms. |
| 1086 |
|
| 1087 |
.. impl-detail:: |
| 1088 |
|
| 1089 |
Some common encodings can bypass the codecs lookup machinery to |
| 1090 |
improve performance. These optimization opportunities are only |
| 1091 |
recognized by CPython for a limited set of (case insensitive) |
| 1092 |
aliases: utf-8, utf8, latin-1, latin1, iso-8859-1, iso8859-1, mbcs |
| 1093 |
(Windows only), ascii, us-ascii, utf-16, utf16, utf-32, utf32, and |
| 1094 |
the same using underscores instead of dashes. Using alternative |
| 1095 |
aliases for these encodings may result in slower execution. |
| 1096 |
|
| 1097 |
.. versionchanged:: 3.6 |
| 1098 |
Optimization opportunity recognized for us-ascii. |
| 1099 |
|
| 1100 |
Many of the character sets support the same languages. They vary in individual |
| 1101 |
characters (e.g. whether the EURO SIGN is supported or not), and in the |
| 1102 |
assignment of characters to code positions. For the European languages in |
| 1103 |
particular, the following variants typically exist: |
| 1104 |
|
| 1105 |
* an ISO 8859 codeset |
| 1106 |
|
| 1107 |
* a Microsoft Windows code page, which is typically derived from an 8859 codeset, |
| 1108 |
but replaces control characters with additional graphic characters |
| 1109 |
|
| 1110 |
* an IBM EBCDIC code page |
| 1111 |
|
| 1112 |
* an IBM PC code page, which is ASCII compatible |
| 1113 |
|
| 1114 |
.. tabularcolumns:: |l|p{0.3\linewidth}|p{0.3\linewidth}| |
| 1115 |
|
| 1116 |
+-----------------+--------------------------------+--------------------------------+ |
| 1117 |
| Codec | Aliases | Languages | |
| 1118 |
+=================+================================+================================+ |
| 1119 |
| ascii | 646, us-ascii | English | |
| 1120 |
+-----------------+--------------------------------+--------------------------------+ |
| 1121 |
| big5 | big5-tw, csbig5 | Traditional Chinese | |
| 1122 |
+-----------------+--------------------------------+--------------------------------+ |
| 1123 |
| big5hkscs | big5-hkscs, hkscs | Traditional Chinese | |
| 1124 |
+-----------------+--------------------------------+--------------------------------+ |
| 1125 |
| cp037 | IBM037, IBM039 | English | |
| 1126 |
+-----------------+--------------------------------+--------------------------------+ |
| 1127 |
| cp273 | 273, IBM273, csIBM273 | German | |
| 1128 |
| | | | |
| 1129 |
| | | .. versionadded:: 3.4 | |
| 1130 |
+-----------------+--------------------------------+--------------------------------+ |
| 1131 |
| cp424 | EBCDIC-CP-HE, IBM424 | Hebrew | |
| 1132 |
+-----------------+--------------------------------+--------------------------------+ |
| 1133 |
| cp437 | 437, IBM437 | English | |
| 1134 |
+-----------------+--------------------------------+--------------------------------+ |
| 1135 |
| cp500 | EBCDIC-CP-BE, EBCDIC-CP-CH, | Western Europe | |
| 1136 |
| | IBM500 | | |
| 1137 |
+-----------------+--------------------------------+--------------------------------+ |
| 1138 |
| cp720 | | Arabic | |
| 1139 |
+-----------------+--------------------------------+--------------------------------+ |
| 1140 |
| cp737 | | Greek | |
| 1141 |
+-----------------+--------------------------------+--------------------------------+ |
| 1142 |
| cp775 | IBM775 | Baltic languages | |
| 1143 |
+-----------------+--------------------------------+--------------------------------+ |
| 1144 |
| cp850 | 850, IBM850 | Western Europe | |
| 1145 |
+-----------------+--------------------------------+--------------------------------+ |
| 1146 |
| cp852 | 852, IBM852 | Central and Eastern Europe | |
| 1147 |
+-----------------+--------------------------------+--------------------------------+ |
| 1148 |
| cp855 | 855, IBM855 | Belarusian, Bulgarian, | |
| 1149 |
| | | Macedonian, Russian, Serbian | |
| 1150 |
+-----------------+--------------------------------+--------------------------------+ |
| 1151 |
| cp856 | | Hebrew | |
| 1152 |
+-----------------+--------------------------------+--------------------------------+ |
| 1153 |
| cp857 | 857, IBM857 | Turkish | |
| 1154 |
+-----------------+--------------------------------+--------------------------------+ |
| 1155 |
| cp858 | 858, IBM858 | Western Europe | |
| 1156 |
+-----------------+--------------------------------+--------------------------------+ |
| 1157 |
| cp860 | 860, IBM860 | Portuguese | |
| 1158 |
+-----------------+--------------------------------+--------------------------------+ |
| 1159 |
| cp861 | 861, CP-IS, IBM861 | Icelandic | |
| 1160 |
+-----------------+--------------------------------+--------------------------------+ |
| 1161 |
| cp862 | 862, IBM862 | Hebrew | |
| 1162 |
+-----------------+--------------------------------+--------------------------------+ |
| 1163 |
| cp863 | 863, IBM863 | Canadian | |
| 1164 |
+-----------------+--------------------------------+--------------------------------+ |
| 1165 |
| cp864 | IBM864 | Arabic | |
| 1166 |
+-----------------+--------------------------------+--------------------------------+ |
| 1167 |
| cp865 | 865, IBM865 | Danish, Norwegian | |
| 1168 |
+-----------------+--------------------------------+--------------------------------+ |
| 1169 |
| cp866 | 866, IBM866 | Russian | |
| 1170 |
+-----------------+--------------------------------+--------------------------------+ |
| 1171 |
| cp869 | 869, CP-GR, IBM869 | Greek | |
| 1172 |
+-----------------+--------------------------------+--------------------------------+ |
| 1173 |
| cp874 | | Thai | |
| 1174 |
+-----------------+--------------------------------+--------------------------------+ |
| 1175 |
| cp875 | | Greek | |
| 1176 |
+-----------------+--------------------------------+--------------------------------+ |
| 1177 |
| cp932 | 932, ms932, mskanji, ms-kanji, | Japanese | |
| 1178 |
| | windows-31j | | |
| 1179 |
+-----------------+--------------------------------+--------------------------------+ |
| 1180 |
| cp949 | 949, ms949, uhc | Korean | |
| 1181 |
+-----------------+--------------------------------+--------------------------------+ |
| 1182 |
| cp950 | 950, ms950 | Traditional Chinese | |
| 1183 |
+-----------------+--------------------------------+--------------------------------+ |
| 1184 |
| cp1006 | | Urdu | |
| 1185 |
+-----------------+--------------------------------+--------------------------------+ |
| 1186 |
| cp1026 | ibm1026 | Turkish | |
| 1187 |
+-----------------+--------------------------------+--------------------------------+ |
| 1188 |
| cp1125 | 1125, ibm1125, cp866u, ruscii | Ukrainian | |
| 1189 |
| | | | |
| 1190 |
| | | .. versionadded:: 3.4 | |
| 1191 |
+-----------------+--------------------------------+--------------------------------+ |
| 1192 |
| cp1140 | ibm1140 | Western Europe | |
| 1193 |
+-----------------+--------------------------------+--------------------------------+ |
| 1194 |
| cp1250 | windows-1250 | Central and Eastern Europe | |
| 1195 |
+-----------------+--------------------------------+--------------------------------+ |
| 1196 |
| cp1251 | windows-1251 | Belarusian, Bulgarian, | |
| 1197 |
| | | Macedonian, Russian, Serbian | |
| 1198 |
+-----------------+--------------------------------+--------------------------------+ |
| 1199 |
| cp1252 | windows-1252 | Western Europe | |
| 1200 |
+-----------------+--------------------------------+--------------------------------+ |
| 1201 |
| cp1253 | windows-1253 | Greek | |
| 1202 |
+-----------------+--------------------------------+--------------------------------+ |
| 1203 |
| cp1254 | windows-1254 | Turkish | |
| 1204 |
+-----------------+--------------------------------+--------------------------------+ |
| 1205 |
| cp1255 | windows-1255 | Hebrew | |
| 1206 |
+-----------------+--------------------------------+--------------------------------+ |
| 1207 |
| cp1256 | windows-1256 | Arabic | |
| 1208 |
+-----------------+--------------------------------+--------------------------------+ |
| 1209 |
| cp1257 | windows-1257 | Baltic languages | |
| 1210 |
+-----------------+--------------------------------+--------------------------------+ |
| 1211 |
| cp1258 | windows-1258 | Vietnamese | |
| 1212 |
+-----------------+--------------------------------+--------------------------------+ |
| 1213 |
| euc_jp | eucjp, ujis, u-jis | Japanese | |
| 1214 |
+-----------------+--------------------------------+--------------------------------+ |
| 1215 |
| euc_jis_2004 | jisx0213, eucjis2004 | Japanese | |
| 1216 |
+-----------------+--------------------------------+--------------------------------+ |
| 1217 |
| euc_jisx0213 | eucjisx0213 | Japanese | |
| 1218 |
+-----------------+--------------------------------+--------------------------------+ |
| 1219 |
| euc_kr | euckr, korean, ksc5601, | Korean | |
| 1220 |
| | ks_c-5601, ks_c-5601-1987, | | |
| 1221 |
| | ksx1001, ks_x-1001 | | |
| 1222 |
+-----------------+--------------------------------+--------------------------------+ |
| 1223 |
| gb2312 | chinese, csiso58gb231280, | Simplified Chinese | |
| 1224 |
| | euc-cn, euccn, eucgb2312-cn, | | |
| 1225 |
| | gb2312-1980, gb2312-80, | | |
| 1226 |
| | iso-ir-58 | | |
| 1227 |
+-----------------+--------------------------------+--------------------------------+ |
| 1228 |
| gbk | 936, cp936, ms936 | Unified Chinese | |
| 1229 |
+-----------------+--------------------------------+--------------------------------+ |
| 1230 |
| gb18030 | gb18030-2000 | Unified Chinese | |
| 1231 |
+-----------------+--------------------------------+--------------------------------+ |
| 1232 |
| hz | hzgb, hz-gb, hz-gb-2312 | Simplified Chinese | |
| 1233 |
+-----------------+--------------------------------+--------------------------------+ |
| 1234 |
| iso2022_jp | csiso2022jp, iso2022jp, | Japanese | |
| 1235 |
| | iso-2022-jp | | |
| 1236 |
+-----------------+--------------------------------+--------------------------------+ |
| 1237 |
| iso2022_jp_1 | iso2022jp-1, iso-2022-jp-1 | Japanese | |
| 1238 |
+-----------------+--------------------------------+--------------------------------+ |
| 1239 |
| iso2022_jp_2 | iso2022jp-2, iso-2022-jp-2 | Japanese, Korean, Simplified | |
| 1240 |
| | | Chinese, Western Europe, Greek | |
| 1241 |
+-----------------+--------------------------------+--------------------------------+ |
| 1242 |
| iso2022_jp_2004 | iso2022jp-2004, | Japanese | |
| 1243 |
| | iso-2022-jp-2004 | | |
| 1244 |
+-----------------+--------------------------------+--------------------------------+ |
| 1245 |
| iso2022_jp_3 | iso2022jp-3, iso-2022-jp-3 | Japanese | |
| 1246 |
+-----------------+--------------------------------+--------------------------------+ |
| 1247 |
| iso2022_jp_ext | iso2022jp-ext, iso-2022-jp-ext | Japanese | |
| 1248 |
+-----------------+--------------------------------+--------------------------------+ |
| 1249 |
| iso2022_kr | csiso2022kr, iso2022kr, | Korean | |
| 1250 |
| | iso-2022-kr | | |
| 1251 |
+-----------------+--------------------------------+--------------------------------+ |
| 1252 |
| latin_1 | iso-8859-1, iso8859-1, 8859, | Western Europe | |
| 1253 |
| | cp819, latin, latin1, L1 | | |
| 1254 |
+-----------------+--------------------------------+--------------------------------+ |
| 1255 |
| iso8859_2 | iso-8859-2, latin2, L2 | Central and Eastern Europe | |
| 1256 |
+-----------------+--------------------------------+--------------------------------+ |
| 1257 |
| iso8859_3 | iso-8859-3, latin3, L3 | Esperanto, Maltese | |
| 1258 |
+-----------------+--------------------------------+--------------------------------+ |
| 1259 |
| iso8859_4 | iso-8859-4, latin4, L4 | Northern Europe | |
| 1260 |
+-----------------+--------------------------------+--------------------------------+ |
| 1261 |
| iso8859_5 | iso-8859-5, cyrillic | Belarusian, Bulgarian, | |
| 1262 |
| | | Macedonian, Russian, Serbian | |
| 1263 |
+-----------------+--------------------------------+--------------------------------+ |
| 1264 |
| iso8859_6 | iso-8859-6, arabic | Arabic | |
| 1265 |
+-----------------+--------------------------------+--------------------------------+ |
| 1266 |
| iso8859_7 | iso-8859-7, greek, greek8 | Greek | |
| 1267 |
+-----------------+--------------------------------+--------------------------------+ |
| 1268 |
| iso8859_8 | iso-8859-8, hebrew | Hebrew | |
| 1269 |
+-----------------+--------------------------------+--------------------------------+ |
| 1270 |
| iso8859_9 | iso-8859-9, latin5, L5 | Turkish | |
| 1271 |
+-----------------+--------------------------------+--------------------------------+ |
| 1272 |
| iso8859_10 | iso-8859-10, latin6, L6 | Nordic languages | |
| 1273 |
+-----------------+--------------------------------+--------------------------------+ |
| 1274 |
| iso8859_11 | iso-8859-11, thai | Thai languages | |
| 1275 |
+-----------------+--------------------------------+--------------------------------+ |
| 1276 |
| iso8859_13 | iso-8859-13, latin7, L7 | Baltic languages | |
| 1277 |
+-----------------+--------------------------------+--------------------------------+ |
| 1278 |
| iso8859_14 | iso-8859-14, latin8, L8 | Celtic languages | |
| 1279 |
+-----------------+--------------------------------+--------------------------------+ |
| 1280 |
| iso8859_15 | iso-8859-15, latin9, L9 | Western Europe | |
| 1281 |
+-----------------+--------------------------------+--------------------------------+ |
| 1282 |
| iso8859_16 | iso-8859-16, latin10, L10 | South-Eastern Europe | |
| 1283 |
+-----------------+--------------------------------+--------------------------------+ |
| 1284 |
| johab | cp1361, ms1361 | Korean | |
| 1285 |
+-----------------+--------------------------------+--------------------------------+ |
| 1286 |
| koi8_r | | Russian | |
| 1287 |
+-----------------+--------------------------------+--------------------------------+ |
| 1288 |
| koi8_t | | Tajik | |
| 1289 |
| | | | |
| 1290 |
| | | .. versionadded:: 3.5 | |
| 1291 |
+-----------------+--------------------------------+--------------------------------+ |
| 1292 |
| koi8_u | | Ukrainian | |
| 1293 |
+-----------------+--------------------------------+--------------------------------+ |
| 1294 |
| kz1048 | kz_1048, strk1048_2002, rk1048 | Kazakh | |
| 1295 |
| | | | |
| 1296 |
| | | .. versionadded:: 3.5 | |
| 1297 |
+-----------------+--------------------------------+--------------------------------+ |
| 1298 |
| mac_cyrillic | maccyrillic | Belarusian, Bulgarian, | |
| 1299 |
| | | Macedonian, Russian, Serbian | |
| 1300 |
+-----------------+--------------------------------+--------------------------------+ |
| 1301 |
| mac_greek | macgreek | Greek | |
| 1302 |
+-----------------+--------------------------------+--------------------------------+ |
| 1303 |
| mac_iceland | maciceland | Icelandic | |
| 1304 |
+-----------------+--------------------------------+--------------------------------+ |
| 1305 |
| mac_latin2 | maclatin2, maccentraleurope, | Central and Eastern Europe | |
| 1306 |
| | mac_centeuro | | |
| 1307 |
+-----------------+--------------------------------+--------------------------------+ |
| 1308 |
| mac_roman | macroman, macintosh | Western Europe | |
| 1309 |
+-----------------+--------------------------------+--------------------------------+ |
| 1310 |
| mac_turkish | macturkish | Turkish | |
| 1311 |
+-----------------+--------------------------------+--------------------------------+ |
| 1312 |
| ptcp154 | csptcp154, pt154, cp154, | Kazakh | |
| 1313 |
| | cyrillic-asian | | |
| 1314 |
+-----------------+--------------------------------+--------------------------------+ |
| 1315 |
| shift_jis | csshiftjis, shiftjis, sjis, | Japanese | |
| 1316 |
| | s_jis | | |
| 1317 |
+-----------------+--------------------------------+--------------------------------+ |
| 1318 |
| shift_jis_2004 | shiftjis2004, sjis_2004, | Japanese | |
| 1319 |
| | sjis2004 | | |
| 1320 |
+-----------------+--------------------------------+--------------------------------+ |
| 1321 |
| shift_jisx0213 | shiftjisx0213, sjisx0213, | Japanese | |
| 1322 |
| | s_jisx0213 | | |
| 1323 |
+-----------------+--------------------------------+--------------------------------+ |
| 1324 |
| utf_32 | U32, utf32 | all languages | |
| 1325 |
+-----------------+--------------------------------+--------------------------------+ |
| 1326 |
| utf_32_be | UTF-32BE | all languages | |
| 1327 |
+-----------------+--------------------------------+--------------------------------+ |
| 1328 |
| utf_32_le | UTF-32LE | all languages | |
| 1329 |
+-----------------+--------------------------------+--------------------------------+ |
| 1330 |
| utf_16 | U16, utf16 | all languages | |
| 1331 |
+-----------------+--------------------------------+--------------------------------+ |
| 1332 |
| utf_16_be | UTF-16BE | all languages | |
| 1333 |
+-----------------+--------------------------------+--------------------------------+ |
| 1334 |
| utf_16_le | UTF-16LE | all languages | |
| 1335 |
+-----------------+--------------------------------+--------------------------------+ |
| 1336 |
| utf_7 | U7, unicode-1-1-utf-7 | all languages | |
| 1337 |
+-----------------+--------------------------------+--------------------------------+ |
| 1338 |
| utf_8 | U8, UTF, utf8, cp65001 | all languages | |
| 1339 |
+-----------------+--------------------------------+--------------------------------+ |
| 1340 |
| utf_8_sig | | all languages | |
| 1341 |
+-----------------+--------------------------------+--------------------------------+ |
| 1342 |
|
| 1343 |
.. versionchanged:: 3.4 |
| 1344 |
The utf-16\* and utf-32\* encoders no longer allow surrogate code points |
| 1345 |
(``U+D800``--``U+DFFF``) to be encoded. |
| 1346 |
The utf-32\* decoders no longer decode |
| 1347 |
byte sequences that correspond to surrogate code points. |
| 1348 |
|
| 1349 |
.. versionchanged:: 3.8 |
| 1350 |
``cp65001`` is now an alias to ``utf_8``. |
| 1351 |
|
| 1352 |
.. versionchanged:: 3.14 |
| 1353 |
On Windows, ``cpXXX`` codecs are now available for all code pages. |
| 1354 |
|
| 1355 |
|
| 1356 |
Python Specific Encodings |
| 1357 |
------------------------- |
| 1358 |
|
| 1359 |
A number of predefined codecs are specific to Python, so their codec names have |
| 1360 |
no meaning outside Python. These are listed in the tables below based on the |
| 1361 |
expected input and output types (note that while text encodings are the most |
| 1362 |
common use case for codecs, the underlying codec infrastructure supports |
| 1363 |
arbitrary data transforms rather than just text encodings). For asymmetric |
| 1364 |
codecs, the stated meaning describes the encoding direction. |
| 1365 |
|
| 1366 |
Text Encodings |
| 1367 |
^^^^^^^^^^^^^^ |
| 1368 |
|
| 1369 |
The following codecs provide :class:`str` to :class:`bytes` encoding and |
| 1370 |
:term:`bytes-like object` to :class:`str` decoding, similar to the Unicode text |
| 1371 |
encodings. |
| 1372 |
|
| 1373 |
.. tabularcolumns:: |l|p{0.3\linewidth}|p{0.3\linewidth}| |
| 1374 |
|
| 1375 |
+--------------------+---------+---------------------------+ |
| 1376 |
| Codec | Aliases | Meaning | |
| 1377 |
+====================+=========+===========================+ |
| 1378 |
| idna | | Implement :rfc:`3490`, | |
| 1379 |
| | | see also | |
| 1380 |
| | | :mod:`encodings.idna`. | |
| 1381 |
| | | Only ``errors='strict'`` | |
| 1382 |
| | | is supported. | |
| 1383 |
+--------------------+---------+---------------------------+ |
| 1384 |
| mbcs | ansi, | Windows only: Encode the | |
| 1385 |
| | dbcs | operand according to the | |
| 1386 |
| | | ANSI codepage (CP_ACP). | |
| 1387 |
+--------------------+---------+---------------------------+ |
| 1388 |
| oem | | Windows only: Encode the | |
| 1389 |
| | | operand according to the | |
| 1390 |
| | | OEM codepage (CP_OEMCP). | |
| 1391 |
| | | | |
| 1392 |
| | | .. versionadded:: 3.6 | |
| 1393 |
+--------------------+---------+---------------------------+ |
| 1394 |
| palmos | | Encoding of PalmOS 3.5. | |
| 1395 |
+--------------------+---------+---------------------------+ |
| 1396 |
| punycode | | Implement :rfc:`3492`. | |
| 1397 |
| | | Stateful codecs are not | |
| 1398 |
| | | supported. | |
| 1399 |
+--------------------+---------+---------------------------+ |
| 1400 |
| raw_unicode_escape | | Latin-1 encoding with | |
| 1401 |
| | | :samp:`\\u{XXXX}` and | |
| 1402 |
| | | :samp:`\\U{XXXXXXXX}` | |
| 1403 |
| | | for other code points. | |
| 1404 |
| | | Existing | |
| 1405 |
| | | backslashes are not | |
| 1406 |
| | | escaped in any way. | |
| 1407 |
| | | It is used in the Python | |
| 1408 |
| | | pickle protocol. | |
| 1409 |
+--------------------+---------+---------------------------+ |
| 1410 |
| undefined | | This Codec should only | |
| 1411 |
| | | be used for testing | |
| 1412 |
| | | purposes. | |
| 1413 |
| | | | |
| 1414 |
| | | Raise an exception for | |
| 1415 |
| | | all conversions, even | |
| 1416 |
| | | empty strings. The error | |
| 1417 |
| | | handler is ignored. | |
| 1418 |
+--------------------+---------+---------------------------+ |
| 1419 |
| unicode_escape | | Encoding suitable as the | |
| 1420 |
| | | contents of a Unicode | |
| 1421 |
| | | literal in ASCII-encoded | |
| 1422 |
| | | Python source code, | |
| 1423 |
| | | except that quotes are | |
| 1424 |
| | | not escaped. Decode | |
| 1425 |
| | | from Latin-1 source code. | |
| 1426 |
| | | Beware that Python source | |
| 1427 |
| | | code actually uses UTF-8 | |
| 1428 |
| | | by default. | |
| 1429 |
+--------------------+---------+---------------------------+ |
| 1430 |
|
| 1431 |
.. versionchanged:: 3.8 |
| 1432 |
"unicode_internal" codec is removed. |
| 1433 |
|
| 1434 |
|
| 1435 |
.. _binary-transforms: |
| 1436 |
|
| 1437 |
Binary Transforms |
| 1438 |
^^^^^^^^^^^^^^^^^ |
| 1439 |
|
| 1440 |
The following codecs provide binary transforms: :term:`bytes-like object` |
| 1441 |
to :class:`bytes` mappings. They are not supported by :meth:`bytes.decode` |
| 1442 |
(which only produces :class:`str` output). |
| 1443 |
|
| 1444 |
|
| 1445 |
.. tabularcolumns:: |l|L|L|L| |
| 1446 |
|
| 1447 |
+----------------------+------------------+------------------------------+------------------------------+ |
| 1448 |
| Codec | Aliases | Meaning | Encoder / decoder | |
| 1449 |
+======================+==================+==============================+==============================+ |
| 1450 |
| base64_codec [#b64]_ | base64, base_64 | Convert the operand to | :meth:`base64.encodebytes` / | |
| 1451 |
| | | multiline MIME base64 (the | :meth:`base64.decodebytes` | |
| 1452 |
| | | result always includes a | | |
| 1453 |
| | | trailing ``'\n'``). | | |
| 1454 |
| | | | | |
| 1455 |
| | | .. versionchanged:: 3.4 | | |
| 1456 |
| | | accepts any | | |
| 1457 |
| | | :term:`bytes-like object` | | |
| 1458 |
| | | as input for encoding and | | |
| 1459 |
| | | decoding | | |
| 1460 |
+----------------------+------------------+------------------------------+------------------------------+ |
| 1461 |
| bz2_codec | bz2 | Compress the operand using | :meth:`bz2.compress` / | |
| 1462 |
| | | bz2. | :meth:`bz2.decompress` | |
| 1463 |
+----------------------+------------------+------------------------------+------------------------------+ |
| 1464 |
| hex_codec | hex | Convert the operand to | :meth:`binascii.b2a_hex` / | |
| 1465 |
| | | hexadecimal | :meth:`binascii.a2b_hex` | |
| 1466 |
| | | representation, with two | | |
| 1467 |
| | | digits per byte. | | |
| 1468 |
+----------------------+------------------+------------------------------+------------------------------+ |
| 1469 |
| quopri_codec | quopri, | Convert the operand to MIME | :meth:`quopri.encode` with | |
| 1470 |
| | quotedprintable, | quoted printable. | ``quotetabs=True`` / | |
| 1471 |
| | quoted_printable | | :meth:`quopri.decode` | |
| 1472 |
+----------------------+------------------+------------------------------+------------------------------+ |
| 1473 |
| uu_codec | uu | Convert the operand using | | |
| 1474 |
| | | uuencode. | | |
| 1475 |
+----------------------+------------------+------------------------------+------------------------------+ |
| 1476 |
| zlib_codec | zip, zlib | Compress the operand using | :meth:`zlib.compress` / | |
| 1477 |
| | | gzip. | :meth:`zlib.decompress` | |
| 1478 |
+----------------------+------------------+------------------------------+------------------------------+ |
| 1479 |
|
| 1480 |
.. [#b64] In addition to :term:`bytes-like objects <bytes-like object>`, |
| 1481 |
``'base64_codec'`` also accepts ASCII-only instances of :class:`str` for |
| 1482 |
decoding |
| 1483 |
|
| 1484 |
.. versionadded:: 3.2 |
| 1485 |
Restoration of the binary transforms. |
| 1486 |
|
| 1487 |
.. versionchanged:: 3.4 |
| 1488 |
Restoration of the aliases for the binary transforms. |
| 1489 |
|
| 1490 |
|
| 1491 |
.. _standalone-codec-functions: |
| 1492 |
|
| 1493 |
Standalone Codec Functions |
| 1494 |
^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 1495 |
|
| 1496 |
The following functions provide encoding and decoding functionality similar to codecs, |
| 1497 |
but are not available as named codecs through :func:`codecs.encode` or :func:`codecs.decode`. |
| 1498 |
They are used internally (for example, by :mod:`pickle`) and behave similarly to the |
| 1499 |
``string_escape`` codec that was removed in Python 3. |
| 1500 |
|
| 1501 |
.. function:: codecs.escape_encode(input, errors=None) |
| 1502 |
|
| 1503 |
Encode *input* using escape sequences. Similar to how :func:`repr` on bytes |
| 1504 |
produces escaped byte values. |
| 1505 |
|
| 1506 |
*input* must be a :class:`bytes` object. |
| 1507 |
|
| 1508 |
Returns a tuple ``(output, length)`` where *output* is a :class:`bytes` |
| 1509 |
object and *length* is the number of bytes consumed. |
| 1510 |
|
| 1511 |
.. function:: codecs.escape_decode(input, errors=None) |
| 1512 |
|
| 1513 |
Decode *input* from escape sequences back to the original bytes. |
| 1514 |
|
| 1515 |
*input* must be a :term:`bytes-like object`. |
| 1516 |
|
| 1517 |
Returns a tuple ``(output, length)`` where *output* is a :class:`bytes` |
| 1518 |
object and *length* is the number of bytes consumed. |
| 1519 |
|
| 1520 |
|
| 1521 |
.. _text-transforms: |
| 1522 |
|
| 1523 |
Text Transforms |
| 1524 |
^^^^^^^^^^^^^^^ |
| 1525 |
|
| 1526 |
The following codec provides a text transform: a :class:`str` to :class:`str` |
| 1527 |
mapping. It is not supported by :meth:`str.encode` (which only produces |
| 1528 |
:class:`bytes` output). |
| 1529 |
|
| 1530 |
.. tabularcolumns:: |l|l|L| |
| 1531 |
|
| 1532 |
+--------------------+---------+---------------------------+ |
| 1533 |
| Codec | Aliases | Meaning | |
| 1534 |
+====================+=========+===========================+ |
| 1535 |
| rot_13 | rot13 | Return the Caesar-cypher | |
| 1536 |
| | | encryption of the | |
| 1537 |
| | | operand. | |
| 1538 |
+--------------------+---------+---------------------------+ |
| 1539 |
|
| 1540 |
.. versionadded:: 3.2 |
| 1541 |
Restoration of the ``rot_13`` text transform. |
| 1542 |
|
| 1543 |
.. versionchanged:: 3.4 |
| 1544 |
Restoration of the ``rot13`` alias. |
| 1545 |
|
| 1546 |
|
| 1547 |
:mod:`!encodings` --- Encodings package |
| 1548 |
--------------------------------------- |
| 1549 |
|
| 1550 |
.. module:: encodings |
| 1551 |
:synopsis: Encodings package |
| 1552 |
|
| 1553 |
This module implements the following functions: |
| 1554 |
|
| 1555 |
.. function:: normalize_encoding(encoding) |
| 1556 |
|
| 1557 |
Normalize encoding name *encoding*. |
| 1558 |
|
| 1559 |
Normalization works as follows: all non-alphanumeric characters except the |
| 1560 |
dot used for Python package names are collapsed and replaced with a single |
| 1561 |
underscore, leading and trailing underscores are removed. |
| 1562 |
For example, ``' -;#'`` becomes ``'_'``. |
| 1563 |
|
| 1564 |
Note that *encoding* should be ASCII only. |
| 1565 |
|
| 1566 |
|
| 1567 |
.. note:: |
| 1568 |
The following functions should not be used directly, except for testing |
| 1569 |
purposes; :func:`codecs.lookup` should be used instead. |
| 1570 |
|
| 1571 |
|
| 1572 |
.. function:: search_function(encoding) |
| 1573 |
|
| 1574 |
Search for the codec module corresponding to the given encoding name |
| 1575 |
*encoding*. |
| 1576 |
|
| 1577 |
This function first normalizes the *encoding* using |
| 1578 |
:func:`normalize_encoding`, then looks for a corresponding alias. |
| 1579 |
It attempts to import a codec module from the encodings package using either |
| 1580 |
the alias or the normalized name. If the module is found and defines a valid |
| 1581 |
``getregentry()`` function that returns a :class:`codecs.CodecInfo` object, |
| 1582 |
the codec is cached and returned. |
| 1583 |
|
| 1584 |
If the codec module defines a ``getaliases()`` function any returned aliases |
| 1585 |
are registered for future use. |
| 1586 |
|
| 1587 |
|
| 1588 |
.. function:: win32_code_page_search_function(encoding) |
| 1589 |
|
| 1590 |
Search for a Windows code page encoding *encoding* of the form ``cpXXXX``. |
| 1591 |
|
| 1592 |
If the code page is valid and supported, return a :class:`codecs.CodecInfo` |
| 1593 |
object for it. |
| 1594 |
|
| 1595 |
.. availability:: Windows. |
| 1596 |
|
| 1597 |
.. versionadded:: 3.14 |
| 1598 |
|
| 1599 |
|
| 1600 |
This module implements the following exception: |
| 1601 |
|
| 1602 |
.. exception:: CodecRegistryError |
| 1603 |
|
| 1604 |
Raised when a codec is invalid or incompatible. |
| 1605 |
|
| 1606 |
|
| 1607 |
:mod:`!encodings.idna` --- Internationalized Domain Names in Applications |
| 1608 |
------------------------------------------------------------------------- |
| 1609 |
|
| 1610 |
.. module:: encodings.idna |
| 1611 |
:synopsis: Internationalized Domain Names implementation |
| 1612 |
.. moduleauthor:: Martin v. Löwis |
| 1613 |
|
| 1614 |
This module implements :rfc:`3490` (Internationalized Domain Names in |
| 1615 |
Applications) and :rfc:`3492` (Nameprep: A Stringprep Profile for |
| 1616 |
Internationalized Domain Names (IDN)). It builds upon the ``punycode`` encoding |
| 1617 |
and :mod:`stringprep`. |
| 1618 |
|
| 1619 |
If you need the IDNA 2008 standard from :rfc:`5891` and :rfc:`5895`, use the |
| 1620 |
third-party :pypi:`idna` module. |
| 1621 |
|
| 1622 |
These RFCs together define a protocol to support non-ASCII characters in domain |
| 1623 |
names. A domain name containing non-ASCII characters (such as |
| 1624 |
``www.Alliancefrançaise.nu``) is converted into an ASCII-compatible encoding |
| 1625 |
(ACE, such as ``www.xn--alliancefranaise-npb.nu``). The ACE form of the domain |
| 1626 |
name is then used in all places where arbitrary characters are not allowed by |
| 1627 |
the protocol, such as DNS queries, HTTP :mailheader:`Host` fields, and so |
| 1628 |
on. This conversion is carried out in the application; if possible invisible to |
| 1629 |
the user: The application should transparently convert Unicode domain labels to |
| 1630 |
IDNA on the wire, and convert back ACE labels to Unicode before presenting them |
| 1631 |
to the user. |
| 1632 |
|
| 1633 |
Python supports this conversion in several ways: the ``idna`` codec performs |
| 1634 |
conversion between Unicode and ACE, separating an input string into labels |
| 1635 |
based on the separator characters defined in :rfc:`section 3.1 of RFC 3490 <3490#section-3.1>` |
| 1636 |
and converting each label to ACE as required, and conversely separating an input |
| 1637 |
byte string into labels based on the ``.`` separator and converting any ACE |
| 1638 |
labels found into unicode. Furthermore, the :mod:`socket` module |
| 1639 |
transparently converts Unicode host names to ACE, so that applications need not |
| 1640 |
be concerned about converting host names themselves when they pass them to the |
| 1641 |
socket module. On top of that, modules that have host names as function |
| 1642 |
parameters, such as :mod:`http.client` and :mod:`ftplib`, accept Unicode host |
| 1643 |
names (:mod:`http.client` then also transparently sends an IDNA hostname in the |
| 1644 |
:mailheader:`Host` field if it sends that field at all). |
| 1645 |
|
| 1646 |
When receiving host names from the wire (such as in reverse name lookup), no |
| 1647 |
automatic conversion to Unicode is performed: applications wishing to present |
| 1648 |
such host names to the user should decode them to Unicode. |
| 1649 |
|
| 1650 |
The module :mod:`!encodings.idna` also implements the nameprep procedure, which |
| 1651 |
performs certain normalizations on host names, to achieve case-insensitivity of |
| 1652 |
international domain names, and to unify similar characters. The nameprep |
| 1653 |
functions can be used directly if desired. |
| 1654 |
|
| 1655 |
|
| 1656 |
.. function:: nameprep(label) |
| 1657 |
|
| 1658 |
Return the nameprepped version of *label*. The implementation currently assumes |
| 1659 |
query strings, so ``AllowUnassigned`` is true. |
| 1660 |
|
| 1661 |
|
| 1662 |
.. function:: ToASCII(label) |
| 1663 |
|
| 1664 |
Convert a label to ASCII, as specified in :rfc:`3490`. ``UseSTD3ASCIIRules`` is |
| 1665 |
assumed to be false. |
| 1666 |
|
| 1667 |
|
| 1668 |
.. function:: ToUnicode(label) |
| 1669 |
|
| 1670 |
Convert a label to Unicode, as specified in :rfc:`3490`. |
| 1671 |
|
| 1672 |
|
| 1673 |
:mod:`!encodings.mbcs` --- Windows ANSI codepage |
| 1674 |
------------------------------------------------ |
| 1675 |
|
| 1676 |
.. module:: encodings.mbcs |
| 1677 |
:synopsis: Windows ANSI codepage |
| 1678 |
|
| 1679 |
This module implements the ANSI codepage (CP_ACP). |
| 1680 |
|
| 1681 |
.. availability:: Windows. |
| 1682 |
|
| 1683 |
.. versionchanged:: 3.2 |
| 1684 |
Before 3.2, the *errors* argument was ignored; ``'replace'`` was always used |
| 1685 |
to encode, and ``'ignore'`` to decode. |
| 1686 |
|
| 1687 |
.. versionchanged:: 3.3 |
| 1688 |
Support any error handler. |
| 1689 |
|
| 1690 |
|
| 1691 |
:mod:`!encodings.utf_8_sig` --- UTF-8 codec with BOM signature |
| 1692 |
-------------------------------------------------------------- |
| 1693 |
|
| 1694 |
.. module:: encodings.utf_8_sig |
| 1695 |
:synopsis: UTF-8 codec with BOM signature |
| 1696 |
.. moduleauthor:: Walter Dörwald |
| 1697 |
|
| 1698 |
This module implements a variant of the UTF-8 codec. On encoding, a UTF-8 encoded |
| 1699 |
BOM will be prepended to the UTF-8 encoded bytes. For the stateful encoder this |
| 1700 |
is only done once (on the first write to the byte stream). On decoding, an |
| 1701 |
optional UTF-8 encoded BOM at the start of the data will be skipped. |