| 1 |
:mod:`!zlib` --- Compression compatible with :program:`gzip` |
| 2 |
============================================================ |
| 3 |
|
| 4 |
.. module:: zlib |
| 5 |
:synopsis: Low-level interface to compression and decompression routines |
| 6 |
compatible with gzip. |
| 7 |
|
| 8 |
-------------- |
| 9 |
|
| 10 |
For applications that require data compression, the functions in this module |
| 11 |
allow compression and decompression, using the `zlib library <https://www.zlib.net>`_. |
| 12 |
|
| 13 |
.. include:: ../includes/optional-module.rst |
| 14 |
|
| 15 |
zlib's functions have many options and often need to be used in a particular |
| 16 |
order. This documentation doesn't attempt to cover all of the permutations; |
| 17 |
consult the `zlib manual <https://www.zlib.net/manual.html>`_ for authoritative |
| 18 |
information. |
| 19 |
|
| 20 |
For reading and writing ``.gz`` files see the :mod:`gzip` module. |
| 21 |
|
| 22 |
The available exception and functions in this module are: |
| 23 |
|
| 24 |
|
| 25 |
.. exception:: error |
| 26 |
|
| 27 |
Exception raised on compression and decompression errors. |
| 28 |
|
| 29 |
|
| 30 |
.. function:: adler32(data[, value]) |
| 31 |
|
| 32 |
Computes an Adler-32 checksum of *data*. (An Adler-32 checksum is almost as |
| 33 |
reliable as a CRC32 but can be computed much more quickly.) The result |
| 34 |
is an unsigned 32-bit integer. If *value* is present, it is used as |
| 35 |
the starting value of the checksum; otherwise, a default value of 1 |
| 36 |
is used. Passing in *value* allows computing a running checksum over the |
| 37 |
concatenation of several inputs. The algorithm is not cryptographically |
| 38 |
strong, and should not be used for authentication or digital signatures. Since |
| 39 |
the algorithm is designed for use as a checksum algorithm, it is not suitable |
| 40 |
for use as a general hash algorithm. |
| 41 |
|
| 42 |
.. versionchanged:: 3.0 |
| 43 |
The result is always unsigned. |
| 44 |
|
| 45 |
.. function:: compress(data, /, level=Z_DEFAULT_COMPRESSION, wbits=MAX_WBITS) |
| 46 |
|
| 47 |
Compresses the bytes in *data*, returning a bytes object containing compressed data. |
| 48 |
*level* is an integer from ``0`` to ``9`` or ``-1`` controlling the level of compression; |
| 49 |
See :const:`Z_BEST_SPEED` (``1``), :const:`Z_BEST_COMPRESSION` (``9``), |
| 50 |
:const:`Z_NO_COMPRESSION` (``0``), and the default, |
| 51 |
:const:`Z_DEFAULT_COMPRESSION` (``-1``) for more information about these values. |
| 52 |
|
| 53 |
.. _compress-wbits: |
| 54 |
|
| 55 |
The *wbits* argument controls the size of the history buffer (or the |
| 56 |
"window size") used when compressing data, and whether a header and |
| 57 |
trailer is included in the output. It can take several ranges of values, |
| 58 |
defaulting to ``15`` (:const:`MAX_WBITS`): |
| 59 |
|
| 60 |
* +9 to +15: The base-two logarithm of the window size, which |
| 61 |
therefore ranges between 512 and 32768. Larger values produce |
| 62 |
better compression at the expense of greater memory usage. The |
| 63 |
resulting output will include a zlib-specific header and trailer. |
| 64 |
|
| 65 |
* −9 to −15: Uses the absolute value of *wbits* as the |
| 66 |
window size logarithm, while producing a raw output stream with no |
| 67 |
header or trailing checksum. |
| 68 |
|
| 69 |
* +25 to +31 = 16 + (9 to 15): Uses the low 4 bits of the value as the |
| 70 |
window size logarithm, while including a basic :program:`gzip` header |
| 71 |
and trailing checksum in the output. |
| 72 |
|
| 73 |
Raises the :exc:`error` exception if any error occurs. |
| 74 |
|
| 75 |
.. versionchanged:: 3.6 |
| 76 |
*level* can now be used as a keyword parameter. |
| 77 |
|
| 78 |
.. versionchanged:: 3.11 |
| 79 |
The *wbits* parameter is now available to set window bits and |
| 80 |
compression type. |
| 81 |
|
| 82 |
.. function:: compressobj(level=Z_DEFAULT_COMPRESSION, method=DEFLATED, wbits=MAX_WBITS, memLevel=DEF_MEM_LEVEL, strategy=Z_DEFAULT_STRATEGY[, zdict]) |
| 83 |
|
| 84 |
Returns a compression object, to be used for compressing data streams that won't |
| 85 |
fit into memory at once. |
| 86 |
|
| 87 |
*level* is the compression level -- an integer from ``0`` to ``9`` or ``-1``. |
| 88 |
See :const:`Z_BEST_SPEED` (``1``), :const:`Z_BEST_COMPRESSION` (``9``), |
| 89 |
:const:`Z_NO_COMPRESSION` (``0``), and the default, |
| 90 |
:const:`Z_DEFAULT_COMPRESSION` (``-1``) for more information about these values. |
| 91 |
|
| 92 |
*method* is the compression algorithm. Currently, the only supported value is |
| 93 |
:const:`DEFLATED`. |
| 94 |
|
| 95 |
The *wbits* parameter controls the size of the history buffer (or the |
| 96 |
"window size"), and what header and trailer format will be used. It has |
| 97 |
the same meaning as `described for compress() <#compress-wbits>`__. |
| 98 |
|
| 99 |
The *memLevel* argument controls the amount of memory used for the |
| 100 |
internal compression state. Valid values range from ``1`` to ``9``. |
| 101 |
Higher values use more memory, but are faster and produce smaller output. |
| 102 |
|
| 103 |
*strategy* is used to tune the compression algorithm. Possible values are |
| 104 |
:const:`Z_DEFAULT_STRATEGY`, :const:`Z_FILTERED`, :const:`Z_HUFFMAN_ONLY`, |
| 105 |
:const:`Z_RLE` and :const:`Z_FIXED`. |
| 106 |
|
| 107 |
*zdict* is a predefined compression dictionary. This is a sequence of bytes |
| 108 |
(such as a :class:`bytes` object) containing subsequences that are expected |
| 109 |
to occur frequently in the data that is to be compressed. Those subsequences |
| 110 |
that are expected to be most common should come at the end of the dictionary. |
| 111 |
|
| 112 |
.. versionchanged:: 3.3 |
| 113 |
Added the *zdict* parameter and keyword argument support. |
| 114 |
|
| 115 |
|
| 116 |
.. function:: crc32(data[, value]) |
| 117 |
|
| 118 |
.. index:: |
| 119 |
single: Cyclic Redundancy Check |
| 120 |
single: checksum; Cyclic Redundancy Check |
| 121 |
|
| 122 |
Computes a CRC (Cyclic Redundancy Check) checksum of *data*. The |
| 123 |
result is an unsigned 32-bit integer. If *value* is present, it is used |
| 124 |
as the starting value of the checksum; otherwise, a default value of 0 |
| 125 |
is used. Passing in *value* allows computing a running checksum over the |
| 126 |
concatenation of several inputs. The algorithm is not cryptographically |
| 127 |
strong, and should not be used for authentication or digital signatures. Since |
| 128 |
the algorithm is designed for use as a checksum algorithm, it is not suitable |
| 129 |
for use as a general hash algorithm. |
| 130 |
|
| 131 |
.. versionchanged:: 3.0 |
| 132 |
The result is always unsigned. |
| 133 |
|
| 134 |
.. function:: decompress(data, /, wbits=MAX_WBITS, bufsize=DEF_BUF_SIZE) |
| 135 |
|
| 136 |
Decompresses the bytes in *data*, returning a bytes object containing the |
| 137 |
uncompressed data. The *wbits* parameter depends on |
| 138 |
the format of *data*, and is discussed further below. |
| 139 |
If *bufsize* is given, it is used as the initial size of the output |
| 140 |
buffer. Raises the :exc:`error` exception if any error occurs. |
| 141 |
|
| 142 |
.. _decompress-wbits: |
| 143 |
|
| 144 |
The *wbits* parameter controls the size of the history buffer |
| 145 |
(or "window size"), and what header and trailer format is expected. |
| 146 |
It is similar to the parameter for :func:`compressobj`, but accepts |
| 147 |
more ranges of values: |
| 148 |
|
| 149 |
* +8 to +15: The base-two logarithm of the window size. The input |
| 150 |
must include a zlib header and trailer. |
| 151 |
|
| 152 |
* 0: Automatically determine the window size from the zlib header. |
| 153 |
Only supported since zlib 1.2.3.5. |
| 154 |
|
| 155 |
* −8 to −15: Uses the absolute value of *wbits* as the window size |
| 156 |
logarithm. The input must be a raw stream with no header or trailer. |
| 157 |
|
| 158 |
* +24 to +31 = 16 + (8 to 15): Uses the low 4 bits of the value as |
| 159 |
the window size logarithm. The input must include a gzip header and |
| 160 |
trailer. |
| 161 |
|
| 162 |
* +40 to +47 = 32 + (8 to 15): Uses the low 4 bits of the value as |
| 163 |
the window size logarithm, and automatically accepts either |
| 164 |
the zlib or gzip format. |
| 165 |
|
| 166 |
When decompressing a stream, the window size must not be smaller |
| 167 |
than the size originally used to compress the stream; using a too-small |
| 168 |
value may result in an :exc:`error` exception. The default *wbits* value |
| 169 |
corresponds to the largest window size and requires a zlib header and |
| 170 |
trailer to be included. |
| 171 |
|
| 172 |
*bufsize* is the initial size of the buffer used to hold decompressed data. If |
| 173 |
more space is required, the buffer size will be increased as needed, so you |
| 174 |
don't have to get this value exactly right; tuning it will only save a few calls |
| 175 |
to :c:func:`malloc`. |
| 176 |
|
| 177 |
.. versionchanged:: 3.6 |
| 178 |
*wbits* and *bufsize* can be used as keyword arguments. |
| 179 |
|
| 180 |
.. function:: decompressobj(wbits=MAX_WBITS[, zdict]) |
| 181 |
|
| 182 |
Returns a decompression object, to be used for decompressing data streams that |
| 183 |
won't fit into memory at once. |
| 184 |
|
| 185 |
The *wbits* parameter controls the size of the history buffer (or the |
| 186 |
"window size"), and what header and trailer format is expected. It has |
| 187 |
the same meaning as `described for decompress() <#decompress-wbits>`__. |
| 188 |
|
| 189 |
The *zdict* parameter specifies a predefined compression dictionary. If |
| 190 |
provided, this must be the same dictionary as was used by the compressor that |
| 191 |
produced the data that is to be decompressed. |
| 192 |
|
| 193 |
.. note:: |
| 194 |
|
| 195 |
If *zdict* is a mutable object (such as a :class:`bytearray`), you must not |
| 196 |
modify its contents between the call to :func:`decompressobj` and the first |
| 197 |
call to the decompressor's ``decompress()`` method. |
| 198 |
|
| 199 |
.. versionchanged:: 3.3 |
| 200 |
Added the *zdict* parameter. |
| 201 |
|
| 202 |
|
| 203 |
Compression objects support the following methods: |
| 204 |
|
| 205 |
|
| 206 |
.. method:: Compress.compress(data) |
| 207 |
|
| 208 |
Compress *data*, returning a bytes object containing compressed data for at least |
| 209 |
part of the data in *data*. This data should be concatenated to the output |
| 210 |
produced by any preceding calls to the :meth:`compress` method. Some input may |
| 211 |
be kept in internal buffers for later processing. |
| 212 |
|
| 213 |
|
| 214 |
.. method:: Compress.flush([mode]) |
| 215 |
|
| 216 |
All pending input is processed, and a bytes object containing the remaining compressed |
| 217 |
output is returned. *mode* can be selected from the constants |
| 218 |
:const:`Z_NO_FLUSH`, :const:`Z_PARTIAL_FLUSH`, :const:`Z_SYNC_FLUSH`, |
| 219 |
:const:`Z_FULL_FLUSH`, :const:`Z_BLOCK`, or :const:`Z_FINISH`, |
| 220 |
defaulting to :const:`Z_FINISH`. Except :const:`Z_FINISH`, all constants |
| 221 |
allow compressing further bytestrings of data, while :const:`Z_FINISH` finishes the |
| 222 |
compressed stream and prevents compressing any more data. After calling :meth:`flush` |
| 223 |
with *mode* set to :const:`Z_FINISH`, the :meth:`compress` method cannot be called again; |
| 224 |
the only realistic action is to delete the object. |
| 225 |
|
| 226 |
|
| 227 |
.. method:: Compress.copy() |
| 228 |
|
| 229 |
Returns a copy of the compression object. This can be used to efficiently |
| 230 |
compress a set of data that share a common initial prefix. |
| 231 |
|
| 232 |
|
| 233 |
.. versionchanged:: 3.8 |
| 234 |
Added :func:`copy.copy` and :func:`copy.deepcopy` support to compression |
| 235 |
objects. |
| 236 |
|
| 237 |
|
| 238 |
Decompression objects support the following methods and attributes: |
| 239 |
|
| 240 |
|
| 241 |
.. attribute:: Decompress.unused_data |
| 242 |
|
| 243 |
A bytes object which contains any bytes past the end of the compressed data. That is, |
| 244 |
this remains ``b""`` until the last byte that contains compression data is |
| 245 |
available. If the whole bytestring turned out to contain compressed data, this is |
| 246 |
``b""``, an empty bytes object. |
| 247 |
|
| 248 |
|
| 249 |
.. attribute:: Decompress.unconsumed_tail |
| 250 |
|
| 251 |
A bytes object that contains any data that was not consumed by the last |
| 252 |
:meth:`decompress` call because it exceeded the limit for the uncompressed data |
| 253 |
buffer. This data has not yet been seen by the zlib machinery, so you must feed |
| 254 |
it (possibly with further data concatenated to it) back to a subsequent |
| 255 |
:meth:`decompress` method call in order to get correct output. |
| 256 |
|
| 257 |
|
| 258 |
.. attribute:: Decompress.eof |
| 259 |
|
| 260 |
A boolean indicating whether the end of the compressed data stream has been |
| 261 |
reached. |
| 262 |
|
| 263 |
This makes it possible to distinguish between a properly formed compressed |
| 264 |
stream, and an incomplete or truncated one. |
| 265 |
|
| 266 |
.. versionadded:: 3.3 |
| 267 |
|
| 268 |
|
| 269 |
.. method:: Decompress.decompress(data, max_length=0) |
| 270 |
|
| 271 |
Decompress *data*, returning a bytes object containing the uncompressed data |
| 272 |
corresponding to at least part of the data in *string*. This data should be |
| 273 |
concatenated to the output produced by any preceding calls to the |
| 274 |
:meth:`decompress` method. Some of the input data may be preserved in internal |
| 275 |
buffers for later processing. |
| 276 |
|
| 277 |
If the optional parameter *max_length* is non-zero then the return value will be |
| 278 |
no longer than *max_length*. This may mean that not all of the compressed input |
| 279 |
can be processed; and unconsumed data will be stored in the attribute |
| 280 |
:attr:`unconsumed_tail`. This bytestring must be passed to a subsequent call to |
| 281 |
:meth:`decompress` if decompression is to continue. If *max_length* is zero |
| 282 |
then the whole input is decompressed, and :attr:`unconsumed_tail` is empty. |
| 283 |
|
| 284 |
.. versionchanged:: 3.6 |
| 285 |
*max_length* can be used as a keyword argument. |
| 286 |
|
| 287 |
|
| 288 |
.. method:: Decompress.flush([length]) |
| 289 |
|
| 290 |
All pending input is processed, and a bytes object containing the remaining |
| 291 |
uncompressed output is returned. After calling :meth:`flush`, the |
| 292 |
:meth:`decompress` method cannot be called again; the only realistic action is |
| 293 |
to delete the object. |
| 294 |
|
| 295 |
The optional parameter *length* sets the initial size of the output buffer. |
| 296 |
|
| 297 |
|
| 298 |
.. method:: Decompress.copy() |
| 299 |
|
| 300 |
Returns a copy of the decompression object. This can be used to save the state |
| 301 |
of the decompressor midway through the data stream in order to speed up random |
| 302 |
seeks into the stream at a future point. |
| 303 |
|
| 304 |
|
| 305 |
.. versionchanged:: 3.8 |
| 306 |
Added :func:`copy.copy` and :func:`copy.deepcopy` support to decompression |
| 307 |
objects. |
| 308 |
|
| 309 |
|
| 310 |
The following constants are available to configure compression and decompression |
| 311 |
behavior: |
| 312 |
|
| 313 |
.. data:: DEFLATED |
| 314 |
|
| 315 |
The deflate compression method. |
| 316 |
|
| 317 |
|
| 318 |
.. data:: MAX_WBITS |
| 319 |
|
| 320 |
The maximum window size, expressed as a power of 2. |
| 321 |
For example, if :const:`!MAX_WBITS` is ``15`` it results in a window size |
| 322 |
of ``32 KiB``. |
| 323 |
|
| 324 |
|
| 325 |
.. data:: DEF_MEM_LEVEL |
| 326 |
|
| 327 |
The default memory level for compression objects. |
| 328 |
|
| 329 |
|
| 330 |
.. data:: DEF_BUF_SIZE |
| 331 |
|
| 332 |
The default buffer size for decompression operations. |
| 333 |
|
| 334 |
|
| 335 |
.. data:: Z_NO_COMPRESSION |
| 336 |
|
| 337 |
Compression level ``0``; no compression. |
| 338 |
|
| 339 |
.. versionadded:: 3.6 |
| 340 |
|
| 341 |
|
| 342 |
.. data:: Z_BEST_SPEED |
| 343 |
|
| 344 |
Compression level ``1``; fastest and produces the least compression. |
| 345 |
|
| 346 |
|
| 347 |
.. data:: Z_BEST_COMPRESSION |
| 348 |
|
| 349 |
Compression level ``9``; slowest and produces the most compression. |
| 350 |
|
| 351 |
|
| 352 |
.. data:: Z_DEFAULT_COMPRESSION |
| 353 |
|
| 354 |
Default compression level (``-1``); a compromise between speed and |
| 355 |
compression. Currently equivalent to compression level ``6``. |
| 356 |
|
| 357 |
|
| 358 |
.. data:: Z_DEFAULT_STRATEGY |
| 359 |
|
| 360 |
Default compression strategy, for normal data. |
| 361 |
|
| 362 |
|
| 363 |
.. data:: Z_FILTERED |
| 364 |
|
| 365 |
Compression strategy for data produced by a filter (or predictor). |
| 366 |
|
| 367 |
|
| 368 |
.. data:: Z_HUFFMAN_ONLY |
| 369 |
|
| 370 |
Compression strategy that forces Huffman coding only. |
| 371 |
|
| 372 |
|
| 373 |
.. data:: Z_RLE |
| 374 |
|
| 375 |
Compression strategy that limits match distances to one (run-length encoding). |
| 376 |
|
| 377 |
This constant is only available if Python was compiled with zlib |
| 378 |
1.2.0.1 or greater. |
| 379 |
|
| 380 |
.. versionadded:: 3.6 |
| 381 |
|
| 382 |
|
| 383 |
.. data:: Z_FIXED |
| 384 |
|
| 385 |
Compression strategy that prevents the use of dynamic Huffman codes. |
| 386 |
|
| 387 |
This constant is only available if Python was compiled with zlib |
| 388 |
1.2.2.2 or greater. |
| 389 |
|
| 390 |
.. versionadded:: 3.6 |
| 391 |
|
| 392 |
|
| 393 |
.. data:: Z_NO_FLUSH |
| 394 |
|
| 395 |
Flush mode ``0``. No special flushing behavior. |
| 396 |
|
| 397 |
.. versionadded:: 3.6 |
| 398 |
|
| 399 |
|
| 400 |
.. data:: Z_PARTIAL_FLUSH |
| 401 |
|
| 402 |
Flush mode ``1``. Flush as much output as possible. |
| 403 |
|
| 404 |
|
| 405 |
.. data:: Z_SYNC_FLUSH |
| 406 |
|
| 407 |
Flush mode ``2``. All output is flushed and the output is aligned to a byte boundary. |
| 408 |
|
| 409 |
|
| 410 |
.. data:: Z_FULL_FLUSH |
| 411 |
|
| 412 |
Flush mode ``3``. All output is flushed and the compression state is reset. |
| 413 |
|
| 414 |
|
| 415 |
.. data:: Z_FINISH |
| 416 |
|
| 417 |
Flush mode ``4``. All pending input is processed, no more input is expected. |
| 418 |
|
| 419 |
|
| 420 |
.. data:: Z_BLOCK |
| 421 |
|
| 422 |
Flush mode ``5``. A deflate block is completed and emitted. |
| 423 |
|
| 424 |
This constant is only available if Python was compiled with zlib |
| 425 |
1.2.2.2 or greater. |
| 426 |
|
| 427 |
.. versionadded:: 3.6 |
| 428 |
|
| 429 |
|
| 430 |
.. data:: Z_TREES |
| 431 |
|
| 432 |
Flush mode ``6``, for inflate operations. Instructs inflate to return when |
| 433 |
it gets to the next deflate block boundary. |
| 434 |
|
| 435 |
This constant is only available if Python was compiled with zlib |
| 436 |
1.2.3.4 or greater. |
| 437 |
|
| 438 |
.. versionadded:: 3.6 |
| 439 |
|
| 440 |
|
| 441 |
Information about the version of the zlib library in use is available through |
| 442 |
the following constants: |
| 443 |
|
| 444 |
|
| 445 |
.. data:: ZLIB_VERSION |
| 446 |
|
| 447 |
The version string of the zlib library that was used for building the module. |
| 448 |
This may be different from the zlib library actually used at runtime, which |
| 449 |
is available as :const:`ZLIB_RUNTIME_VERSION`. |
| 450 |
|
| 451 |
|
| 452 |
.. data:: ZLIB_RUNTIME_VERSION |
| 453 |
|
| 454 |
The version string of the zlib library actually loaded by the interpreter. |
| 455 |
|
| 456 |
.. versionadded:: 3.3 |
| 457 |
|
| 458 |
|
| 459 |
.. data:: ZLIBNG_VERSION |
| 460 |
|
| 461 |
The version string of the zlib-ng library that was used for building the |
| 462 |
module if zlib-ng was used. When present, the :data:`ZLIB_VERSION` and |
| 463 |
:data:`ZLIB_RUNTIME_VERSION` constants reflect the version of the zlib API |
| 464 |
provided by zlib-ng. |
| 465 |
|
| 466 |
If zlib-ng was not used to build the module, this constant will be absent. |
| 467 |
|
| 468 |
.. versionadded:: 3.14 |
| 469 |
|
| 470 |
|
| 471 |
.. seealso:: |
| 472 |
|
| 473 |
Module :mod:`gzip` |
| 474 |
Reading and writing :program:`gzip`\ -format files. |
| 475 |
|
| 476 |
https://www.zlib.net |
| 477 |
The zlib library home page. |
| 478 |
|
| 479 |
https://www.zlib.net/manual.html |
| 480 |
The zlib manual explains the semantics and usage of the library's many |
| 481 |
functions. |
| 482 |
|
| 483 |
In case gzip (de)compression is a bottleneck, the `python-isal`_ |
| 484 |
package speeds up (de)compression with a mostly compatible API. |
| 485 |
|
| 486 |
.. _python-isal: https://github.com/pycompression/python-isal |