Debian Codegraph @

Doc/library/zlib.rst

raw
1 :mod:`!zlib` --- Compression compatible with :program:`gzip`
2 ============================================================
3
4 .. module:: zlib
5 :synopsis: Low-level interface to compression and decompression routines
6 compatible with gzip.
7
8 --------------
9
10 For applications that require data compression, the functions in this module
11 allow compression and decompression, using the `zlib library <https://www.zlib.net>`_.
12
13 .. include:: ../includes/optional-module.rst
14
15 zlib's functions have many options and often need to be used in a particular
16 order. This documentation doesn't attempt to cover all of the permutations;
17 consult the `zlib manual <https://www.zlib.net/manual.html>`_ for authoritative
18 information.
19
20 For reading and writing ``.gz`` files see the :mod:`gzip` module.
21
22 The available exception and functions in this module are:
23
24
25 .. exception:: error
26
27 Exception raised on compression and decompression errors.
28
29
30 .. function:: adler32(data[, value])
31
32 Computes an Adler-32 checksum of *data*. (An Adler-32 checksum is almost as
33 reliable as a CRC32 but can be computed much more quickly.) The result
34 is an unsigned 32-bit integer. If *value* is present, it is used as
35 the starting value of the checksum; otherwise, a default value of 1
36 is used. Passing in *value* allows computing a running checksum over the
37 concatenation of several inputs. The algorithm is not cryptographically
38 strong, and should not be used for authentication or digital signatures. Since
39 the algorithm is designed for use as a checksum algorithm, it is not suitable
40 for use as a general hash algorithm.
41
42 .. versionchanged:: 3.0
43 The result is always unsigned.
44
45 .. function:: compress(data, /, level=Z_DEFAULT_COMPRESSION, wbits=MAX_WBITS)
46
47 Compresses the bytes in *data*, returning a bytes object containing compressed data.
48 *level* is an integer from ``0`` to ``9`` or ``-1`` controlling the level of compression;
49 See :const:`Z_BEST_SPEED` (``1``), :const:`Z_BEST_COMPRESSION` (``9``),
50 :const:`Z_NO_COMPRESSION` (``0``), and the default,
51 :const:`Z_DEFAULT_COMPRESSION` (``-1``) for more information about these values.
52
53 .. _compress-wbits:
54
55 The *wbits* argument controls the size of the history buffer (or the
56 "window size") used when compressing data, and whether a header and
57 trailer is included in the output. It can take several ranges of values,
58 defaulting to ``15`` (:const:`MAX_WBITS`):
59
60 * +9 to +15: The base-two logarithm of the window size, which
61 therefore ranges between 512 and 32768. Larger values produce
62 better compression at the expense of greater memory usage. The
63 resulting output will include a zlib-specific header and trailer.
64
65 * −9 to −15: Uses the absolute value of *wbits* as the
66 window size logarithm, while producing a raw output stream with no
67 header or trailing checksum.
68
69 * +25 to +31 = 16 + (9 to 15): Uses the low 4 bits of the value as the
70 window size logarithm, while including a basic :program:`gzip` header
71 and trailing checksum in the output.
72
73 Raises the :exc:`error` exception if any error occurs.
74
75 .. versionchanged:: 3.6
76 *level* can now be used as a keyword parameter.
77
78 .. versionchanged:: 3.11
79 The *wbits* parameter is now available to set window bits and
80 compression type.
81
82 .. function:: compressobj(level=Z_DEFAULT_COMPRESSION, method=DEFLATED, wbits=MAX_WBITS, memLevel=DEF_MEM_LEVEL, strategy=Z_DEFAULT_STRATEGY[, zdict])
83
84 Returns a compression object, to be used for compressing data streams that won't
85 fit into memory at once.
86
87 *level* is the compression level -- an integer from ``0`` to ``9`` or ``-1``.
88 See :const:`Z_BEST_SPEED` (``1``), :const:`Z_BEST_COMPRESSION` (``9``),
89 :const:`Z_NO_COMPRESSION` (``0``), and the default,
90 :const:`Z_DEFAULT_COMPRESSION` (``-1``) for more information about these values.
91
92 *method* is the compression algorithm. Currently, the only supported value is
93 :const:`DEFLATED`.
94
95 The *wbits* parameter controls the size of the history buffer (or the
96 "window size"), and what header and trailer format will be used. It has
97 the same meaning as `described for compress() <#compress-wbits>`__.
98
99 The *memLevel* argument controls the amount of memory used for the
100 internal compression state. Valid values range from ``1`` to ``9``.
101 Higher values use more memory, but are faster and produce smaller output.
102
103 *strategy* is used to tune the compression algorithm. Possible values are
104 :const:`Z_DEFAULT_STRATEGY`, :const:`Z_FILTERED`, :const:`Z_HUFFMAN_ONLY`,
105 :const:`Z_RLE` and :const:`Z_FIXED`.
106
107 *zdict* is a predefined compression dictionary. This is a sequence of bytes
108 (such as a :class:`bytes` object) containing subsequences that are expected
109 to occur frequently in the data that is to be compressed. Those subsequences
110 that are expected to be most common should come at the end of the dictionary.
111
112 .. versionchanged:: 3.3
113 Added the *zdict* parameter and keyword argument support.
114
115
116 .. function:: crc32(data[, value])
117
118 .. index::
119 single: Cyclic Redundancy Check
120 single: checksum; Cyclic Redundancy Check
121
122 Computes a CRC (Cyclic Redundancy Check) checksum of *data*. The
123 result is an unsigned 32-bit integer. If *value* is present, it is used
124 as the starting value of the checksum; otherwise, a default value of 0
125 is used. Passing in *value* allows computing a running checksum over the
126 concatenation of several inputs. The algorithm is not cryptographically
127 strong, and should not be used for authentication or digital signatures. Since
128 the algorithm is designed for use as a checksum algorithm, it is not suitable
129 for use as a general hash algorithm.
130
131 .. versionchanged:: 3.0
132 The result is always unsigned.
133
134 .. function:: decompress(data, /, wbits=MAX_WBITS, bufsize=DEF_BUF_SIZE)
135
136 Decompresses the bytes in *data*, returning a bytes object containing the
137 uncompressed data. The *wbits* parameter depends on
138 the format of *data*, and is discussed further below.
139 If *bufsize* is given, it is used as the initial size of the output
140 buffer. Raises the :exc:`error` exception if any error occurs.
141
142 .. _decompress-wbits:
143
144 The *wbits* parameter controls the size of the history buffer
145 (or "window size"), and what header and trailer format is expected.
146 It is similar to the parameter for :func:`compressobj`, but accepts
147 more ranges of values:
148
149 * +8 to +15: The base-two logarithm of the window size. The input
150 must include a zlib header and trailer.
151
152 * 0: Automatically determine the window size from the zlib header.
153 Only supported since zlib 1.2.3.5.
154
155 * −8 to −15: Uses the absolute value of *wbits* as the window size
156 logarithm. The input must be a raw stream with no header or trailer.
157
158 * +24 to +31 = 16 + (8 to 15): Uses the low 4 bits of the value as
159 the window size logarithm. The input must include a gzip header and
160 trailer.
161
162 * +40 to +47 = 32 + (8 to 15): Uses the low 4 bits of the value as
163 the window size logarithm, and automatically accepts either
164 the zlib or gzip format.
165
166 When decompressing a stream, the window size must not be smaller
167 than the size originally used to compress the stream; using a too-small
168 value may result in an :exc:`error` exception. The default *wbits* value
169 corresponds to the largest window size and requires a zlib header and
170 trailer to be included.
171
172 *bufsize* is the initial size of the buffer used to hold decompressed data. If
173 more space is required, the buffer size will be increased as needed, so you
174 don't have to get this value exactly right; tuning it will only save a few calls
175 to :c:func:`malloc`.
176
177 .. versionchanged:: 3.6
178 *wbits* and *bufsize* can be used as keyword arguments.
179
180 .. function:: decompressobj(wbits=MAX_WBITS[, zdict])
181
182 Returns a decompression object, to be used for decompressing data streams that
183 won't fit into memory at once.
184
185 The *wbits* parameter controls the size of the history buffer (or the
186 "window size"), and what header and trailer format is expected. It has
187 the same meaning as `described for decompress() <#decompress-wbits>`__.
188
189 The *zdict* parameter specifies a predefined compression dictionary. If
190 provided, this must be the same dictionary as was used by the compressor that
191 produced the data that is to be decompressed.
192
193 .. note::
194
195 If *zdict* is a mutable object (such as a :class:`bytearray`), you must not
196 modify its contents between the call to :func:`decompressobj` and the first
197 call to the decompressor's ``decompress()`` method.
198
199 .. versionchanged:: 3.3
200 Added the *zdict* parameter.
201
202
203 Compression objects support the following methods:
204
205
206 .. method:: Compress.compress(data)
207
208 Compress *data*, returning a bytes object containing compressed data for at least
209 part of the data in *data*. This data should be concatenated to the output
210 produced by any preceding calls to the :meth:`compress` method. Some input may
211 be kept in internal buffers for later processing.
212
213
214 .. method:: Compress.flush([mode])
215
216 All pending input is processed, and a bytes object containing the remaining compressed
217 output is returned. *mode* can be selected from the constants
218 :const:`Z_NO_FLUSH`, :const:`Z_PARTIAL_FLUSH`, :const:`Z_SYNC_FLUSH`,
219 :const:`Z_FULL_FLUSH`, :const:`Z_BLOCK`, or :const:`Z_FINISH`,
220 defaulting to :const:`Z_FINISH`. Except :const:`Z_FINISH`, all constants
221 allow compressing further bytestrings of data, while :const:`Z_FINISH` finishes the
222 compressed stream and prevents compressing any more data. After calling :meth:`flush`
223 with *mode* set to :const:`Z_FINISH`, the :meth:`compress` method cannot be called again;
224 the only realistic action is to delete the object.
225
226
227 .. method:: Compress.copy()
228
229 Returns a copy of the compression object. This can be used to efficiently
230 compress a set of data that share a common initial prefix.
231
232
233 .. versionchanged:: 3.8
234 Added :func:`copy.copy` and :func:`copy.deepcopy` support to compression
235 objects.
236
237
238 Decompression objects support the following methods and attributes:
239
240
241 .. attribute:: Decompress.unused_data
242
243 A bytes object which contains any bytes past the end of the compressed data. That is,
244 this remains ``b""`` until the last byte that contains compression data is
245 available. If the whole bytestring turned out to contain compressed data, this is
246 ``b""``, an empty bytes object.
247
248
249 .. attribute:: Decompress.unconsumed_tail
250
251 A bytes object that contains any data that was not consumed by the last
252 :meth:`decompress` call because it exceeded the limit for the uncompressed data
253 buffer. This data has not yet been seen by the zlib machinery, so you must feed
254 it (possibly with further data concatenated to it) back to a subsequent
255 :meth:`decompress` method call in order to get correct output.
256
257
258 .. attribute:: Decompress.eof
259
260 A boolean indicating whether the end of the compressed data stream has been
261 reached.
262
263 This makes it possible to distinguish between a properly formed compressed
264 stream, and an incomplete or truncated one.
265
266 .. versionadded:: 3.3
267
268
269 .. method:: Decompress.decompress(data, max_length=0)
270
271 Decompress *data*, returning a bytes object containing the uncompressed data
272 corresponding to at least part of the data in *string*. This data should be
273 concatenated to the output produced by any preceding calls to the
274 :meth:`decompress` method. Some of the input data may be preserved in internal
275 buffers for later processing.
276
277 If the optional parameter *max_length* is non-zero then the return value will be
278 no longer than *max_length*. This may mean that not all of the compressed input
279 can be processed; and unconsumed data will be stored in the attribute
280 :attr:`unconsumed_tail`. This bytestring must be passed to a subsequent call to
281 :meth:`decompress` if decompression is to continue. If *max_length* is zero
282 then the whole input is decompressed, and :attr:`unconsumed_tail` is empty.
283
284 .. versionchanged:: 3.6
285 *max_length* can be used as a keyword argument.
286
287
288 .. method:: Decompress.flush([length])
289
290 All pending input is processed, and a bytes object containing the remaining
291 uncompressed output is returned. After calling :meth:`flush`, the
292 :meth:`decompress` method cannot be called again; the only realistic action is
293 to delete the object.
294
295 The optional parameter *length* sets the initial size of the output buffer.
296
297
298 .. method:: Decompress.copy()
299
300 Returns a copy of the decompression object. This can be used to save the state
301 of the decompressor midway through the data stream in order to speed up random
302 seeks into the stream at a future point.
303
304
305 .. versionchanged:: 3.8
306 Added :func:`copy.copy` and :func:`copy.deepcopy` support to decompression
307 objects.
308
309
310 The following constants are available to configure compression and decompression
311 behavior:
312
313 .. data:: DEFLATED
314
315 The deflate compression method.
316
317
318 .. data:: MAX_WBITS
319
320 The maximum window size, expressed as a power of 2.
321 For example, if :const:`!MAX_WBITS` is ``15`` it results in a window size
322 of ``32 KiB``.
323
324
325 .. data:: DEF_MEM_LEVEL
326
327 The default memory level for compression objects.
328
329
330 .. data:: DEF_BUF_SIZE
331
332 The default buffer size for decompression operations.
333
334
335 .. data:: Z_NO_COMPRESSION
336
337 Compression level ``0``; no compression.
338
339 .. versionadded:: 3.6
340
341
342 .. data:: Z_BEST_SPEED
343
344 Compression level ``1``; fastest and produces the least compression.
345
346
347 .. data:: Z_BEST_COMPRESSION
348
349 Compression level ``9``; slowest and produces the most compression.
350
351
352 .. data:: Z_DEFAULT_COMPRESSION
353
354 Default compression level (``-1``); a compromise between speed and
355 compression. Currently equivalent to compression level ``6``.
356
357
358 .. data:: Z_DEFAULT_STRATEGY
359
360 Default compression strategy, for normal data.
361
362
363 .. data:: Z_FILTERED
364
365 Compression strategy for data produced by a filter (or predictor).
366
367
368 .. data:: Z_HUFFMAN_ONLY
369
370 Compression strategy that forces Huffman coding only.
371
372
373 .. data:: Z_RLE
374
375 Compression strategy that limits match distances to one (run-length encoding).
376
377 This constant is only available if Python was compiled with zlib
378 1.2.0.1 or greater.
379
380 .. versionadded:: 3.6
381
382
383 .. data:: Z_FIXED
384
385 Compression strategy that prevents the use of dynamic Huffman codes.
386
387 This constant is only available if Python was compiled with zlib
388 1.2.2.2 or greater.
389
390 .. versionadded:: 3.6
391
392
393 .. data:: Z_NO_FLUSH
394
395 Flush mode ``0``. No special flushing behavior.
396
397 .. versionadded:: 3.6
398
399
400 .. data:: Z_PARTIAL_FLUSH
401
402 Flush mode ``1``. Flush as much output as possible.
403
404
405 .. data:: Z_SYNC_FLUSH
406
407 Flush mode ``2``. All output is flushed and the output is aligned to a byte boundary.
408
409
410 .. data:: Z_FULL_FLUSH
411
412 Flush mode ``3``. All output is flushed and the compression state is reset.
413
414
415 .. data:: Z_FINISH
416
417 Flush mode ``4``. All pending input is processed, no more input is expected.
418
419
420 .. data:: Z_BLOCK
421
422 Flush mode ``5``. A deflate block is completed and emitted.
423
424 This constant is only available if Python was compiled with zlib
425 1.2.2.2 or greater.
426
427 .. versionadded:: 3.6
428
429
430 .. data:: Z_TREES
431
432 Flush mode ``6``, for inflate operations. Instructs inflate to return when
433 it gets to the next deflate block boundary.
434
435 This constant is only available if Python was compiled with zlib
436 1.2.3.4 or greater.
437
438 .. versionadded:: 3.6
439
440
441 Information about the version of the zlib library in use is available through
442 the following constants:
443
444
445 .. data:: ZLIB_VERSION
446
447 The version string of the zlib library that was used for building the module.
448 This may be different from the zlib library actually used at runtime, which
449 is available as :const:`ZLIB_RUNTIME_VERSION`.
450
451
452 .. data:: ZLIB_RUNTIME_VERSION
453
454 The version string of the zlib library actually loaded by the interpreter.
455
456 .. versionadded:: 3.3
457
458
459 .. data:: ZLIBNG_VERSION
460
461 The version string of the zlib-ng library that was used for building the
462 module if zlib-ng was used. When present, the :data:`ZLIB_VERSION` and
463 :data:`ZLIB_RUNTIME_VERSION` constants reflect the version of the zlib API
464 provided by zlib-ng.
465
466 If zlib-ng was not used to build the module, this constant will be absent.
467
468 .. versionadded:: 3.14
469
470
471 .. seealso::
472
473 Module :mod:`gzip`
474 Reading and writing :program:`gzip`\ -format files.
475
476 https://www.zlib.net
477 The zlib library home page.
478
479 https://www.zlib.net/manual.html
480 The zlib manual explains the semantics and usage of the library's many
481 functions.
482
483 In case gzip (de)compression is a bottleneck, the `python-isal`_
484 package speeds up (de)compression with a mostly compatible API.
485
486 .. _python-isal: https://github.com/pycompression/python-isal