Debian Codegraph @

Doc/library/zlib.rst

raw
1 :mod:`!zlib` --- Compression compatible with :program:`gzip`
2 ============================================================
3
4 .. module:: zlib
5 :synopsis: Low-level interface to compression and decompression routines
6 compatible with gzip.
7
8 --------------
9
10 For applications that require data compression, the functions in this module
11 allow compression and decompression, using the zlib library. The zlib library
12 has its own home page at https://www.zlib.net. There are known
13 incompatibilities between the Python module and versions of the zlib library
14 earlier than 1.1.3; 1.1.3 has a `security vulnerability <https://zlib.net/zlib_faq.html#faq33>`_, so we recommend using
15 1.1.4 or later.
16
17 zlib's functions have many options and often need to be used in a particular
18 order. This documentation doesn't attempt to cover all of the permutations;
19 consult the `zlib manual <https://www.zlib.net/manual.html>`_ for authoritative
20 information.
21
22 For reading and writing ``.gz`` files see the :mod:`gzip` module.
23
24 The available exception and functions in this module are:
25
26
27 .. exception:: error
28
29 Exception raised on compression and decompression errors.
30
31
32 .. function:: adler32(data[, value])
33
34 Computes an Adler-32 checksum of *data*. (An Adler-32 checksum is almost as
35 reliable as a CRC32 but can be computed much more quickly.) The result
36 is an unsigned 32-bit integer. If *value* is present, it is used as
37 the starting value of the checksum; otherwise, a default value of 1
38 is used. Passing in *value* allows computing a running checksum over the
39 concatenation of several inputs. The algorithm is not cryptographically
40 strong, and should not be used for authentication or digital signatures. Since
41 the algorithm is designed for use as a checksum algorithm, it is not suitable
42 for use as a general hash algorithm.
43
44 .. versionchanged:: 3.0
45 The result is always unsigned.
46
47 .. function:: compress(data, /, level=Z_DEFAULT_COMPRESSION, wbits=MAX_WBITS)
48
49 Compresses the bytes in *data*, returning a bytes object containing compressed data.
50 *level* is an integer from ``0`` to ``9`` or ``-1`` controlling the level of compression;
51 See :const:`Z_BEST_SPEED` (``1``), :const:`Z_BEST_COMPRESSION` (``9``),
52 :const:`Z_NO_COMPRESSION` (``0``), and the default,
53 :const:`Z_DEFAULT_COMPRESSION` (``-1``) for more information about these values.
54
55 .. _compress-wbits:
56
57 The *wbits* argument controls the size of the history buffer (or the
58 "window size") used when compressing data, and whether a header and
59 trailer is included in the output. It can take several ranges of values,
60 defaulting to ``15`` (:const:`MAX_WBITS`):
61
62 * +9 to +15: The base-two logarithm of the window size, which
63 therefore ranges between 512 and 32768. Larger values produce
64 better compression at the expense of greater memory usage. The
65 resulting output will include a zlib-specific header and trailer.
66
67 * −9 to −15: Uses the absolute value of *wbits* as the
68 window size logarithm, while producing a raw output stream with no
69 header or trailing checksum.
70
71 * +25 to +31 = 16 + (9 to 15): Uses the low 4 bits of the value as the
72 window size logarithm, while including a basic :program:`gzip` header
73 and trailing checksum in the output.
74
75 Raises the :exc:`error` exception if any error occurs.
76
77 .. versionchanged:: 3.6
78 *level* can now be used as a keyword parameter.
79
80 .. versionchanged:: 3.11
81 The *wbits* parameter is now available to set window bits and
82 compression type.
83
84 .. function:: compressobj(level=Z_DEFAULT_COMPRESSION, method=DEFLATED, wbits=MAX_WBITS, memLevel=DEF_MEM_LEVEL, strategy=Z_DEFAULT_STRATEGY[, zdict])
85
86 Returns a compression object, to be used for compressing data streams that won't
87 fit into memory at once.
88
89 *level* is the compression level -- an integer from ``0`` to ``9`` or ``-1``.
90 See :const:`Z_BEST_SPEED` (``1``), :const:`Z_BEST_COMPRESSION` (``9``),
91 :const:`Z_NO_COMPRESSION` (``0``), and the default,
92 :const:`Z_DEFAULT_COMPRESSION` (``-1``) for more information about these values.
93
94 *method* is the compression algorithm. Currently, the only supported value is
95 :const:`DEFLATED`.
96
97 The *wbits* parameter controls the size of the history buffer (or the
98 "window size"), and what header and trailer format will be used. It has
99 the same meaning as `described for compress() <#compress-wbits>`__.
100
101 The *memLevel* argument controls the amount of memory used for the
102 internal compression state. Valid values range from ``1`` to ``9``.
103 Higher values use more memory, but are faster and produce smaller output.
104
105 *strategy* is used to tune the compression algorithm. Possible values are
106 :const:`Z_DEFAULT_STRATEGY`, :const:`Z_FILTERED`, :const:`Z_HUFFMAN_ONLY`,
107 :const:`Z_RLE` and :const:`Z_FIXED`.
108
109 *zdict* is a predefined compression dictionary. This is a sequence of bytes
110 (such as a :class:`bytes` object) containing subsequences that are expected
111 to occur frequently in the data that is to be compressed. Those subsequences
112 that are expected to be most common should come at the end of the dictionary.
113
114 .. versionchanged:: 3.3
115 Added the *zdict* parameter and keyword argument support.
116
117
118 .. function:: crc32(data[, value])
119
120 .. index::
121 single: Cyclic Redundancy Check
122 single: checksum; Cyclic Redundancy Check
123
124 Computes a CRC (Cyclic Redundancy Check) checksum of *data*. The
125 result is an unsigned 32-bit integer. If *value* is present, it is used
126 as the starting value of the checksum; otherwise, a default value of 0
127 is used. Passing in *value* allows computing a running checksum over the
128 concatenation of several inputs. The algorithm is not cryptographically
129 strong, and should not be used for authentication or digital signatures. Since
130 the algorithm is designed for use as a checksum algorithm, it is not suitable
131 for use as a general hash algorithm.
132
133 .. versionchanged:: 3.0
134 The result is always unsigned.
135
136 .. function:: decompress(data, /, wbits=MAX_WBITS, bufsize=DEF_BUF_SIZE)
137
138 Decompresses the bytes in *data*, returning a bytes object containing the
139 uncompressed data. The *wbits* parameter depends on
140 the format of *data*, and is discussed further below.
141 If *bufsize* is given, it is used as the initial size of the output
142 buffer. Raises the :exc:`error` exception if any error occurs.
143
144 .. _decompress-wbits:
145
146 The *wbits* parameter controls the size of the history buffer
147 (or "window size"), and what header and trailer format is expected.
148 It is similar to the parameter for :func:`compressobj`, but accepts
149 more ranges of values:
150
151 * +8 to +15: The base-two logarithm of the window size. The input
152 must include a zlib header and trailer.
153
154 * 0: Automatically determine the window size from the zlib header.
155 Only supported since zlib 1.2.3.5.
156
157 * −8 to −15: Uses the absolute value of *wbits* as the window size
158 logarithm. The input must be a raw stream with no header or trailer.
159
160 * +24 to +31 = 16 + (8 to 15): Uses the low 4 bits of the value as
161 the window size logarithm. The input must include a gzip header and
162 trailer.
163
164 * +40 to +47 = 32 + (8 to 15): Uses the low 4 bits of the value as
165 the window size logarithm, and automatically accepts either
166 the zlib or gzip format.
167
168 When decompressing a stream, the window size must not be smaller
169 than the size originally used to compress the stream; using a too-small
170 value may result in an :exc:`error` exception. The default *wbits* value
171 corresponds to the largest window size and requires a zlib header and
172 trailer to be included.
173
174 *bufsize* is the initial size of the buffer used to hold decompressed data. If
175 more space is required, the buffer size will be increased as needed, so you
176 don't have to get this value exactly right; tuning it will only save a few calls
177 to :c:func:`malloc`.
178
179 .. versionchanged:: 3.6
180 *wbits* and *bufsize* can be used as keyword arguments.
181
182 .. function:: decompressobj(wbits=MAX_WBITS[, zdict])
183
184 Returns a decompression object, to be used for decompressing data streams that
185 won't fit into memory at once.
186
187 The *wbits* parameter controls the size of the history buffer (or the
188 "window size"), and what header and trailer format is expected. It has
189 the same meaning as `described for decompress() <#decompress-wbits>`__.
190
191 The *zdict* parameter specifies a predefined compression dictionary. If
192 provided, this must be the same dictionary as was used by the compressor that
193 produced the data that is to be decompressed.
194
195 .. note::
196
197 If *zdict* is a mutable object (such as a :class:`bytearray`), you must not
198 modify its contents between the call to :func:`decompressobj` and the first
199 call to the decompressor's ``decompress()`` method.
200
201 .. versionchanged:: 3.3
202 Added the *zdict* parameter.
203
204
205 Compression objects support the following methods:
206
207
208 .. method:: Compress.compress(data)
209
210 Compress *data*, returning a bytes object containing compressed data for at least
211 part of the data in *data*. This data should be concatenated to the output
212 produced by any preceding calls to the :meth:`compress` method. Some input may
213 be kept in internal buffers for later processing.
214
215
216 .. method:: Compress.flush([mode])
217
218 All pending input is processed, and a bytes object containing the remaining compressed
219 output is returned. *mode* can be selected from the constants
220 :const:`Z_NO_FLUSH`, :const:`Z_PARTIAL_FLUSH`, :const:`Z_SYNC_FLUSH`,
221 :const:`Z_FULL_FLUSH`, :const:`Z_BLOCK`, or :const:`Z_FINISH`,
222 defaulting to :const:`Z_FINISH`. Except :const:`Z_FINISH`, all constants
223 allow compressing further bytestrings of data, while :const:`Z_FINISH` finishes the
224 compressed stream and prevents compressing any more data. After calling :meth:`flush`
225 with *mode* set to :const:`Z_FINISH`, the :meth:`compress` method cannot be called again;
226 the only realistic action is to delete the object.
227
228
229 .. method:: Compress.copy()
230
231 Returns a copy of the compression object. This can be used to efficiently
232 compress a set of data that share a common initial prefix.
233
234
235 .. versionchanged:: 3.8
236 Added :func:`copy.copy` and :func:`copy.deepcopy` support to compression
237 objects.
238
239
240 Decompression objects support the following methods and attributes:
241
242
243 .. attribute:: Decompress.unused_data
244
245 A bytes object which contains any bytes past the end of the compressed data. That is,
246 this remains ``b""`` until the last byte that contains compression data is
247 available. If the whole bytestring turned out to contain compressed data, this is
248 ``b""``, an empty bytes object.
249
250
251 .. attribute:: Decompress.unconsumed_tail
252
253 A bytes object that contains any data that was not consumed by the last
254 :meth:`decompress` call because it exceeded the limit for the uncompressed data
255 buffer. This data has not yet been seen by the zlib machinery, so you must feed
256 it (possibly with further data concatenated to it) back to a subsequent
257 :meth:`decompress` method call in order to get correct output.
258
259
260 .. attribute:: Decompress.eof
261
262 A boolean indicating whether the end of the compressed data stream has been
263 reached.
264
265 This makes it possible to distinguish between a properly formed compressed
266 stream, and an incomplete or truncated one.
267
268 .. versionadded:: 3.3
269
270
271 .. method:: Decompress.decompress(data, max_length=0)
272
273 Decompress *data*, returning a bytes object containing the uncompressed data
274 corresponding to at least part of the data in *string*. This data should be
275 concatenated to the output produced by any preceding calls to the
276 :meth:`decompress` method. Some of the input data may be preserved in internal
277 buffers for later processing.
278
279 If the optional parameter *max_length* is non-zero then the return value will be
280 no longer than *max_length*. This may mean that not all of the compressed input
281 can be processed; and unconsumed data will be stored in the attribute
282 :attr:`unconsumed_tail`. This bytestring must be passed to a subsequent call to
283 :meth:`decompress` if decompression is to continue. If *max_length* is zero
284 then the whole input is decompressed, and :attr:`unconsumed_tail` is empty.
285
286 .. versionchanged:: 3.6
287 *max_length* can be used as a keyword argument.
288
289
290 .. method:: Decompress.flush([length])
291
292 All pending input is processed, and a bytes object containing the remaining
293 uncompressed output is returned. After calling :meth:`flush`, the
294 :meth:`decompress` method cannot be called again; the only realistic action is
295 to delete the object.
296
297 The optional parameter *length* sets the initial size of the output buffer.
298
299
300 .. method:: Decompress.copy()
301
302 Returns a copy of the decompression object. This can be used to save the state
303 of the decompressor midway through the data stream in order to speed up random
304 seeks into the stream at a future point.
305
306
307 .. versionchanged:: 3.8
308 Added :func:`copy.copy` and :func:`copy.deepcopy` support to decompression
309 objects.
310
311
312 The following constants are available to configure compression and decompression
313 behavior:
314
315 .. data:: DEFLATED
316
317 The deflate compression method.
318
319
320 .. data:: MAX_WBITS
321
322 The maximum window size, expressed as a power of 2.
323 For example, if :const:`!MAX_WBITS` is ``15`` it results in a window size
324 of ``32 KiB``.
325
326
327 .. data:: DEF_MEM_LEVEL
328
329 The default memory level for compression objects.
330
331
332 .. data:: DEF_BUF_SIZE
333
334 The default buffer size for decompression operations.
335
336
337 .. data:: Z_NO_COMPRESSION
338
339 Compression level ``0``; no compression.
340
341 .. versionadded:: 3.6
342
343
344 .. data:: Z_BEST_SPEED
345
346 Compression level ``1``; fastest and produces the least compression.
347
348
349 .. data:: Z_BEST_COMPRESSION
350
351 Compression level ``9``; slowest and produces the most compression.
352
353
354 .. data:: Z_DEFAULT_COMPRESSION
355
356 Default compression level (``-1``); a compromise between speed and
357 compression. Currently equivalent to compression level ``6``.
358
359
360 .. data:: Z_DEFAULT_STRATEGY
361
362 Default compression strategy, for normal data.
363
364
365 .. data:: Z_FILTERED
366
367 Compression strategy for data produced by a filter (or predictor).
368
369
370 .. data:: Z_HUFFMAN_ONLY
371
372 Compression strategy that forces Huffman coding only.
373
374
375 .. data:: Z_RLE
376
377 Compression strategy that limits match distances to one (run-length encoding).
378
379 This constant is only available if Python was compiled with zlib
380 1.2.0.1 or greater.
381
382 .. versionadded:: 3.6
383
384
385 .. data:: Z_FIXED
386
387 Compression strategy that prevents the use of dynamic Huffman codes.
388
389 This constant is only available if Python was compiled with zlib
390 1.2.2.2 or greater.
391
392 .. versionadded:: 3.6
393
394
395 .. data:: Z_NO_FLUSH
396
397 Flush mode ``0``. No special flushing behavior.
398
399 .. versionadded:: 3.6
400
401
402 .. data:: Z_PARTIAL_FLUSH
403
404 Flush mode ``1``. Flush as much output as possible.
405
406
407 .. data:: Z_SYNC_FLUSH
408
409 Flush mode ``2``. All output is flushed and the output is aligned to a byte boundary.
410
411
412 .. data:: Z_FULL_FLUSH
413
414 Flush mode ``3``. All output is flushed and the compression state is reset.
415
416
417 .. data:: Z_FINISH
418
419 Flush mode ``4``. All pending input is processed, no more input is expected.
420
421
422 .. data:: Z_BLOCK
423
424 Flush mode ``5``. A deflate block is completed and emitted.
425
426 This constant is only available if Python was compiled with zlib
427 1.2.2.2 or greater.
428
429 .. versionadded:: 3.6
430
431
432 .. data:: Z_TREES
433
434 Flush mode ``6``, for inflate operations. Instructs inflate to return when
435 it gets to the next deflate block boundary.
436
437 This constant is only available if Python was compiled with zlib
438 1.2.3.4 or greater.
439
440 .. versionadded:: 3.6
441
442
443 Information about the version of the zlib library in use is available through
444 the following constants:
445
446
447 .. data:: ZLIB_VERSION
448
449 The version string of the zlib library that was used for building the module.
450 This may be different from the zlib library actually used at runtime, which
451 is available as :const:`ZLIB_RUNTIME_VERSION`.
452
453
454 .. data:: ZLIB_RUNTIME_VERSION
455
456 The version string of the zlib library actually loaded by the interpreter.
457
458 .. versionadded:: 3.3
459
460
461 .. seealso::
462
463 Module :mod:`gzip`
464 Reading and writing :program:`gzip`\ -format files.
465
466 https://www.zlib.net
467 The zlib library home page.
468
469 https://www.zlib.net/manual.html
470 The zlib manual explains the semantics and usage of the library's many
471 functions.
472
473 In case gzip (de)compression is a bottleneck, the `python-isal`_
474 package speeds up (de)compression with a mostly compatible API.
475
476 .. _python-isal: https://github.com/pycompression/python-isal