Debian Codegraph @

Doc/library/compression.zstd.rst

raw
1 :mod:`!compression.zstd` --- Compression compatible with the Zstandard format
2 =============================================================================
3
4 .. module:: compression.zstd
5 :synopsis: Low-level interface to compression and decompression routines in
6 the zstd library.
7
8 .. versionadded:: 3.14
9
10 **Source code:** :source:`Lib/compression/zstd/__init__.py`
11
12 --------------
13
14 This module provides classes and functions for compressing and decompressing
15 data using the Zstandard (or *zstd*) compression algorithm. The
16 `zstd manual <https://facebook.github.io/zstd/doc/api_manual_latest.html>`__
17 describes Zstandard as "a fast lossless compression algorithm, targeting
18 real-time compression scenarios at zlib-level and better compression ratios."
19 Also included is a file interface that supports reading and writing the
20 contents of ``.zst`` files created by the :program:`zstd` utility, as well as
21 raw zstd compressed streams.
22
23 The :mod:`!compression.zstd` module contains:
24
25 * The :func:`.open` function and :class:`ZstdFile` class for reading and
26 writing compressed files.
27 * The :class:`ZstdCompressor` and :class:`ZstdDecompressor` classes for
28 incremental (de)compression.
29 * The :func:`compress` and :func:`decompress` functions for one-shot
30 (de)compression.
31 * The :func:`train_dict` and :func:`finalize_dict` functions and the
32 :class:`ZstdDict` class to train and manage Zstandard dictionaries.
33 * The :class:`CompressionParameter`, :class:`DecompressionParameter`, and
34 :class:`Strategy` classes for setting advanced (de)compression parameters.
35
36 .. include:: ../includes/optional-module.rst
37
38
39 Exceptions
40 ----------
41
42 .. exception:: ZstdError
43
44 This exception is raised when an error occurs during compression or
45 decompression, or while initializing the (de)compressor state.
46
47
48 Reading and writing compressed files
49 ------------------------------------
50
51 .. function:: open(file, /, mode='rb', *, level=None, options=None, \
52 zstd_dict=None, encoding=None, errors=None, newline=None)
53
54 Open a Zstandard-compressed file in binary or text mode, returning a
55 :term:`file object`.
56
57 The *file* argument can be either a file name (given as a
58 :class:`str`, :class:`bytes` or :term:`path-like <path-like object>`
59 object), in which case the named file is opened, or it can be an existing
60 file object to read from or write to.
61
62 The mode argument can be either ``'rb'`` for reading (default), ``'wb'`` for
63 overwriting, ``'ab'`` for appending, or ``'xb'`` for exclusive creation.
64 These can equivalently be given as ``'r'``, ``'w'``, ``'a'``, and ``'x'``
65 respectively. You may also open in text mode with ``'rt'``, ``'wt'``,
66 ``'at'``, and ``'xt'`` respectively.
67
68 When reading, the *options* argument can be a dictionary providing advanced
69 decompression parameters; see :class:`DecompressionParameter` for detailed
70 information about supported
71 parameters. The *zstd_dict* argument is a :class:`ZstdDict` instance to be
72 used during decompression. When reading, if the *level*
73 argument is not None, a :exc:`!TypeError` will be raised.
74
75 When writing, the *options* argument can be a dictionary
76 providing advanced compression parameters; see
77 :class:`CompressionParameter` for detailed information about supported
78 parameters. The *level* argument is the compression level to use when
79 writing compressed data. Only one of *level* or *options* may be non-None.
80 The *zstd_dict* argument is a :class:`ZstdDict` instance to be used during
81 compression.
82
83 In binary mode, this function is equivalent to the :class:`ZstdFile`
84 constructor: ``ZstdFile(file, mode, ...)``. In this case, the
85 *encoding*, *errors*, and *newline* parameters must not be provided.
86
87 In text mode, a :class:`ZstdFile` object is created, and wrapped in an
88 :class:`io.TextIOWrapper` instance with the specified encoding, error
89 handling behavior, and line endings.
90
91
92 .. class:: ZstdFile(file, /, mode='rb', *, level=None, options=None, \
93 zstd_dict=None)
94
95 Open a Zstandard-compressed file in binary mode.
96
97 A :class:`ZstdFile` can wrap an already-open :term:`file object`, or operate
98 directly on a named file. The *file* argument specifies either the file
99 object to wrap, or the name of the file to open (as a :class:`str`,
100 :class:`bytes` or :term:`path-like <path-like object>` object). If
101 wrapping an existing file object, the wrapped file will not be closed when
102 the :class:`ZstdFile` is closed.
103
104 The *mode* argument can be either ``'rb'`` for reading (default), ``'wb'``
105 for overwriting, ``'xb'`` for exclusive creation, or ``'ab'`` for appending.
106 These can equivalently be given as ``'r'``, ``'w'``, ``'x'`` and ``'a'``
107 respectively.
108
109 If *file* is a file object (rather than an actual file name), a mode of
110 ``'w'`` does not truncate the file, and is instead equivalent to ``'a'``.
111
112 When reading, the *options* argument can be a dictionary
113 providing advanced decompression parameters; see
114 :class:`DecompressionParameter` for detailed information about supported
115 parameters. The *zstd_dict* argument is a :class:`ZstdDict` instance to be
116 used during decompression. When reading, if the *level*
117 argument is not None, a :exc:`!TypeError` will be raised.
118
119 When writing, the *options* argument can be a dictionary
120 providing advanced compression parameters; see
121 :class:`CompressionParameter` for detailed information about supported
122 parameters. The *level* argument is the compression level to use when
123 writing compressed data. Only one of *level* or *options* may be passed. The
124 *zstd_dict* argument is a :class:`ZstdDict` instance to be used during
125 compression.
126
127 :class:`!ZstdFile` supports all the members specified by
128 :class:`io.BufferedIOBase`, except for :meth:`~io.BufferedIOBase.detach`
129 and :meth:`~io.IOBase.truncate`.
130 Iteration and the :keyword:`with` statement are supported.
131
132 The following method and attributes are also provided:
133
134 .. method:: peek(size=-1)
135
136 Return buffered data without advancing the file position. At least one
137 byte of data will be returned, unless EOF has been reached. The exact
138 number of bytes returned is unspecified (the *size* argument is ignored).
139
140 .. note:: While calling :meth:`peek` does not change the file position of
141 the :class:`ZstdFile`, it may change the position of the underlying
142 file object (for example, if the :class:`ZstdFile` was constructed by
143 passing a file object for *file*).
144
145 .. attribute:: mode
146
147 ``'rb'`` for reading and ``'wb'`` for writing.
148
149 .. attribute:: name
150
151 The name of the Zstandard file. Equivalent to the :attr:`~io.FileIO.name`
152 attribute of the underlying :term:`file object`.
153
154
155 Compressing and decompressing data in memory
156 --------------------------------------------
157
158 .. function:: compress(data, level=None, options=None, zstd_dict=None)
159
160 Compress *data* (a :term:`bytes-like object`), returning the compressed
161 data as a :class:`bytes` object.
162
163 The *level* argument is an integer controlling the level of
164 compression. *level* is an alternative to setting
165 :attr:`CompressionParameter.compression_level` in *options*. Use
166 :meth:`~CompressionParameter.bounds` on
167 :attr:`~CompressionParameter.compression_level` to get the values that can
168 be passed for *level*. If advanced compression options are needed, the
169 *level* argument must be omitted and in the *options* dictionary the
170 :attr:`!CompressionParameter.compression_level` parameter should be set.
171
172 The *options* argument is a Python dictionary containing advanced
173 compression parameters. The valid keys and values for compression parameters
174 are documented as part of the :class:`CompressionParameter` documentation.
175
176 The *zstd_dict* argument is an instance of :class:`ZstdDict`
177 containing trained data to improve compression efficiency. The
178 function :func:`train_dict` can be used to generate a Zstandard dictionary.
179
180
181 .. function:: decompress(data, zstd_dict=None, options=None)
182
183 Decompress *data* (a :term:`bytes-like object`), returning the uncompressed
184 data as a :class:`bytes` object.
185
186 The *options* argument is a Python dictionary containing advanced
187 decompression parameters. The valid keys and values for compression
188 parameters are documented as part of the :class:`DecompressionParameter`
189 documentation.
190
191 The *zstd_dict* argument is an instance of :class:`ZstdDict`
192 containing trained data used during compression. This must be
193 the same Zstandard dictionary used during compression.
194
195 If *data* is the concatenation of multiple distinct compressed frames,
196 decompress all of these frames, and return the concatenation of the results.
197
198
199 .. class:: ZstdCompressor(level=None, options=None, zstd_dict=None)
200
201 Create a compressor object, which can be used to compress data
202 incrementally.
203
204 For a more convenient way of compressing a single chunk of data, see the
205 module-level function :func:`compress`.
206
207 The *level* argument is an integer controlling the level of
208 compression. *level* is an alternative to setting
209 :attr:`CompressionParameter.compression_level` in *options*. Use
210 :meth:`~CompressionParameter.bounds` on
211 :attr:`~CompressionParameter.compression_level` to get the values that can
212 be passed for *level*. If advanced compression options are needed, the
213 *level* argument must be omitted and in the *options* dictionary the
214 :attr:`!CompressionParameter.compression_level` parameter should be set.
215
216 The *options* argument is a Python dictionary containing advanced
217 compression parameters. The valid keys and values for compression parameters
218 are documented as part of the :class:`CompressionParameter` documentation.
219
220 The *zstd_dict* argument is an optional instance of :class:`ZstdDict`
221 containing trained data to improve compression efficiency. The
222 function :func:`train_dict` can be used to generate a Zstandard dictionary.
223
224
225 .. method:: compress(data, mode=ZstdCompressor.CONTINUE)
226
227 Compress *data* (a :term:`bytes-like object`), returning a :class:`bytes`
228 object with compressed data if possible, or otherwise an empty
229 :class:`!bytes` object. Some of *data* may be buffered internally, for
230 use in later calls to :meth:`!compress` and :meth:`~.flush`. The returned
231 data should be concatenated with the output of any previous calls to
232 :meth:`~.compress`.
233
234 The *mode* argument is a :class:`ZstdCompressor` attribute, either
235 :attr:`~.CONTINUE`, :attr:`~.FLUSH_BLOCK`,
236 or :attr:`~.FLUSH_FRAME`.
237
238 When all data has been provided to the compressor, call the
239 :meth:`~.flush` method to finish the compression process. If
240 :meth:`~.compress` is called with *mode* set to :attr:`~.FLUSH_FRAME`,
241 :meth:`~.flush` should not be called, as it would write out a new empty
242 frame.
243
244 .. method:: flush(mode=ZstdCompressor.FLUSH_FRAME)
245
246 Finish the compression process, returning a :class:`bytes` object
247 containing any data stored in the compressor's internal buffers.
248
249 The *mode* argument is a :class:`ZstdCompressor` attribute, either
250 :attr:`~.FLUSH_BLOCK`, or :attr:`~.FLUSH_FRAME`.
251
252 .. method:: set_pledged_input_size(size)
253
254 Specify the amount of uncompressed data *size* that will be provided for
255 the next frame. *size* will be written into the frame header of the next
256 frame unless :attr:`CompressionParameter.content_size_flag` is ``False``
257 or ``0``. A size of ``0`` means that the frame is empty. If *size* is
258 ``None``, the frame header will omit the frame size. Frames that include
259 the uncompressed data size require less memory to decompress, especially
260 at higher compression levels.
261
262 If :attr:`last_mode` is not :attr:`FLUSH_FRAME`, a
263 :exc:`ValueError` is raised as the compressor is not at the start of
264 a frame. If the pledged size does not match the actual size of data
265 provided to :meth:`.compress`, future calls to :meth:`!compress` or
266 :meth:`flush` may raise :exc:`ZstdError` and the last chunk of data may
267 be lost.
268
269 After :meth:`flush` or :meth:`.compress` are called with mode
270 :attr:`FLUSH_FRAME`, the next frame will not include the frame size into
271 the header unless :meth:`!set_pledged_input_size` is called again.
272
273 .. attribute:: CONTINUE
274
275 Collect more data for compression, which may or may not generate output
276 immediately. This mode optimizes the compression ratio by maximizing the
277 amount of data per block and frame.
278
279 .. attribute:: FLUSH_BLOCK
280
281 Complete and write a block to the data stream. The data returned so far
282 can be immediately decompressed. Past data can still be referenced in
283 future blocks generated by calls to :meth:`~.compress`,
284 improving compression.
285
286 .. attribute:: FLUSH_FRAME
287
288 Complete and write out a frame. Future data provided to
289 :meth:`~.compress` will be written into a new frame and
290 *cannot* reference past data.
291
292 .. attribute:: last_mode
293
294 The last mode passed to either :meth:`~.compress` or :meth:`~.flush`.
295 The value can be one of :attr:`~.CONTINUE`, :attr:`~.FLUSH_BLOCK`, or
296 :attr:`~.FLUSH_FRAME`. The initial value is :attr:`~.FLUSH_FRAME`,
297 signifying that the compressor is at the start of a new frame.
298
299
300 .. class:: ZstdDecompressor(zstd_dict=None, options=None)
301
302 Create a decompressor object, which can be used to decompress data
303 incrementally.
304
305 For a more convenient way of decompressing an entire compressed stream at
306 once, see the module-level function :func:`decompress`.
307
308 The *options* argument is a Python dictionary containing advanced
309 decompression parameters. The valid keys and values for compression
310 parameters are documented as part of the :class:`DecompressionParameter`
311 documentation.
312
313 The *zstd_dict* argument is an instance of :class:`ZstdDict`
314 containing trained data used during compression. This must be
315 the same Zstandard dictionary used during compression.
316
317 .. note::
318 This class does not transparently handle inputs containing multiple
319 compressed frames, unlike the :func:`decompress` function and
320 :class:`ZstdFile` class. To decompress a multi-frame input, you should
321 use :func:`decompress`, :class:`ZstdFile` if working with a
322 :term:`file object`, or multiple :class:`!ZstdDecompressor` instances.
323
324 .. method:: decompress(data, max_length=-1)
325
326 Decompress *data* (a :term:`bytes-like object`), returning
327 uncompressed data as bytes. Some of *data* may be buffered
328 internally, for use in later calls to :meth:`!decompress`.
329 The returned data should be concatenated with the output of any previous
330 calls to :meth:`!decompress`.
331
332 If *max_length* is non-negative, the method returns at most *max_length*
333 bytes of decompressed data. If this limit is reached and further
334 output can be produced, the :attr:`~.needs_input` attribute will
335 be set to ``False``. In this case, the next call to
336 :meth:`~.decompress` may provide *data* as ``b''`` to obtain
337 more of the output.
338
339 If all of the input data was decompressed and returned (either
340 because this was less than *max_length* bytes, or because
341 *max_length* was negative), the :attr:`~.needs_input` attribute
342 will be set to ``True``.
343
344 Attempting to decompress data after the end of a frame will raise a
345 :exc:`ZstdError`. Any data found after the end of the frame is ignored
346 and saved in the :attr:`~.unused_data` attribute.
347
348 .. attribute:: eof
349
350 ``True`` if the end-of-stream marker has been reached.
351
352 .. attribute:: unused_data
353
354 Data found after the end of the compressed stream.
355
356 Before the end of the stream is reached, this will be ``b''``.
357
358 .. attribute:: needs_input
359
360 ``False`` if the :meth:`.decompress` method can provide more
361 decompressed data before requiring new compressed input.
362
363
364 Zstandard dictionaries
365 ----------------------
366
367
368 .. function:: train_dict(samples, dict_size)
369
370 Train a Zstandard dictionary, returning a :class:`ZstdDict` instance.
371 Zstandard dictionaries enable more efficient compression of smaller sizes
372 of data, which is traditionally difficult to compress due to less
373 repetition. If you are compressing multiple similar groups of data (such as
374 similar files), Zstandard dictionaries can improve compression ratios and
375 speed significantly.
376
377 The *samples* argument (an iterable of :class:`bytes` objects), is the
378 population of samples used to train the Zstandard dictionary.
379
380 The *dict_size* argument, an integer, is the maximum size (in bytes) the
381 Zstandard dictionary should be. The Zstandard documentation suggests an
382 absolute maximum of no more than 100 KB, but the maximum can often be smaller
383 depending on the data. Larger dictionaries generally slow down compression,
384 but improve compression ratios. Smaller dictionaries lead to faster
385 compression, but reduce the compression ratio.
386
387
388 .. function:: finalize_dict(zstd_dict, /, samples, dict_size, level)
389
390 An advanced function for converting a "raw content" Zstandard dictionary into
391 a regular Zstandard dictionary. "Raw content" dictionaries are a sequence of
392 bytes that do not need to follow the structure of a normal Zstandard
393 dictionary.
394
395 The *zstd_dict* argument is a :class:`ZstdDict` instance with
396 the :attr:`~ZstdDict.dict_content` containing the raw dictionary contents.
397
398 The *samples* argument (an iterable of :class:`bytes` objects), contains
399 sample data for generating the Zstandard dictionary.
400
401 The *dict_size* argument, an integer, is the maximum size (in bytes) the
402 Zstandard dictionary should be. See :func:`train_dict` for
403 suggestions on the maximum dictionary size.
404
405 The *level* argument (an integer) is the compression level expected to be
406 passed to the compressors using this dictionary. The dictionary information
407 varies for each compression level, so tuning for the proper compression
408 level can make compression more efficient.
409
410
411 .. class:: ZstdDict(dict_content, /, *, is_raw=False)
412
413 A wrapper around Zstandard dictionaries. Dictionaries can be used to improve
414 the compression of many small chunks of data. Use :func:`train_dict` if you
415 need to train a new dictionary from sample data.
416
417 The *dict_content* argument (a :term:`bytes-like object`), is the already
418 trained dictionary information.
419
420 The *is_raw* argument, a boolean, is an advanced parameter controlling the
421 meaning of *dict_content*. ``True`` means *dict_content* is a "raw content"
422 dictionary, without any format restrictions. ``False`` means *dict_content*
423 is an ordinary Zstandard dictionary, created from Zstandard functions,
424 for example, :func:`train_dict` or the external :program:`zstd` CLI.
425
426 When passing a :class:`!ZstdDict` to a function, the
427 :attr:`!as_digested_dict` and :attr:`!as_undigested_dict` attributes can
428 control how the dictionary is loaded by passing them as the ``zstd_dict``
429 argument, for example, ``compress(data, zstd_dict=zd.as_digested_dict)``.
430 Digesting a dictionary is a costly operation that occurs when loading a
431 Zstandard dictionary. When making multiple calls to compression or
432 decompression, passing a digested dictionary will reduce the overhead of
433 loading the dictionary.
434
435 .. list-table:: Difference for compression
436 :widths: 10 14 10
437 :header-rows: 1
438
439 * -
440 - Digested dictionary
441 - Undigested dictionary
442 * - Advanced parameters of the compressor which may be overridden by
443 the dictionary's parameters
444 - ``window_log``, ``hash_log``, ``chain_log``, ``search_log``,
445 ``min_match``, ``target_length``, ``strategy``,
446 ``enable_long_distance_matching``, ``ldm_hash_log``,
447 ``ldm_min_match``, ``ldm_bucket_size_log``, ``ldm_hash_rate_log``,
448 and some non-public parameters.
449 - None
450 * - :class:`!ZstdDict` internally caches the dictionary
451 - Yes. It's faster when loading a digested dictionary again with the
452 same compression level.
453 - No. If you wish to load an undigested dictionary multiple times,
454 consider reusing a compressor object.
455
456 If passing a :class:`!ZstdDict` without any attribute, an undigested
457 dictionary is passed by default when compressing and a digested dictionary
458 is generated if necessary and passed by default when decompressing.
459
460 .. attribute:: dict_content
461
462 The content of the Zstandard dictionary, a ``bytes`` object. It's the
463 same as the *dict_content* argument in the ``__init__`` method. It can
464 be used with other programs, such as the ``zstd`` CLI program.
465
466 .. attribute:: dict_id
467
468 Identifier of the Zstandard dictionary, a non-negative int value.
469
470 Non-zero means the dictionary is ordinary, created by Zstandard
471 functions and following the Zstandard format.
472
473 ``0`` means a "raw content" dictionary, free of any format restriction,
474 used for advanced users.
475
476 .. note::
477
478 The meaning of ``0`` for :attr:`!ZstdDict.dict_id` is different
479 from the ``dictionary_id`` attribute to the :func:`get_frame_info`
480 function.
481
482 .. attribute:: as_digested_dict
483
484 Load as a digested dictionary.
485
486 .. attribute:: as_undigested_dict
487
488 Load as an undigested dictionary.
489
490
491 Advanced parameter control
492 --------------------------
493
494 .. class:: CompressionParameter()
495
496 An :class:`~enum.IntEnum` containing the advanced compression parameter
497 keys that can be used when compressing data.
498
499 The :meth:`~.bounds` method can be used on any attribute to get the valid
500 values for that parameter.
501
502 Parameters are optional; any omitted parameter will have it's value selected
503 automatically.
504
505 Example getting the lower and upper bound of :attr:`~.compression_level`::
506
507 lower, upper = CompressionParameter.compression_level.bounds()
508
509 Example setting the :attr:`~.window_log` to the maximum size::
510
511 _lower, upper = CompressionParameter.window_log.bounds()
512 options = {CompressionParameter.window_log: upper}
513 compress(b'venezuelan beaver cheese', options=options)
514
515 .. method:: bounds()
516
517 Return the tuple of int bounds, ``(lower, upper)``, of a compression
518 parameter. This method should be called on the attribute you wish to
519 retrieve the bounds of. For example, to get the valid values for
520 :attr:`~.compression_level`, one may check the result of
521 ``CompressionParameter.compression_level.bounds()``.
522
523 Both the lower and upper bounds are inclusive.
524
525 .. attribute:: compression_level
526
527 A high-level means of setting other compression parameters that affect
528 the speed and ratio of compressing data.
529
530 Regular compression levels are greater than ``0``. Values greater than
531 ``20`` are considered "ultra" compression and require more memory than
532 other levels. Negative values can be used to trade off faster compression
533 for worse compression ratios.
534
535 Setting the level to zero uses :attr:`COMPRESSION_LEVEL_DEFAULT`.
536
537 .. attribute:: window_log
538
539 Maximum allowed back-reference distance the compressor can use when
540 compressing data, expressed as power of two, ``1 << window_log`` bytes.
541 This parameter greatly influences the memory usage of compression. Higher
542 values require more memory but gain better compression values.
543
544 A value of zero causes the value to be selected automatically.
545
546 .. attribute:: hash_log
547
548 Size of the initial probe table, as a power of two. The resulting memory
549 usage is ``1 << (hash_log+2)`` bytes. Larger tables improve compression
550 ratio of strategies <= :attr:`~Strategy.dfast`, and improve compression
551 speed of strategies > :attr:`~Strategy.dfast`.
552
553 A value of zero causes the value to be selected automatically.
554
555 .. attribute:: chain_log
556
557 Size of the multi-probe search table, as a power of two. The resulting
558 memory usage is ``1 << (chain_log+2)`` bytes. Larger tables result in
559 better and slower compression. This parameter has no effect for the
560 :attr:`~Strategy.fast` strategy. It's still useful when using
561 :attr:`~Strategy.dfast` strategy, in which case it defines a secondary
562 probe table.
563
564 A value of zero causes the value to be selected automatically.
565
566 .. attribute:: search_log
567
568 Number of search attempts, as a power of two. More attempts result in
569 better and slower compression. This parameter is useless for
570 :attr:`~Strategy.fast` and :attr:`~Strategy.dfast` strategies.
571
572 A value of zero causes the value to be selected automatically.
573
574 .. attribute:: min_match
575
576 Minimum size of searched matches. Larger values increase compression and
577 decompression speed, but decrease ratio. Note that Zstandard can still
578 find matches of smaller size, it just tweaks its search algorithm to look
579 for this size and larger. For all strategies < :attr:`~Strategy.btopt`,
580 the effective minimum is ``4``; for all strategies
581 > :attr:`~Strategy.fast`, the effective maximum is ``6``.
582
583 A value of zero causes the value to be selected automatically.
584
585 .. attribute:: target_length
586
587 The impact of this field depends on the selected :class:`Strategy`.
588
589 For strategies :attr:`~Strategy.btopt`, :attr:`~Strategy.btultra` and
590 :attr:`~Strategy.btultra2`, the value is the length of a match
591 considered "good enough" to stop searching. Larger values make
592 compression ratios better, but compresses slower.
593
594 For strategy :attr:`~Strategy.fast`, it is the distance between match
595 sampling. Larger values make compression faster, but with a worse
596 compression ratio.
597
598 A value of zero causes the value to be selected automatically.
599
600 .. attribute:: strategy
601
602 The higher the value of selected strategy, the more complex the
603 compression technique used by zstd, resulting in higher compression
604 ratios but slower compression.
605
606 .. seealso:: :class:`Strategy`
607
608 .. attribute:: enable_long_distance_matching
609
610 Long distance matching can be used to improve compression for large
611 inputs by finding large matches at greater distances. It increases memory
612 usage and window size.
613
614 ``True`` or ``1`` enable long distance matching while ``False`` or ``0``
615 disable it.
616
617 Enabling this parameter increases default
618 :attr:`~CompressionParameter.window_log` to 128 MiB except when expressly
619 set to a different value. This setting is enabled by default if
620 :attr:`!window_log` >= 128 MiB and the compression
621 strategy >= :attr:`~Strategy.btopt` (compression level 16+).
622
623 .. attribute:: ldm_hash_log
624
625 Size of the table for long distance matching, as a power of two. Larger
626 values increase memory usage and compression ratio, but decrease
627 compression speed.
628
629 A value of zero causes the value to be selected automatically.
630
631 .. attribute:: ldm_min_match
632
633 Minimum match size for long distance matcher. Larger or too small values
634 can often decrease the compression ratio.
635
636 A value of zero causes the value to be selected automatically.
637
638 .. attribute:: ldm_bucket_size_log
639
640 Log size of each bucket in the long distance matcher hash table for
641 collision resolution. Larger values improve collision resolution but
642 decrease compression speed.
643
644 A value of zero causes the value to be selected automatically.
645
646 .. attribute:: ldm_hash_rate_log
647
648 Frequency of inserting/looking up entries into the long distance matcher
649 hash table. Larger values improve compression speed. Deviating far from
650 the default value will likely result in a compression ratio decrease.
651
652 A value of zero causes the value to be selected automatically.
653
654 .. attribute:: content_size_flag
655
656 Write the size of the data to be compressed into the Zstandard frame
657 header when known prior to compressing.
658
659 This flag only takes effect under the following scenarios:
660
661 * Calling :func:`compress` for one-shot compression
662 * Providing all of the data to be compressed in the frame in a single
663 :meth:`ZstdCompressor.compress` call, with the
664 :attr:`ZstdCompressor.FLUSH_FRAME` mode.
665 * Calling :meth:`ZstdCompressor.set_pledged_input_size` with the exact
666 amount of data that will be provided to the compressor prior to any
667 calls to :meth:`ZstdCompressor.compress` for the current frame.
668 :meth:`!ZstdCompressor.set_pledged_input_size` must be called for each
669 new frame.
670
671 All other compression calls may not write the size information into the
672 frame header.
673
674 ``True`` or ``1`` enable the content size flag while ``False`` or ``0``
675 disable it.
676
677 .. attribute:: checksum_flag
678
679 A four-byte checksum using XXHash64 of the uncompressed content is
680 written at the end of each frame. Zstandard's decompression code verifies
681 the checksum. If there is a mismatch a :class:`ZstdError` exception is
682 raised.
683
684 ``True`` or ``1`` enable checksum generation while ``False`` or ``0``
685 disable it.
686
687 .. attribute:: dict_id_flag
688
689 When compressing with a :class:`ZstdDict`, the dictionary's ID is written
690 into the frame header.
691
692 ``True`` or ``1`` enable storing the dictionary ID while ``False`` or
693 ``0`` disable it.
694
695 .. attribute:: nb_workers
696
697 Select how many threads will be spawned to compress in parallel. When
698 :attr:`!nb_workers` > 0, enables multi-threaded compression, a value of
699 ``1`` means "one-thread multi-threaded mode". More workers improve speed,
700 but also increase memory usage and slightly reduce compression ratio.
701
702 A value of zero disables multi-threading.
703
704 .. attribute:: job_size
705
706 Size of a compression job, in bytes. This value is enforced only when
707 :attr:`~CompressionParameter.nb_workers` >= 1. Each compression job is
708 completed in parallel, so this value can indirectly impact the number of
709 active threads.
710
711 A value of zero causes the value to be selected automatically.
712
713 .. attribute:: overlap_log
714
715 Sets how much data is reloaded from previous jobs (threads) for new jobs
716 to be used by the look behind window during compression. This value is
717 only used when :attr:`~CompressionParameter.nb_workers` >= 1. Acceptable
718 values vary from 0 to 9.
719
720 * 0 means dynamically set the overlap amount
721 * 1 means no overlap
722 * 9 means use a full window size from the previous job
723
724 Each increment halves/doubles the overlap size. "8" means an overlap of
725 ``window_size/2``, "7" means an overlap of ``window_size/4``, etc.
726
727 .. class:: DecompressionParameter()
728
729 An :class:`~enum.IntEnum` containing the advanced decompression parameter
730 keys that can be used when decompressing data. Parameters are optional; any
731 omitted parameter will have it's value selected automatically.
732
733 The :meth:`~.bounds` method can be used on any attribute to get the valid
734 values for that parameter.
735
736 Example setting the :attr:`~.window_log_max` to the maximum size::
737
738 data = compress(b'Some very long buffer of bytes...')
739
740 _lower, upper = DecompressionParameter.window_log_max.bounds()
741
742 options = {DecompressionParameter.window_log_max: upper}
743 decompress(data, options=options)
744
745 .. method:: bounds()
746
747 Return the tuple of int bounds, ``(lower, upper)``, of a decompression
748 parameter. This method should be called on the attribute you wish to
749 retrieve the bounds of.
750
751 Both the lower and upper bounds are inclusive.
752
753 .. attribute:: window_log_max
754
755 The base-two logarithm of the maximum size of the window used during
756 decompression. This can be useful to limit the amount of memory used when
757 decompressing data. A larger maximum window size leads to faster
758 decompression.
759
760 A value of zero causes the value to be selected automatically.
761
762
763 .. class:: Strategy()
764
765 An :class:`~enum.IntEnum` containing strategies for compression.
766 Higher-numbered strategies correspond to more complex and slower
767 compression.
768
769 .. note::
770
771 The values of attributes of :class:`!Strategy` are not necessarily stable
772 across zstd versions. Only the ordering of the attributes may be relied
773 upon. The attributes are listed below in order.
774
775 The following strategies are available:
776
777 .. attribute:: fast
778
779 .. attribute:: dfast
780
781 .. attribute:: greedy
782
783 .. attribute:: lazy
784
785 .. attribute:: lazy2
786
787 .. attribute:: btlazy2
788
789 .. attribute:: btopt
790
791 .. attribute:: btultra
792
793 .. attribute:: btultra2
794
795
796 Miscellaneous
797 -------------
798
799 .. function:: get_frame_info(frame_buffer)
800
801 Retrieve a :class:`FrameInfo` object containing metadata about a Zstandard
802 frame. Frames contain metadata related to the compressed data they hold.
803
804
805 .. class:: FrameInfo
806
807 Metadata related to a Zstandard frame.
808
809 .. attribute:: decompressed_size
810
811 The size of the decompressed contents of the frame.
812
813 .. attribute:: dictionary_id
814
815 An integer representing the Zstandard dictionary ID needed for
816 decompressing the frame. ``0`` means the dictionary ID was not
817 recorded in the frame header. This may mean that a Zstandard dictionary
818 is not needed, or that the ID of a required dictionary was not recorded.
819
820
821 .. attribute:: COMPRESSION_LEVEL_DEFAULT
822
823 The default compression level for Zstandard: ``3``.
824
825
826 .. attribute:: zstd_version_info
827
828 Version number of the runtime zstd library as a tuple of integers
829 (major, minor, release).
830
831
832 Examples
833 --------
834
835 Reading in a compressed file:
836
837 .. code-block:: python
838
839 from compression import zstd
840
841 with zstd.open("file.zst") as f:
842 file_content = f.read()
843
844 Creating a compressed file:
845
846 .. code-block:: python
847
848 from compression import zstd
849
850 data = b"Insert Data Here"
851 with zstd.open("file.zst", "w") as f:
852 f.write(data)
853
854 Compressing data in memory:
855
856 .. code-block:: python
857
858 from compression import zstd
859
860 data_in = b"Insert Data Here"
861 data_out = zstd.compress(data_in)
862
863 Incremental compression:
864
865 .. code-block:: python
866
867 from compression import zstd
868
869 comp = zstd.ZstdCompressor()
870 out1 = comp.compress(b"Some data\n")
871 out2 = comp.compress(b"Another piece of data\n")
872 out3 = comp.compress(b"Even more data\n")
873 out4 = comp.flush()
874 # Concatenate all the partial results:
875 result = b"".join([out1, out2, out3, out4])
876
877 Writing compressed data to an already-open file:
878
879 .. code-block:: python
880
881 from compression import zstd
882
883 with open("myfile", "wb") as f:
884 f.write(b"This data will not be compressed\n")
885 with zstd.open(f, "w") as zstf:
886 zstf.write(b"This *will* be compressed\n")
887 f.write(b"Not compressed\n")
888
889 Creating a compressed file using compression parameters:
890
891 .. code-block:: python
892
893 from compression import zstd
894
895 options = {
896 zstd.CompressionParameter.checksum_flag: 1
897 }
898 with zstd.open("file.zst", "w", options=options) as f:
899 f.write(b"Mind if I squeeze in?")