Debian Codegraph @

Doc/library/lzma.rst

raw
1 :mod:`!lzma` --- Compression using the LZMA algorithm
2 =====================================================
3
4 .. module:: lzma
5 :synopsis: A Python wrapper for the liblzma compression library.
6
7 .. moduleauthor:: Nadeem Vawda <[email protected]>
8 .. sectionauthor:: Nadeem Vawda <[email protected]>
9
10 .. versionadded:: 3.3
11
12 **Source code:** :source:`Lib/lzma.py`
13
14 --------------
15
16 This module provides classes and convenience functions for compressing and
17 decompressing data using the LZMA compression algorithm. Also included is a file
18 interface supporting the ``.xz`` and legacy ``.lzma`` file formats used by the
19 :program:`xz` utility, as well as raw compressed streams.
20
21 The interface provided by this module is very similar to that of the :mod:`bz2`
22 module. Note that :class:`LZMAFile` and :class:`bz2.BZ2File` are *not*
23 thread-safe, so if you need to use a single :class:`LZMAFile` instance
24 from multiple threads, it is necessary to protect it with a lock.
25
26
27 .. exception:: LZMAError
28
29 This exception is raised when an error occurs during compression or
30 decompression, or while initializing the compressor/decompressor state.
31
32
33 Reading and writing compressed files
34 ------------------------------------
35
36 .. function:: open(filename, mode="rb", *, format=None, check=-1, preset=None, filters=None, encoding=None, errors=None, newline=None)
37
38 Open an LZMA-compressed file in binary or text mode, returning a :term:`file
39 object`.
40
41 The *filename* argument can be either an actual file name (given as a
42 :class:`str`, :class:`bytes` or :term:`path-like <path-like object>` object), in
43 which case the named file is opened, or it can be an existing file object
44 to read from or write to.
45
46 The *mode* argument can be any of ``"r"``, ``"rb"``, ``"w"``, ``"wb"``,
47 ``"x"``, ``"xb"``, ``"a"`` or ``"ab"`` for binary mode, or ``"rt"``,
48 ``"wt"``, ``"xt"``, or ``"at"`` for text mode. The default is ``"rb"``.
49
50 When opening a file for reading, the *format* and *filters* arguments have
51 the same meanings as for :class:`LZMADecompressor`. In this case, the *check*
52 and *preset* arguments should not be used.
53
54 When opening a file for writing, the *format*, *check*, *preset* and
55 *filters* arguments have the same meanings as for :class:`LZMACompressor`.
56
57 For binary mode, this function is equivalent to the :class:`LZMAFile`
58 constructor: ``LZMAFile(filename, mode, ...)``. In this case, the *encoding*,
59 *errors* and *newline* arguments must not be provided.
60
61 For text mode, a :class:`LZMAFile` object is created, and wrapped in an
62 :class:`io.TextIOWrapper` instance with the specified encoding, error
63 handling behavior, and line ending(s).
64
65 .. versionchanged:: 3.4
66 Added support for the ``"x"``, ``"xb"`` and ``"xt"`` modes.
67
68 .. versionchanged:: 3.6
69 Accepts a :term:`path-like object`.
70
71
72 .. class:: LZMAFile(filename=None, mode="r", *, format=None, check=-1, preset=None, filters=None)
73
74 Open an LZMA-compressed file in binary mode.
75
76 An :class:`LZMAFile` can wrap an already-open :term:`file object`, or operate
77 directly on a named file. The *filename* argument specifies either the file
78 object to wrap, or the name of the file to open (as a :class:`str`,
79 :class:`bytes` or :term:`path-like <path-like object>` object). When wrapping an
80 existing file object, the wrapped file will not be closed when the
81 :class:`LZMAFile` is closed.
82
83 The *mode* argument can be either ``"r"`` for reading (default), ``"w"`` for
84 overwriting, ``"x"`` for exclusive creation, or ``"a"`` for appending. These
85 can equivalently be given as ``"rb"``, ``"wb"``, ``"xb"`` and ``"ab"``
86 respectively.
87
88 If *filename* is a file object (rather than an actual file name), a mode of
89 ``"w"`` does not truncate the file, and is instead equivalent to ``"a"``.
90
91 When opening a file for reading, the input file may be the concatenation of
92 multiple separate compressed streams. These are transparently decoded as a
93 single logical stream.
94
95 When opening a file for reading, the *format* and *filters* arguments have
96 the same meanings as for :class:`LZMADecompressor`. In this case, the *check*
97 and *preset* arguments should not be used.
98
99 When opening a file for writing, the *format*, *check*, *preset* and
100 *filters* arguments have the same meanings as for :class:`LZMACompressor`.
101
102 :class:`LZMAFile` supports all the members specified by
103 :class:`io.BufferedIOBase`, except for :meth:`~io.BufferedIOBase.detach`
104 and :meth:`~io.IOBase.truncate`.
105 Iteration and the :keyword:`with` statement are supported.
106
107 The following method and attributes are also provided:
108
109 .. method:: peek(size=-1)
110
111 Return buffered data without advancing the file position. At least one
112 byte of data will be returned, unless EOF has been reached. The exact
113 number of bytes returned is unspecified (the *size* argument is ignored).
114
115 .. note:: While calling :meth:`peek` does not change the file position of
116 the :class:`LZMAFile`, it may change the position of the underlying
117 file object (e.g. if the :class:`LZMAFile` was constructed by passing a
118 file object for *filename*).
119
120 .. attribute:: mode
121
122 ``'rb'`` for reading and ``'wb'`` for writing.
123
124 .. versionadded:: 3.13
125
126 .. attribute:: name
127
128 The lzma file name. Equivalent to the :attr:`~io.FileIO.name`
129 attribute of the underlying :term:`file object`.
130
131 .. versionadded:: 3.13
132
133
134 .. versionchanged:: 3.4
135 Added support for the ``"x"`` and ``"xb"`` modes.
136
137 .. versionchanged:: 3.5
138 The :meth:`~io.BufferedIOBase.read` method now accepts an argument of
139 ``None``.
140
141 .. versionchanged:: 3.6
142 Accepts a :term:`path-like object`.
143
144
145 Compressing and decompressing data in memory
146 --------------------------------------------
147
148 .. class:: LZMACompressor(format=FORMAT_XZ, check=-1, preset=None, filters=None)
149
150 Create a compressor object, which can be used to compress data incrementally.
151
152 For a more convenient way of compressing a single chunk of data, see
153 :func:`compress`.
154
155 The *format* argument specifies what container format should be used.
156 Possible values are :const:`FORMAT_XZ` (the default),
157 :const:`FORMAT_ALONE` and :const:`FORMAT_RAW`.
158
159 The *check* argument specifies the type of integrity check to include in the
160 compressed data. This check is used when decompressing, to ensure that the
161 data has not been corrupted. Possible values are :const:`CHECK_NONE`,
162 :const:`CHECK_CRC32`, :const:`CHECK_CRC64` (the default for
163 :const:`FORMAT_XZ`) and :const:`CHECK_SHA256`.
164
165 If the specified check is not supported, an :class:`LZMAError` is raised.
166
167 The compression settings can be specified either as a preset compression
168 level (with the *preset* argument), or in detail as a custom filter chain
169 (with the *filters* argument).
170
171 The *preset* argument (if provided) should be an integer between ``0`` and
172 ``9`` (inclusive), optionally OR-ed with the constant
173 :const:`PRESET_EXTREME`. If neither *preset* nor *filters* are given, the
174 default behavior is to use :const:`PRESET_DEFAULT` (preset level ``6``).
175 Higher presets produce smaller output, but make the compression process
176 slower.
177
178 .. note::
179
180 In addition to being more CPU-intensive, compression with higher presets
181 also requires much more memory (and produces output that needs more memory
182 to decompress). With preset ``9`` for example, the overhead for an
183 :class:`LZMACompressor` object can be as high as 800 MiB. For this reason,
184 it is generally best to stick with the default preset.
185
186 The *filters* argument (if provided) should be a filter chain specifier.
187 See :ref:`filter-chain-specs` for details.
188
189 .. method:: compress(data)
190
191 Compress *data* (a :class:`bytes` object), returning a :class:`bytes`
192 object containing compressed data for at least part of the input. Some of
193 *data* may be buffered internally, for use in later calls to
194 :meth:`compress` and :meth:`flush`. The returned data should be
195 concatenated with the output of any previous calls to :meth:`compress`.
196
197 .. method:: flush()
198
199 Finish the compression process, returning a :class:`bytes` object
200 containing any data stored in the compressor's internal buffers.
201
202 The compressor cannot be used after this method has been called.
203
204
205 .. class:: LZMADecompressor(format=FORMAT_AUTO, memlimit=None, filters=None)
206
207 Create a decompressor object, which can be used to decompress data
208 incrementally.
209
210 For a more convenient way of decompressing an entire compressed stream at
211 once, see :func:`decompress`.
212
213 The *format* argument specifies the container format that should be used. The
214 default is :const:`FORMAT_AUTO`, which can decompress both ``.xz`` and
215 ``.lzma`` files. Other possible values are :const:`FORMAT_XZ`,
216 :const:`FORMAT_ALONE`, and :const:`FORMAT_RAW`.
217
218 The *memlimit* argument specifies a limit (in bytes) on the amount of memory
219 that the decompressor can use. When this argument is used, decompression will
220 fail with an :class:`LZMAError` if it is not possible to decompress the input
221 within the given memory limit.
222
223 The *filters* argument specifies the filter chain that was used to create
224 the stream being decompressed. This argument is required if *format* is
225 :const:`FORMAT_RAW`, but should not be used for other formats.
226 See :ref:`filter-chain-specs` for more information about filter chains.
227
228 .. note::
229 This class does not transparently handle inputs containing multiple
230 compressed streams, unlike :func:`decompress` and :class:`LZMAFile`. To
231 decompress a multi-stream input with :class:`LZMADecompressor`, you must
232 create a new decompressor for each stream.
233
234 .. method:: decompress(data, max_length=-1)
235
236 Decompress *data* (a :term:`bytes-like object`), returning
237 uncompressed data as bytes. Some of *data* may be buffered
238 internally, for use in later calls to :meth:`decompress`. The
239 returned data should be concatenated with the output of any
240 previous calls to :meth:`decompress`.
241
242 If *max_length* is nonnegative, returns at most *max_length*
243 bytes of decompressed data. If this limit is reached and further
244 output can be produced, the :attr:`~.needs_input` attribute will
245 be set to ``False``. In this case, the next call to
246 :meth:`~.decompress` may provide *data* as ``b''`` to obtain
247 more of the output.
248
249 If all of the input data was decompressed and returned (either
250 because this was less than *max_length* bytes, or because
251 *max_length* was negative), the :attr:`~.needs_input` attribute
252 will be set to ``True``.
253
254 Attempting to decompress data after the end of stream is reached
255 raises an :exc:`EOFError`. Any data found after the end of the
256 stream is ignored and saved in the :attr:`~.unused_data` attribute.
257
258 .. versionchanged:: 3.5
259 Added the *max_length* parameter.
260
261 .. attribute:: check
262
263 The ID of the integrity check used by the input stream. This may be
264 :const:`CHECK_UNKNOWN` until enough of the input has been decoded to
265 determine what integrity check it uses.
266
267 .. attribute:: eof
268
269 ``True`` if the end-of-stream marker has been reached.
270
271 .. attribute:: unused_data
272
273 Data found after the end of the compressed stream.
274
275 Before the end of the stream is reached, this will be ``b""``.
276
277 .. attribute:: needs_input
278
279 ``False`` if the :meth:`.decompress` method can provide more
280 decompressed data before requiring new uncompressed input.
281
282 .. versionadded:: 3.5
283
284 .. function:: compress(data, format=FORMAT_XZ, check=-1, preset=None, filters=None)
285
286 Compress *data* (a :class:`bytes` object), returning the compressed data as a
287 :class:`bytes` object.
288
289 See :class:`LZMACompressor` above for a description of the *format*, *check*,
290 *preset* and *filters* arguments.
291
292
293 .. function:: decompress(data, format=FORMAT_AUTO, memlimit=None, filters=None)
294
295 Decompress *data* (a :class:`bytes` object), returning the uncompressed data
296 as a :class:`bytes` object.
297
298 If *data* is the concatenation of multiple distinct compressed streams,
299 decompress all of these streams, and return the concatenation of the results.
300
301 See :class:`LZMADecompressor` above for a description of the *format*,
302 *memlimit* and *filters* arguments.
303
304
305 Miscellaneous
306 -------------
307
308 .. function:: is_check_supported(check)
309
310 Return ``True`` if the given integrity check is supported on this system.
311
312 :const:`CHECK_NONE` and :const:`CHECK_CRC32` are always supported.
313 :const:`CHECK_CRC64` and :const:`CHECK_SHA256` may be unavailable if you are
314 using a version of :program:`liblzma` that was compiled with a limited
315 feature set.
316
317
318 .. _filter-chain-specs:
319
320 Specifying custom filter chains
321 -------------------------------
322
323 A filter chain specifier is a sequence of dictionaries, where each dictionary
324 contains the ID and options for a single filter. Each dictionary must contain
325 the key ``"id"``, and may contain additional keys to specify filter-dependent
326 options. Valid filter IDs are as follows:
327
328 * Compression filters:
329
330 * :const:`FILTER_LZMA1` (for use with :const:`FORMAT_ALONE`)
331 * :const:`FILTER_LZMA2` (for use with :const:`FORMAT_XZ` and :const:`FORMAT_RAW`)
332
333 * Delta filter:
334
335 * :const:`FILTER_DELTA`
336
337 * Branch-Call-Jump (BCJ) filters:
338
339 * :const:`!FILTER_X86`
340 * :const:`!FILTER_IA64`
341 * :const:`!FILTER_ARM`
342 * :const:`!FILTER_ARMTHUMB`
343 * :const:`!FILTER_POWERPC`
344 * :const:`!FILTER_SPARC`
345
346 A filter chain can consist of up to 4 filters, and cannot be empty. The last
347 filter in the chain must be a compression filter, and any other filters must be
348 delta or BCJ filters.
349
350 Compression filters support the following options (specified as additional
351 entries in the dictionary representing the filter):
352
353 * ``preset``: A compression preset to use as a source of default values for
354 options that are not specified explicitly.
355 * ``dict_size``: Dictionary size in bytes. This should be between 4 KiB and
356 1.5 GiB (inclusive).
357 * ``lc``: Number of literal context bits.
358 * ``lp``: Number of literal position bits. The sum ``lc + lp`` must be at
359 most 4.
360 * ``pb``: Number of position bits; must be at most 4.
361 * ``mode``: :const:`MODE_FAST` or :const:`MODE_NORMAL`.
362 * ``nice_len``: What should be considered a "nice length" for a match.
363 This should be 273 or less.
364 * ``mf``: What match finder to use -- :const:`MF_HC3`, :const:`MF_HC4`,
365 :const:`MF_BT2`, :const:`MF_BT3`, or :const:`MF_BT4`.
366 * ``depth``: Maximum search depth used by match finder. 0 (default) means to
367 select automatically based on other filter options.
368
369 The delta filter stores the differences between bytes, producing more repetitive
370 input for the compressor in certain circumstances. It supports one option,
371 ``dist``. This indicates the distance between bytes to be subtracted. The
372 default is 1, i.e. take the differences between adjacent bytes.
373
374 The BCJ filters are intended to be applied to machine code. They convert
375 relative branches, calls and jumps in the code to use absolute addressing, with
376 the aim of increasing the redundancy that can be exploited by the compressor.
377 These filters support one option, ``start_offset``. This specifies the address
378 that should be mapped to the beginning of the input data. The default is 0.
379
380
381 Constants
382 ---------
383
384 The following module-level constants are provided for use as the *format*,
385 *check*, *preset* and *filters* arguments of the classes and functions above.
386
387 Container formats:
388
389 .. data:: FORMAT_XZ
390
391 The ``.xz`` container format.
392
393 .. data:: FORMAT_ALONE
394
395 The legacy ``.lzma`` container format. This format is more limited than
396 ``.xz`` -- it does not support integrity checks or multiple filters.
397
398 .. data:: FORMAT_RAW
399
400 A raw data stream, not using any container format. This format specifier
401 does not support integrity checks, and requires that you always specify a
402 custom filter chain (for both compression and decompression). Additionally,
403 data compressed in this manner cannot be decompressed using
404 :const:`FORMAT_AUTO`.
405
406 .. data:: FORMAT_AUTO
407
408 Used for decompression only. The container format is detected
409 automatically, so that both ``.xz`` and ``.lzma`` files can be decompressed.
410
411 Integrity checks:
412
413 .. data:: CHECK_NONE
414
415 No integrity check. This is the default (and the only acceptable value) for
416 :const:`FORMAT_ALONE` and :const:`FORMAT_RAW`.
417
418 .. data:: CHECK_CRC32
419
420 A 32-bit Cyclic Redundancy Check.
421
422 .. data:: CHECK_CRC64
423
424 A 64-bit Cyclic Redundancy Check. This is the default for
425 :const:`FORMAT_XZ`.
426
427 .. data:: CHECK_SHA256
428
429 A 256-bit Secure Hash Algorithm.
430
431 .. data:: CHECK_UNKNOWN
432
433 The integrity check used by a stream could not yet be determined. This may
434 be the value of the :attr:`LZMADecompressor.check` attribute until enough of
435 the input has been decoded.
436
437 .. data:: CHECK_ID_MAX
438
439 The largest supported integrity-check ID.
440
441 Compression presets:
442
443 .. data:: PRESET_DEFAULT
444
445 The default compression preset, equivalent to preset level ``6``.
446
447 .. data:: PRESET_EXTREME
448
449 A flag that may be bitwise OR-ed with a preset level (``0`` to ``9``) to
450 select a slower but more thorough variant of that preset.
451
452 Filter IDs and options:
453
454 .. data:: FILTER_LZMA1
455 FILTER_LZMA2
456
457 The LZMA1 and LZMA2 compression filters. :const:`FILTER_LZMA1` is for use
458 with :const:`FORMAT_ALONE`, while :const:`FILTER_LZMA2` is for use with
459 :const:`FORMAT_XZ` and :const:`FORMAT_RAW`.
460
461 .. data:: FILTER_DELTA
462
463 The delta filter.
464
465 .. data:: MODE_FAST
466 MODE_NORMAL
467
468 Compression modes that may be used as the ``mode`` option of a filter
469 specifier (see :ref:`filter-chain-specs`).
470
471 .. data:: MF_HC3
472 MF_HC4
473 MF_BT2
474 MF_BT3
475 MF_BT4
476
477 Match finders that may be used as the ``mf`` option of a filter specifier
478 (see :ref:`filter-chain-specs`).
479
480
481 Examples
482 --------
483
484 Reading in a compressed file::
485
486 import lzma
487 with lzma.open("file.xz") as f:
488 file_content = f.read()
489
490 Creating a compressed file::
491
492 import lzma
493 data = b"Insert Data Here"
494 with lzma.open("file.xz", "w") as f:
495 f.write(data)
496
497 Compressing data in memory::
498
499 import lzma
500 data_in = b"Insert Data Here"
501 data_out = lzma.compress(data_in)
502
503 Incremental compression::
504
505 import lzma
506 lzc = lzma.LZMACompressor()
507 out1 = lzc.compress(b"Some data\n")
508 out2 = lzc.compress(b"Another piece of data\n")
509 out3 = lzc.compress(b"Even more data\n")
510 out4 = lzc.flush()
511 # Concatenate all the partial results:
512 result = b"".join([out1, out2, out3, out4])
513
514 Writing compressed data to an already-open file::
515
516 import lzma
517 with open("file.xz", "wb") as f:
518 f.write(b"This data will not be compressed\n")
519 with lzma.open(f, "w") as lzf:
520 lzf.write(b"This *will* be compressed\n")
521 f.write(b"Not compressed\n")
522
523 Creating a compressed file using a custom filter chain::
524
525 import lzma
526 my_filters = [
527 {"id": lzma.FILTER_DELTA, "dist": 5},
528 {"id": lzma.FILTER_LZMA2, "preset": 7 | lzma.PRESET_EXTREME},
529 ]
530 with lzma.open("file.xz", "w", filters=my_filters) as f:
531 f.write(b"blah blah blah")