Debian Codegraph @

Doc/library/lzma.rst

raw
1 :mod:`!lzma` --- Compression using the LZMA algorithm
2 =====================================================
3
4 .. module:: lzma
5 :synopsis: A Python wrapper for the liblzma compression library.
6
7 .. moduleauthor:: Nadeem Vawda <[email protected]>
8 .. sectionauthor:: Nadeem Vawda <[email protected]>
9
10 .. versionadded:: 3.3
11
12 **Source code:** :source:`Lib/lzma.py`
13
14 --------------
15
16 This module provides classes and convenience functions for compressing and
17 decompressing data using the LZMA compression algorithm. Also included is a file
18 interface supporting the ``.xz`` and legacy ``.lzma`` file formats used by the
19 :program:`xz` utility, as well as raw compressed streams.
20
21 The interface provided by this module is very similar to that of the :mod:`bz2`
22 module. Note that :class:`LZMAFile` and :class:`bz2.BZ2File` are *not*
23 thread-safe, so if you need to use a single :class:`LZMAFile` instance
24 from multiple threads, it is necessary to protect it with a lock.
25
26 .. include:: ../includes/optional-module.rst
27
28
29 .. exception:: LZMAError
30
31 This exception is raised when an error occurs during compression or
32 decompression, or while initializing the compressor/decompressor state.
33
34
35 Reading and writing compressed files
36 ------------------------------------
37
38 .. function:: open(filename, mode="rb", *, format=None, check=-1, preset=None, filters=None, encoding=None, errors=None, newline=None)
39
40 Open an LZMA-compressed file in binary or text mode, returning a :term:`file
41 object`.
42
43 The *filename* argument can be either an actual file name (given as a
44 :class:`str`, :class:`bytes` or :term:`path-like <path-like object>` object), in
45 which case the named file is opened, or it can be an existing file object
46 to read from or write to.
47
48 The *mode* argument can be any of ``"r"``, ``"rb"``, ``"w"``, ``"wb"``,
49 ``"x"``, ``"xb"``, ``"a"`` or ``"ab"`` for binary mode, or ``"rt"``,
50 ``"wt"``, ``"xt"``, or ``"at"`` for text mode. The default is ``"rb"``.
51
52 When opening a file for reading, the *format* and *filters* arguments have
53 the same meanings as for :class:`LZMADecompressor`. In this case, the *check*
54 and *preset* arguments should not be used.
55
56 When opening a file for writing, the *format*, *check*, *preset* and
57 *filters* arguments have the same meanings as for :class:`LZMACompressor`.
58
59 For binary mode, this function is equivalent to the :class:`LZMAFile`
60 constructor: ``LZMAFile(filename, mode, ...)``. In this case, the *encoding*,
61 *errors* and *newline* arguments must not be provided.
62
63 For text mode, a :class:`LZMAFile` object is created, and wrapped in an
64 :class:`io.TextIOWrapper` instance with the specified encoding, error
65 handling behavior, and line ending(s).
66
67 .. versionchanged:: 3.4
68 Added support for the ``"x"``, ``"xb"`` and ``"xt"`` modes.
69
70 .. versionchanged:: 3.6
71 Accepts a :term:`path-like object`.
72
73
74 .. class:: LZMAFile(filename=None, mode="r", *, format=None, check=-1, preset=None, filters=None)
75
76 Open an LZMA-compressed file in binary mode.
77
78 An :class:`LZMAFile` can wrap an already-open :term:`file object`, or operate
79 directly on a named file. The *filename* argument specifies either the file
80 object to wrap, or the name of the file to open (as a :class:`str`,
81 :class:`bytes` or :term:`path-like <path-like object>` object). When wrapping an
82 existing file object, the wrapped file will not be closed when the
83 :class:`LZMAFile` is closed.
84
85 The *mode* argument can be either ``"r"`` for reading (default), ``"w"`` for
86 overwriting, ``"x"`` for exclusive creation, or ``"a"`` for appending. These
87 can equivalently be given as ``"rb"``, ``"wb"``, ``"xb"`` and ``"ab"``
88 respectively.
89
90 If *filename* is a file object (rather than an actual file name), a mode of
91 ``"w"`` does not truncate the file, and is instead equivalent to ``"a"``.
92
93 When opening a file for reading, the input file may be the concatenation of
94 multiple separate compressed streams. These are transparently decoded as a
95 single logical stream.
96
97 When opening a file for reading, the *format* and *filters* arguments have
98 the same meanings as for :class:`LZMADecompressor`. In this case, the *check*
99 and *preset* arguments should not be used.
100
101 When opening a file for writing, the *format*, *check*, *preset* and
102 *filters* arguments have the same meanings as for :class:`LZMACompressor`.
103
104 :class:`LZMAFile` supports all the members specified by
105 :class:`io.BufferedIOBase`, except for :meth:`~io.BufferedIOBase.detach`
106 and :meth:`~io.IOBase.truncate`.
107 Iteration and the :keyword:`with` statement are supported.
108
109 The following method and attributes are also provided:
110
111 .. method:: peek(size=-1)
112
113 Return buffered data without advancing the file position. At least one
114 byte of data will be returned, unless EOF has been reached. The exact
115 number of bytes returned is unspecified (the *size* argument is ignored).
116
117 .. note:: While calling :meth:`peek` does not change the file position of
118 the :class:`LZMAFile`, it may change the position of the underlying
119 file object (e.g. if the :class:`LZMAFile` was constructed by passing a
120 file object for *filename*).
121
122 .. attribute:: mode
123
124 ``'rb'`` for reading and ``'wb'`` for writing.
125
126 .. versionadded:: 3.13
127
128 .. attribute:: name
129
130 The lzma file name. Equivalent to the :attr:`~io.FileIO.name`
131 attribute of the underlying :term:`file object`.
132
133 .. versionadded:: 3.13
134
135
136 .. versionchanged:: 3.4
137 Added support for the ``"x"`` and ``"xb"`` modes.
138
139 .. versionchanged:: 3.5
140 The :meth:`~io.BufferedIOBase.read` method now accepts an argument of
141 ``None``.
142
143 .. versionchanged:: 3.6
144 Accepts a :term:`path-like object`.
145
146
147 Compressing and decompressing data in memory
148 --------------------------------------------
149
150 .. class:: LZMACompressor(format=FORMAT_XZ, check=-1, preset=None, filters=None)
151
152 Create a compressor object, which can be used to compress data incrementally.
153
154 For a more convenient way of compressing a single chunk of data, see
155 :func:`compress`.
156
157 The *format* argument specifies what container format should be used.
158 Possible values are:
159
160 * :const:`FORMAT_XZ`: The ``.xz`` container format.
161 This is the default format.
162
163 * :const:`FORMAT_ALONE`: The legacy ``.lzma`` container format.
164 This format is more limited than ``.xz`` -- it does not support integrity
165 checks or multiple filters.
166
167 * :const:`FORMAT_RAW`: A raw data stream, not using any container format.
168 This format specifier does not support integrity checks, and requires that
169 you always specify a custom filter chain (for both compression and
170 decompression). Additionally, data compressed in this manner cannot be
171 decompressed using :const:`FORMAT_AUTO` (see :class:`LZMADecompressor`).
172
173 The *check* argument specifies the type of integrity check to include in the
174 compressed data. This check is used when decompressing, to ensure that the
175 data has not been corrupted. Possible values are:
176
177 * :const:`CHECK_NONE`: No integrity check.
178 This is the default (and the only acceptable value) for
179 :const:`FORMAT_ALONE` and :const:`FORMAT_RAW`.
180
181 * :const:`CHECK_CRC32`: 32-bit Cyclic Redundancy Check.
182
183 * :const:`CHECK_CRC64`: 64-bit Cyclic Redundancy Check.
184 This is the default for :const:`FORMAT_XZ`.
185
186 * :const:`CHECK_SHA256`: 256-bit Secure Hash Algorithm.
187
188 If the specified check is not supported, an :class:`LZMAError` is raised.
189
190 The compression settings can be specified either as a preset compression
191 level (with the *preset* argument), or in detail as a custom filter chain
192 (with the *filters* argument).
193
194 The *preset* argument (if provided) should be an integer between ``0`` and
195 ``9`` (inclusive), optionally OR-ed with the constant
196 :const:`PRESET_EXTREME`. If neither *preset* nor *filters* are given, the
197 default behavior is to use :const:`PRESET_DEFAULT` (preset level ``6``).
198 Higher presets produce smaller output, but make the compression process
199 slower.
200
201 .. note::
202
203 In addition to being more CPU-intensive, compression with higher presets
204 also requires much more memory (and produces output that needs more memory
205 to decompress). With preset ``9`` for example, the overhead for an
206 :class:`LZMACompressor` object can be as high as 800 MiB. For this reason,
207 it is generally best to stick with the default preset.
208
209 The *filters* argument (if provided) should be a filter chain specifier.
210 See :ref:`filter-chain-specs` for details.
211
212 .. method:: compress(data)
213
214 Compress *data* (a :class:`bytes` object), returning a :class:`bytes`
215 object containing compressed data for at least part of the input. Some of
216 *data* may be buffered internally, for use in later calls to
217 :meth:`compress` and :meth:`flush`. The returned data should be
218 concatenated with the output of any previous calls to :meth:`compress`.
219
220 .. method:: flush()
221
222 Finish the compression process, returning a :class:`bytes` object
223 containing any data stored in the compressor's internal buffers.
224
225 The compressor cannot be used after this method has been called.
226
227
228 .. class:: LZMADecompressor(format=FORMAT_AUTO, memlimit=None, filters=None)
229
230 Create a decompressor object, which can be used to decompress data
231 incrementally.
232
233 For a more convenient way of decompressing an entire compressed stream at
234 once, see :func:`decompress`.
235
236 The *format* argument specifies the container format that should be used. The
237 default is :const:`FORMAT_AUTO`, which can decompress both ``.xz`` and
238 ``.lzma`` files. Other possible values are :const:`FORMAT_XZ`,
239 :const:`FORMAT_ALONE`, and :const:`FORMAT_RAW`.
240
241 The *memlimit* argument specifies a limit (in bytes) on the amount of memory
242 that the decompressor can use. When this argument is used, decompression will
243 fail with an :class:`LZMAError` if it is not possible to decompress the input
244 within the given memory limit.
245
246 The *filters* argument specifies the filter chain that was used to create
247 the stream being decompressed. This argument is required if *format* is
248 :const:`FORMAT_RAW`, but should not be used for other formats.
249 See :ref:`filter-chain-specs` for more information about filter chains.
250
251 .. note::
252 This class does not transparently handle inputs containing multiple
253 compressed streams, unlike :func:`decompress` and :class:`LZMAFile`. To
254 decompress a multi-stream input with :class:`LZMADecompressor`, you must
255 create a new decompressor for each stream.
256
257 .. method:: decompress(data, max_length=-1)
258
259 Decompress *data* (a :term:`bytes-like object`), returning
260 uncompressed data as bytes. Some of *data* may be buffered
261 internally, for use in later calls to :meth:`decompress`. The
262 returned data should be concatenated with the output of any
263 previous calls to :meth:`decompress`.
264
265 If *max_length* is nonnegative, returns at most *max_length*
266 bytes of decompressed data. If this limit is reached and further
267 output can be produced, the :attr:`~.needs_input` attribute will
268 be set to ``False``. In this case, the next call to
269 :meth:`~.decompress` may provide *data* as ``b''`` to obtain
270 more of the output.
271
272 If all of the input data was decompressed and returned (either
273 because this was less than *max_length* bytes, or because
274 *max_length* was negative), the :attr:`~.needs_input` attribute
275 will be set to ``True``.
276
277 Attempting to decompress data after the end of stream is reached
278 raises an :exc:`EOFError`. Any data found after the end of the
279 stream is ignored and saved in the :attr:`~.unused_data` attribute.
280
281 .. versionchanged:: 3.5
282 Added the *max_length* parameter.
283
284 .. attribute:: check
285
286 The ID of the integrity check used by the input stream. This may be
287 :const:`CHECK_UNKNOWN` until enough of the input has been decoded to
288 determine what integrity check it uses.
289
290 .. attribute:: eof
291
292 ``True`` if the end-of-stream marker has been reached.
293
294 .. attribute:: unused_data
295
296 Data found after the end of the compressed stream.
297
298 Before the end of the stream is reached, this will be ``b""``.
299
300 .. attribute:: needs_input
301
302 ``False`` if the :meth:`.decompress` method can provide more
303 decompressed data before requiring new uncompressed input.
304
305 .. versionadded:: 3.5
306
307 .. function:: compress(data, format=FORMAT_XZ, check=-1, preset=None, filters=None)
308
309 Compress *data* (a :class:`bytes` object), returning the compressed data as a
310 :class:`bytes` object.
311
312 See :class:`LZMACompressor` above for a description of the *format*, *check*,
313 *preset* and *filters* arguments.
314
315
316 .. function:: decompress(data, format=FORMAT_AUTO, memlimit=None, filters=None)
317
318 Decompress *data* (a :class:`bytes` object), returning the uncompressed data
319 as a :class:`bytes` object.
320
321 If *data* is the concatenation of multiple distinct compressed streams,
322 decompress all of these streams, and return the concatenation of the results.
323
324 See :class:`LZMADecompressor` above for a description of the *format*,
325 *memlimit* and *filters* arguments.
326
327
328 Miscellaneous
329 -------------
330
331 .. function:: is_check_supported(check)
332
333 Return ``True`` if the given integrity check is supported on this system.
334
335 :const:`CHECK_NONE` and :const:`CHECK_CRC32` are always supported.
336 :const:`CHECK_CRC64` and :const:`CHECK_SHA256` may be unavailable if you are
337 using a version of :program:`liblzma` that was compiled with a limited
338 feature set.
339
340
341 .. _filter-chain-specs:
342
343 Specifying custom filter chains
344 -------------------------------
345
346 A filter chain specifier is a sequence of dictionaries, where each dictionary
347 contains the ID and options for a single filter. Each dictionary must contain
348 the key ``"id"``, and may contain additional keys to specify filter-dependent
349 options. Valid filter IDs are as follows:
350
351 * Compression filters:
352
353 * :const:`FILTER_LZMA1` (for use with :const:`FORMAT_ALONE`)
354 * :const:`FILTER_LZMA2` (for use with :const:`FORMAT_XZ` and :const:`FORMAT_RAW`)
355
356 * Delta filter:
357
358 * :const:`FILTER_DELTA`
359
360 * Branch-Call-Jump (BCJ) filters:
361
362 * :const:`FILTER_X86`
363 * :const:`FILTER_IA64`
364 * :const:`FILTER_ARM`
365 * :const:`FILTER_ARMTHUMB`
366 * :const:`FILTER_POWERPC`
367 * :const:`FILTER_SPARC`
368
369 A filter chain can consist of up to 4 filters, and cannot be empty. The last
370 filter in the chain must be a compression filter, and any other filters must be
371 delta or BCJ filters.
372
373 Compression filters support the following options (specified as additional
374 entries in the dictionary representing the filter):
375
376 * ``preset``: A compression preset to use as a source of default values for
377 options that are not specified explicitly.
378 * ``dict_size``: Dictionary size in bytes. This should be between 4 KiB and
379 1.5 GiB (inclusive).
380 * ``lc``: Number of literal context bits.
381 * ``lp``: Number of literal position bits. The sum ``lc + lp`` must be at
382 most 4.
383 * ``pb``: Number of position bits; must be at most 4.
384 * ``mode``: :const:`MODE_FAST` or :const:`MODE_NORMAL`.
385 * ``nice_len``: What should be considered a "nice length" for a match.
386 This should be 273 or less.
387 * ``mf``: What match finder to use -- :const:`MF_HC3`, :const:`MF_HC4`,
388 :const:`MF_BT2`, :const:`MF_BT3`, or :const:`MF_BT4`.
389 * ``depth``: Maximum search depth used by match finder. 0 (default) means to
390 select automatically based on other filter options.
391
392 The delta filter stores the differences between bytes, producing more repetitive
393 input for the compressor in certain circumstances. It supports one option,
394 ``dist``. This indicates the distance between bytes to be subtracted. The
395 default is 1, i.e. take the differences between adjacent bytes.
396
397 The BCJ filters are intended to be applied to machine code. They convert
398 relative branches, calls and jumps in the code to use absolute addressing, with
399 the aim of increasing the redundancy that can be exploited by the compressor.
400 These filters support one option, ``start_offset``. This specifies the address
401 that should be mapped to the beginning of the input data. The default is 0.
402
403
404 Examples
405 --------
406
407 Reading in a compressed file::
408
409 import lzma
410 with lzma.open("file.xz") as f:
411 file_content = f.read()
412
413 Creating a compressed file::
414
415 import lzma
416 data = b"Insert Data Here"
417 with lzma.open("file.xz", "w") as f:
418 f.write(data)
419
420 Compressing data in memory::
421
422 import lzma
423 data_in = b"Insert Data Here"
424 data_out = lzma.compress(data_in)
425
426 Incremental compression::
427
428 import lzma
429 lzc = lzma.LZMACompressor()
430 out1 = lzc.compress(b"Some data\n")
431 out2 = lzc.compress(b"Another piece of data\n")
432 out3 = lzc.compress(b"Even more data\n")
433 out4 = lzc.flush()
434 # Concatenate all the partial results:
435 result = b"".join([out1, out2, out3, out4])
436
437 Writing compressed data to an already-open file::
438
439 import lzma
440 with open("file.xz", "wb") as f:
441 f.write(b"This data will not be compressed\n")
442 with lzma.open(f, "w") as lzf:
443 lzf.write(b"This *will* be compressed\n")
444 f.write(b"Not compressed\n")
445
446 Creating a compressed file using a custom filter chain::
447
448 import lzma
449 my_filters = [
450 {"id": lzma.FILTER_DELTA, "dist": 5},
451 {"id": lzma.FILTER_LZMA2, "preset": 7 | lzma.PRESET_EXTREME},
452 ]
453 with lzma.open("file.xz", "w", filters=my_filters) as f:
454 f.write(b"blah blah blah")