| 1 |
.. SPDX-License-Identifier: 0BSD |
| 2 |
|
| 3 |
============================ |
| 4 |
XZ data compression in Linux |
| 5 |
============================ |
| 6 |
|
| 7 |
Introduction |
| 8 |
============ |
| 9 |
|
| 10 |
XZ is a general purpose data compression format with high compression |
| 11 |
ratio. The XZ decompressor in Linux is called XZ Embedded. It supports |
| 12 |
the LZMA2 filter and optionally also Branch/Call/Jump (BCJ) filters |
| 13 |
for executable code. CRC32 is supported for integrity checking. |
| 14 |
|
| 15 |
See the `XZ Embedded`_ home page for the latest version which includes |
| 16 |
a few optional extra features that aren't required in the Linux kernel |
| 17 |
and information about using the code outside the Linux kernel. |
| 18 |
|
| 19 |
For userspace, `XZ Utils`_ provide a zlib-like compression library |
| 20 |
and a gzip-like command line tool. |
| 21 |
|
| 22 |
.. _XZ Embedded: https://tukaani.org/xz/embedded.html |
| 23 |
.. _XZ Utils: https://tukaani.org/xz/ |
| 24 |
|
| 25 |
XZ related components in the kernel |
| 26 |
=================================== |
| 27 |
|
| 28 |
The xz_dec module provides XZ decompressor with single-call (buffer |
| 29 |
to buffer) and multi-call (stateful) APIs in include/linux/xz.h. |
| 30 |
|
| 31 |
For decompressing the kernel image, initramfs, and initrd, there |
| 32 |
is a wrapper function in lib/decompress_unxz.c. Its API is the |
| 33 |
same as in other decompress_*.c files, which is defined in |
| 34 |
include/linux/decompress/generic.h. |
| 35 |
|
| 36 |
For kernel makefiles, three commands are provided for use with |
| 37 |
``$(call if_changed)``. They require the xz tool from XZ Utils. |
| 38 |
|
| 39 |
- ``$(call if_changed,xzkern)`` is for compressing the kernel image. |
| 40 |
It runs the script scripts/xz_wrap.sh which uses arch-optimized |
| 41 |
options and a big LZMA2 dictionary. |
| 42 |
|
| 43 |
- ``$(call if_changed,xzkern_with_size)`` is like ``xzkern`` above but |
| 44 |
this also appends a four-byte trailer containing the uncompressed size |
| 45 |
of the file. The trailer is needed by the boot code on some archs. |
| 46 |
|
| 47 |
- Other things can be compressed with ``$(call if_needed,xzmisc)`` |
| 48 |
which will use no BCJ filter and 1 MiB LZMA2 dictionary. |
| 49 |
|
| 50 |
Notes on compression options |
| 51 |
============================ |
| 52 |
|
| 53 |
Since the XZ Embedded supports only streams with CRC32 or no integrity |
| 54 |
check, make sure that you don't use some other integrity check type |
| 55 |
when encoding files that are supposed to be decoded by the kernel. |
| 56 |
With liblzma from XZ Utils, you need to use either ``LZMA_CHECK_CRC32`` |
| 57 |
or ``LZMA_CHECK_NONE`` when encoding. With the ``xz`` command line tool, |
| 58 |
use ``--check=crc32`` or ``--check=none`` to override the default |
| 59 |
``--check=crc64``. |
| 60 |
|
| 61 |
Using CRC32 is strongly recommended unless there is some other layer |
| 62 |
which will verify the integrity of the uncompressed data anyway. |
| 63 |
Double checking the integrity would probably be waste of CPU cycles. |
| 64 |
Note that the headers will always have a CRC32 which will be validated |
| 65 |
by the decoder; you can only change the integrity check type (or |
| 66 |
disable it) for the actual uncompressed data. |
| 67 |
|
| 68 |
In userspace, LZMA2 is typically used with dictionary sizes of several |
| 69 |
megabytes. The decoder needs to have the dictionary in RAM: |
| 70 |
|
| 71 |
- In multi-call mode the dictionary is allocated as part of the |
| 72 |
decoder state. The reasonable maximum dictionary size for in-kernel |
| 73 |
use will depend on the target hardware: a few megabytes is fine for |
| 74 |
desktop systems while 64 KiB to 1 MiB might be more appropriate on |
| 75 |
some embedded systems. |
| 76 |
|
| 77 |
- In single-call mode the output buffer is used as the dictionary |
| 78 |
buffer. That is, the size of the dictionary doesn't affect the |
| 79 |
decompressor memory usage at all. Only the base data structures |
| 80 |
are allocated which take a little less than 30 KiB of memory. |
| 81 |
For the best compression, the dictionary should be at least |
| 82 |
as big as the uncompressed data. A notable example of single-call |
| 83 |
mode is decompressing the kernel itself (except on PowerPC). |
| 84 |
|
| 85 |
The compression presets in XZ Utils may not be optimal when creating |
| 86 |
files for the kernel, so don't hesitate to use custom settings to, |
| 87 |
for example, set the dictionary size. Also, xz may produce a smaller |
| 88 |
file in single-threaded mode so setting that explicitly is recommended. |
| 89 |
Example:: |
| 90 |
|
| 91 |
xz --threads=1 --check=crc32 --lzma2=dict=512KiB inputfile |
| 92 |
|
| 93 |
xz_dec API |
| 94 |
========== |
| 95 |
|
| 96 |
This is available with ``#include <linux/xz.h>``. |
| 97 |
|
| 98 |
.. kernel-doc:: include/linux/xz.h |