| 1 |
.TH bzip2 1 |
| 2 |
.SH NAME |
| 3 |
bzip2, bunzip2 \- a block-sorting file compressor, v1.0.8 |
| 4 |
.br |
| 5 |
bzcat \- decompresses files to stdout |
| 6 |
.br |
| 7 |
bzip2recover \- recovers data from damaged bzip2 files |
| 8 |
|
| 9 |
.SH SYNOPSIS |
| 10 |
.ll +8 |
| 11 |
.B bzip2 |
| 12 |
.RB [ " \-cdfkqstvzVL123456789 " ] |
| 13 |
[ |
| 14 |
.I "filenames \&..." |
| 15 |
] |
| 16 |
.br |
| 17 |
.B bzip2 |
| 18 |
.RB [ " \-h|\-\-help " ] |
| 19 |
.ll -8 |
| 20 |
.br |
| 21 |
.B bunzip2 |
| 22 |
.RB [ " \-fkvsVL " ] |
| 23 |
[ |
| 24 |
.I "filenames \&..." |
| 25 |
] |
| 26 |
.br |
| 27 |
.B bunzip2 |
| 28 |
.RB [ " \-h|\-\-help " ] |
| 29 |
.br |
| 30 |
.B bzcat |
| 31 |
.RB [ " \-s " ] |
| 32 |
[ |
| 33 |
.I "filenames \&..." |
| 34 |
] |
| 35 |
.br |
| 36 |
.B bzcat |
| 37 |
.RB [ " \-h|\-\-help " ] |
| 38 |
.br |
| 39 |
.B bzip2recover |
| 40 |
.I "filename" |
| 41 |
|
| 42 |
.SH DESCRIPTION |
| 43 |
.I bzip2 |
| 44 |
compresses files using the Burrows-Wheeler block sorting |
| 45 |
text compression algorithm, and Huffman coding. Compression is |
| 46 |
generally considerably better than that achieved by more conventional |
| 47 |
LZ77/LZ78-based compressors, and approaches the performance of the PPM |
| 48 |
family of statistical compressors. |
| 49 |
|
| 50 |
The command-line options are deliberately very similar to |
| 51 |
those of |
| 52 |
.I GNU gzip, |
| 53 |
but they are not identical. |
| 54 |
|
| 55 |
.I bzip2 |
| 56 |
expects a list of file names to accompany the |
| 57 |
command-line flags. Each file is replaced by a compressed version of |
| 58 |
itself, with the name "original_name.bz2". |
| 59 |
Each compressed file |
| 60 |
has the same modification date, permissions, and, when possible, |
| 61 |
ownership as the corresponding original, so that these properties can |
| 62 |
be correctly restored at decompression time. File name handling is |
| 63 |
naive in the sense that there is no mechanism for preserving original |
| 64 |
file names, permissions, ownerships or dates in filesystems which lack |
| 65 |
these concepts, or have serious file name length restrictions, such as |
| 66 |
MS-DOS. |
| 67 |
|
| 68 |
.I bzip2 |
| 69 |
and |
| 70 |
.I bunzip2 |
| 71 |
will by default not overwrite existing |
| 72 |
files. If you want this to happen, specify the \-f flag. |
| 73 |
|
| 74 |
If no file names are specified, |
| 75 |
.I bzip2 |
| 76 |
compresses from standard |
| 77 |
input to standard output. In this case, |
| 78 |
.I bzip2 |
| 79 |
will decline to |
| 80 |
write compressed output to a terminal, as this would be entirely |
| 81 |
incomprehensible and therefore pointless. |
| 82 |
|
| 83 |
.I bunzip2 |
| 84 |
(or |
| 85 |
.I bzip2 \-d) |
| 86 |
decompresses all |
| 87 |
specified files. Files which were not created by |
| 88 |
.I bzip2 |
| 89 |
will be detected and ignored, and a warning issued. |
| 90 |
.I bzip2 |
| 91 |
attempts to guess the filename for the decompressed file |
| 92 |
from that of the compressed file as follows: |
| 93 |
|
| 94 |
filename.bz2 becomes filename |
| 95 |
filename.bz becomes filename |
| 96 |
filename.tbz2 becomes filename.tar |
| 97 |
filename.tbz becomes filename.tar |
| 98 |
anyothername becomes anyothername.out |
| 99 |
|
| 100 |
If the file does not end in one of the recognised endings, |
| 101 |
.I .bz2, |
| 102 |
.I .bz, |
| 103 |
.I .tbz2 |
| 104 |
or |
| 105 |
.I .tbz, |
| 106 |
.I bzip2 |
| 107 |
complains that it cannot |
| 108 |
guess the name of the original file, and uses the original name |
| 109 |
with |
| 110 |
.I .out |
| 111 |
appended. |
| 112 |
|
| 113 |
As with compression, supplying no |
| 114 |
filenames causes decompression from |
| 115 |
standard input to standard output. |
| 116 |
|
| 117 |
.I bunzip2 |
| 118 |
will correctly decompress a file which is the |
| 119 |
concatenation of two or more compressed files. The result is the |
| 120 |
concatenation of the corresponding uncompressed files. Integrity |
| 121 |
testing (\-t) |
| 122 |
of concatenated |
| 123 |
compressed files is also supported. |
| 124 |
|
| 125 |
You can also compress or decompress files to the standard output by |
| 126 |
giving the \-c flag. Multiple files may be compressed and |
| 127 |
decompressed like this. The resulting outputs are fed sequentially to |
| 128 |
stdout. Compression of multiple files |
| 129 |
in this manner generates a stream |
| 130 |
containing multiple compressed file representations. Such a stream |
| 131 |
can be decompressed correctly only by |
| 132 |
.I bzip2 |
| 133 |
version 0.9.0 or |
| 134 |
later. Earlier versions of |
| 135 |
.I bzip2 |
| 136 |
will stop after decompressing |
| 137 |
the first file in the stream. |
| 138 |
|
| 139 |
.I bzcat |
| 140 |
(or |
| 141 |
.I bzip2 -dc) |
| 142 |
decompresses all specified files to |
| 143 |
the standard output. |
| 144 |
|
| 145 |
.I bzip2 |
| 146 |
will read arguments from the environment variables |
| 147 |
.I BZIP2 |
| 148 |
and |
| 149 |
.I BZIP, |
| 150 |
in that order, and will process them |
| 151 |
before any arguments read from the command line. This gives a |
| 152 |
convenient way to supply default arguments. |
| 153 |
|
| 154 |
Compression is always performed, even if the compressed |
| 155 |
file is slightly |
| 156 |
larger than the original. Files of less than about one hundred bytes |
| 157 |
tend to get larger, since the compression mechanism has a constant |
| 158 |
overhead in the region of 50 bytes. Random data (including the output |
| 159 |
of most file compressors) is coded at about 8.05 bits per byte, giving |
| 160 |
an expansion of around 0.5%. |
| 161 |
|
| 162 |
As a self-check for your protection, |
| 163 |
.I bzip2 |
| 164 |
uses 32-bit CRCs to |
| 165 |
make sure that the decompressed version of a file is identical to the |
| 166 |
original. This guards against corruption of the compressed data, and |
| 167 |
against undetected bugs in |
| 168 |
.I bzip2 |
| 169 |
(hopefully very unlikely). The |
| 170 |
chances of data corruption going undetected is microscopic, about one |
| 171 |
chance in four billion for each file processed. Be aware, though, that |
| 172 |
the check occurs upon decompression, so it can only tell you that |
| 173 |
something is wrong. It can't help you |
| 174 |
recover the original uncompressed |
| 175 |
data. You can use |
| 176 |
.I bzip2recover |
| 177 |
to try to recover data from |
| 178 |
damaged files. |
| 179 |
|
| 180 |
Return values: 0 for a normal exit, 1 for environmental problems (file |
| 181 |
not found, invalid flags, I/O errors, &c), 2 to indicate a corrupt |
| 182 |
compressed file, 3 for an internal consistency error (eg, bug) which |
| 183 |
caused |
| 184 |
.I bzip2 |
| 185 |
to panic. |
| 186 |
|
| 187 |
.SH OPTIONS |
| 188 |
.TP |
| 189 |
.B \-c --stdout |
| 190 |
Compress or decompress to standard output. |
| 191 |
.TP |
| 192 |
.B \-d --decompress |
| 193 |
Force decompression. |
| 194 |
.I bzip2, |
| 195 |
.I bunzip2 |
| 196 |
and |
| 197 |
.I bzcat |
| 198 |
are |
| 199 |
really the same program, and the decision about what actions to take is |
| 200 |
done on the basis of which name is used. This flag overrides that |
| 201 |
mechanism, and forces |
| 202 |
.I bzip2 |
| 203 |
to decompress. |
| 204 |
.TP |
| 205 |
.B \-z --compress |
| 206 |
The complement to \-d: forces compression, regardless of the |
| 207 |
invocation name. |
| 208 |
.TP |
| 209 |
.B \-t --test |
| 210 |
Check integrity of the specified file(s), but don't decompress them. |
| 211 |
This really performs a trial decompression and throws away the result. |
| 212 |
.TP |
| 213 |
.B \-f --force |
| 214 |
Force overwrite of output files. Normally, |
| 215 |
.I bzip2 |
| 216 |
will not overwrite |
| 217 |
existing output files. Also forces |
| 218 |
.I bzip2 |
| 219 |
to break hard links |
| 220 |
to files, which it otherwise wouldn't do. |
| 221 |
|
| 222 |
bzip2 normally declines to decompress files which don't have the |
| 223 |
correct magic header bytes. If forced (-f), however, it will pass |
| 224 |
such files through unmodified. This is how GNU gzip behaves. |
| 225 |
.TP |
| 226 |
.B \-k --keep |
| 227 |
Keep (don't delete) input files during compression |
| 228 |
or decompression. |
| 229 |
.TP |
| 230 |
.B \-s --small |
| 231 |
Reduce memory usage, for compression, decompression and testing. Files |
| 232 |
are decompressed and tested using a modified algorithm which only |
| 233 |
requires 2.5 bytes per block byte. This means any file can be |
| 234 |
decompressed in 2300\ k of memory, albeit at about half the normal speed. |
| 235 |
|
| 236 |
During compression, \-s selects a block size of 200\ k, which limits |
| 237 |
memory use to around the same figure, at the expense of your compression |
| 238 |
ratio. In short, if your machine is low on memory (8 megabytes or |
| 239 |
less), use \-s for everything. See MEMORY MANAGEMENT below. |
| 240 |
.TP |
| 241 |
.B \-q --quiet |
| 242 |
Suppress non-essential warning messages. Messages pertaining to |
| 243 |
I/O errors and other critical events will not be suppressed. |
| 244 |
.TP |
| 245 |
.B \-v --verbose |
| 246 |
Verbose mode -- show the compression ratio for each file processed. |
| 247 |
Further \-v's increase the verbosity level, spewing out lots of |
| 248 |
information which is primarily of interest for diagnostic purposes. |
| 249 |
.TP |
| 250 |
.B \-h \-\-help |
| 251 |
Print a help message and exit. |
| 252 |
.TP |
| 253 |
.B \-L --license -V --version |
| 254 |
Display the software version, license terms and conditions. |
| 255 |
.TP |
| 256 |
.B \-1 (or \-\-fast) to \-9 (or \-\-best) |
| 257 |
Set the block size to 100 k, 200 k ... 900 k when compressing. Has no |
| 258 |
effect when decompressing. See MEMORY MANAGEMENT below. |
| 259 |
The \-\-fast and \-\-best aliases are primarily for GNU gzip |
| 260 |
compatibility. In particular, \-\-fast doesn't make things |
| 261 |
significantly faster. |
| 262 |
And \-\-best merely selects the default behaviour. |
| 263 |
.TP |
| 264 |
.B \-- |
| 265 |
Treats all subsequent arguments as file names, even if they start |
| 266 |
with a dash. This is so you can handle files with names beginning |
| 267 |
with a dash, for example: bzip2 \-- \-myfilename. |
| 268 |
.TP |
| 269 |
.B \--repetitive-fast --repetitive-best |
| 270 |
These flags are redundant in versions 0.9.5 and above. They provided |
| 271 |
some coarse control over the behaviour of the sorting algorithm in |
| 272 |
earlier versions, which was sometimes useful. 0.9.5 and above have an |
| 273 |
improved algorithm which renders these flags irrelevant. |
| 274 |
|
| 275 |
.SH MEMORY MANAGEMENT |
| 276 |
.I bzip2 |
| 277 |
compresses large files in blocks. The block size affects |
| 278 |
both the compression ratio achieved, and the amount of memory needed for |
| 279 |
compression and decompression. The flags \-1 through \-9 |
| 280 |
specify the block size to be 100,000 bytes through 900,000 bytes (the |
| 281 |
default) respectively. At decompression time, the block size used for |
| 282 |
compression is read from the header of the compressed file, and |
| 283 |
.I bunzip2 |
| 284 |
then allocates itself just enough memory to decompress |
| 285 |
the file. Since block sizes are stored in compressed files, it follows |
| 286 |
that the flags \-1 to \-9 are irrelevant to and so ignored |
| 287 |
during decompression. |
| 288 |
|
| 289 |
Compression and decompression requirements, |
| 290 |
in bytes, can be estimated as: |
| 291 |
|
| 292 |
Compression: 400\ k + ( 8 x block size ) |
| 293 |
|
| 294 |
Decompression: 100\ k + ( 4 x block size ), or |
| 295 |
100\ k + ( 2.5 x block size ) |
| 296 |
|
| 297 |
Larger block sizes give rapidly diminishing marginal returns. Most of |
| 298 |
the compression comes from the first two or three hundred k of block |
| 299 |
size, a fact worth bearing in mind when using |
| 300 |
.I bzip2 |
| 301 |
on small machines. |
| 302 |
It is also important to appreciate that the decompression memory |
| 303 |
requirement is set at compression time by the choice of block size. |
| 304 |
|
| 305 |
For files compressed with the default 900\ k block size, |
| 306 |
.I bunzip2 |
| 307 |
will require about 3700 kbytes to decompress. To support decompression |
| 308 |
of any file on a 4 megabyte machine, |
| 309 |
.I bunzip2 |
| 310 |
has an option to |
| 311 |
decompress using approximately half this amount of memory, about 2300 |
| 312 |
kbytes. Decompression speed is also halved, so you should use this |
| 313 |
option only where necessary. The relevant flag is -s. |
| 314 |
|
| 315 |
In general, try and use the largest block size memory constraints allow, |
| 316 |
since that maximises the compression achieved. Compression and |
| 317 |
decompression speed are virtually unaffected by block size. |
| 318 |
|
| 319 |
Another significant point applies to files which fit in a single block |
| 320 |
-- that means most files you'd encounter using a large block size. The |
| 321 |
amount of real memory touched is proportional to the size of the file, |
| 322 |
since the file is smaller than a block. For example, compressing a file |
| 323 |
20,000 bytes long with the flag -9 will cause the compressor to |
| 324 |
allocate around 7600\ k of memory, but only touch 400\ k + 20000 * 8 = 560 |
| 325 |
kbytes of it. Similarly, the decompressor will allocate 3700\ k but only |
| 326 |
touch 100\ k + 20000 * 4 = 180 kbytes. |
| 327 |
|
| 328 |
Here is a table which summarises the maximum memory usage for different |
| 329 |
block sizes. Also recorded is the total compressed size for 14 files of |
| 330 |
the Calgary Text Compression Corpus totalling 3,141,622 bytes. This |
| 331 |
column gives some feel for how compression varies with block size. |
| 332 |
These figures tend to understate the advantage of larger block sizes for |
| 333 |
larger files, since the Corpus is dominated by smaller files. |
| 334 |
|
| 335 |
Compress Decompress Decompress Corpus |
| 336 |
Flag usage usage -s usage Size |
| 337 |
|
| 338 |
-1 1200k 500k 350k 914704 |
| 339 |
-2 2000k 900k 600k 877703 |
| 340 |
-3 2800k 1300k 850k 860338 |
| 341 |
-4 3600k 1700k 1100k 846899 |
| 342 |
-5 4400k 2100k 1350k 845160 |
| 343 |
-6 5200k 2500k 1600k 838626 |
| 344 |
-7 6100k 2900k 1850k 834096 |
| 345 |
-8 6800k 3300k 2100k 828642 |
| 346 |
-9 7600k 3700k 2350k 828642 |
| 347 |
|
| 348 |
.SH RECOVERING DATA FROM DAMAGED FILES |
| 349 |
.I bzip2 |
| 350 |
compresses files in blocks, usually 900\ kbytes long. Each |
| 351 |
block is handled independently. If a media or transmission error causes |
| 352 |
a multi-block .bz2 |
| 353 |
file to become damaged, it may be possible to |
| 354 |
recover data from the undamaged blocks in the file. |
| 355 |
|
| 356 |
The compressed representation of each block is delimited by a 48-bit |
| 357 |
pattern, which makes it possible to find the block boundaries with |
| 358 |
reasonable certainty. Each block also carries its own 32-bit CRC, so |
| 359 |
damaged blocks can be distinguished from undamaged ones. |
| 360 |
|
| 361 |
.I bzip2recover |
| 362 |
is a simple program whose purpose is to search for |
| 363 |
blocks in .bz2 files, and write each block out into its own .bz2 |
| 364 |
file. You can then use |
| 365 |
.I bzip2 |
| 366 |
\-t |
| 367 |
to test the |
| 368 |
integrity of the resulting files, and decompress those which are |
| 369 |
undamaged. |
| 370 |
|
| 371 |
.I bzip2recover |
| 372 |
takes a single argument, the name of the damaged file, |
| 373 |
and writes a number of files "rec00001file.bz2", |
| 374 |
"rec00002file.bz2", etc., containing the extracted blocks. |
| 375 |
The output filenames are designed so that the use of |
| 376 |
wildcards in subsequent processing -- for example, |
| 377 |
"bzip2 -dc rec*file.bz2 > recovered_data" -- processes the files in |
| 378 |
the correct order. |
| 379 |
|
| 380 |
.I bzip2recover |
| 381 |
should be of most use dealing with large .bz2 |
| 382 |
files, as these will contain many blocks. It is clearly |
| 383 |
futile to use it on damaged single-block files, since a |
| 384 |
damaged block cannot be recovered. If you wish to minimise |
| 385 |
any potential data loss through media or transmission errors, |
| 386 |
you might consider compressing with a smaller |
| 387 |
block size. |
| 388 |
|
| 389 |
.SH PERFORMANCE NOTES |
| 390 |
The sorting phase of compression gathers together similar strings in the |
| 391 |
file. Because of this, files containing very long runs of repeated |
| 392 |
symbols, like "aabaabaabaab ...\&" (repeated several hundred times) may |
| 393 |
compress more slowly than normal. Versions 0.9.5 and above fare much |
| 394 |
better than previous versions in this respect. The ratio between |
| 395 |
worst-case and average-case compression time is in the region of 10:1. |
| 396 |
For previous versions, this figure was more like 100:1. You can use the |
| 397 |
\-vvvv option to monitor progress in great detail, if you want. |
| 398 |
|
| 399 |
Decompression speed is unaffected by these phenomena. |
| 400 |
|
| 401 |
.I bzip2 |
| 402 |
usually allocates several megabytes of memory to operate |
| 403 |
in, and then charges all over it in a fairly random fashion. This means |
| 404 |
that performance, both for compressing and decompressing, is largely |
| 405 |
determined by the speed at which your machine can service cache misses. |
| 406 |
Because of this, small changes to the code to reduce the miss rate have |
| 407 |
been observed to give disproportionately large performance improvements. |
| 408 |
I imagine |
| 409 |
.I bzip2 |
| 410 |
will perform best on machines with very large caches. |
| 411 |
|
| 412 |
.SH CAVEATS |
| 413 |
I/O error messages are not as helpful as they could be. |
| 414 |
.I bzip2 |
| 415 |
tries hard to detect I/O errors and exit cleanly, but the details of |
| 416 |
what the problem is sometimes seem rather misleading. |
| 417 |
|
| 418 |
This manual page pertains to version 1.0.8 of |
| 419 |
.I bzip2. |
| 420 |
Compressed data created by this version is entirely forwards and |
| 421 |
backwards compatible with the previous public releases, versions |
| 422 |
0.1pl2, 0.9.0, 0.9.5, 1.0.0, 1.0.1, 1.0.2 and above, but with the following |
| 423 |
exception: 0.9.0 and above can correctly decompress multiple |
| 424 |
concatenated compressed files. 0.1pl2 cannot do this; it will stop |
| 425 |
after decompressing just the first file in the stream. |
| 426 |
|
| 427 |
.I bzip2recover |
| 428 |
versions prior to 1.0.2 used 32-bit integers to represent |
| 429 |
bit positions in compressed files, so they could not handle compressed |
| 430 |
files more than 512 megabytes long. Versions 1.0.2 and above use |
| 431 |
64-bit ints on some platforms which support them (GNU supported |
| 432 |
targets, and Windows). To establish whether or not bzip2recover was |
| 433 |
built with such a limitation, run it without arguments. In any event |
| 434 |
you can build yourself an unlimited version if you can recompile it |
| 435 |
with MaybeUInt64 set to be an unsigned 64-bit integer. |
| 436 |
|
| 437 |
|
| 438 |
|
| 439 |
.SH AUTHOR |
| 440 |
Julian Seward, [email protected]. |
| 441 |
|
| 442 |
https://sourceware.org/bzip2/ |
| 443 |
|
| 444 |
The ideas embodied in |
| 445 |
.I bzip2 |
| 446 |
are due to (at least) the following |
| 447 |
people: Michael Burrows and David Wheeler (for the block sorting |
| 448 |
transformation), David Wheeler (again, for the Huffman coder), Peter |
| 449 |
Fenwick (for the structured coding model in the original |
| 450 |
.I bzip, |
| 451 |
and many refinements), and Alistair Moffat, Radford Neal and Ian Witten |
| 452 |
(for the arithmetic coder in the original |
| 453 |
.I bzip). |
| 454 |
I am much |
| 455 |
indebted for their help, support and advice. See the manual in the |
| 456 |
source distribution for pointers to sources of documentation. Christian |
| 457 |
von Roques encouraged me to look for faster sorting algorithms, so as to |
| 458 |
speed up compression. Bela Lubkin encouraged me to improve the |
| 459 |
worst-case compression performance. |
| 460 |
Donna Robinson XMLised the documentation. |
| 461 |
The bz* scripts are derived from those of GNU gzip. |
| 462 |
Many people sent patches, helped |
| 463 |
with portability problems, lent machines, gave advice and were generally |
| 464 |
helpful. |