| 1 |
% bup-midx(1) Bup %BUP_VERSION% |
| 2 |
% Avery Pennarun <[email protected]> |
| 3 |
% %BUP_DATE% |
| 4 |
|
| 5 |
# NAME |
| 6 |
|
| 7 |
bup-midx - create a multi-index (`.midx`) file from several `.idx` files |
| 8 |
|
| 9 |
# SYNOPSIS |
| 10 |
|
| 11 |
bup midx [-o *outfile*] \<-a|-f|*idxnames*...\> |
| 12 |
|
| 13 |
# DESCRIPTION |
| 14 |
|
| 15 |
`bup midx` creates a multi-index (`.midx`) file from one or more |
| 16 |
git pack index (`.idx`) files. |
| 17 |
|
| 18 |
Note: you should no longer need to run this command by hand. |
| 19 |
It gets run automatically by `bup-save`(1) and similar |
| 20 |
commands. |
| 21 |
|
| 22 |
# OPTIONS |
| 23 |
|
| 24 |
-o, \--output=*filename.midx* |
| 25 |
: use the given output filename for the `.midx` file. |
| 26 |
Default is auto-generated. |
| 27 |
|
| 28 |
-a, \--auto |
| 29 |
: automatically generate new `.midx` files for any `.idx` |
| 30 |
files where it would be appropriate. |
| 31 |
|
| 32 |
-f, \--force |
| 33 |
: force generation of a single new `.midx` file containing |
| 34 |
*all* your `.idx` files, even if other `.midx` files |
| 35 |
already exist. This will result in the fastest backup |
| 36 |
performance, but may take a long time to run. |
| 37 |
|
| 38 |
\--dir=*packdir* |
| 39 |
: specify the directory containing the `.idx`/`.midx` files |
| 40 |
to work with. The default is `$BUP_DIR/objects/pack`. |
| 41 |
|
| 42 |
\--max-files |
| 43 |
: maximum number of `.idx` files to open at a time. You |
| 44 |
can use this if you have an especially small number of file |
| 45 |
descriptors available, so that midx can complete |
| 46 |
(though possibly non-optimally) even if it can't open |
| 47 |
all your `.idx` files at once. The default value of this |
| 48 |
option should be fine for most people. |
| 49 |
|
| 50 |
\--check |
| 51 |
: validate a `.midx` file by ensuring that all objects in |
| 52 |
its contained `.idx` files exist inside the `.midx`. May |
| 53 |
be useful for debugging. |
| 54 |
|
| 55 |
|
| 56 |
# EXAMPLES |
| 57 |
$ bup midx -a |
| 58 |
Merging 21 indexes (2278559 objects). |
| 59 |
Table size: 524288 (17 bits) |
| 60 |
Reading indexes: 100.00% (2278559/2278559), done. |
| 61 |
midx-b66d7c9afc4396187218f2936a87b865cf342672.midx |
| 62 |
|
| 63 |
# DISCUSSION |
| 64 |
|
| 65 |
By default, bup uses git-formatted pack files, which |
| 66 |
consist of a pack file (containing objects) and an idx |
| 67 |
file (containing a sorted list of object names and their |
| 68 |
offsets in the .pack file). |
| 69 |
|
| 70 |
Normal idx files are convenient because it means you can use |
| 71 |
`git`(1) to access your backup datasets. However, idx |
| 72 |
files can get slow when you have a lot of very large packs |
| 73 |
(which git typically doesn't have, but bup often does). |
| 74 |
|
| 75 |
bup `.midx` files consist of a single sorted list of all the objects |
| 76 |
contained in all the .pack files it references. This list |
| 77 |
can be binary searched in about log2(m) steps, where m is |
| 78 |
the total number of objects. |
| 79 |
|
| 80 |
To further speed up the search, midx files also have a |
| 81 |
variable-sized fanout table that reduces the first n |
| 82 |
steps of the binary search. With the help of this fanout |
| 83 |
table, bup can narrow down which page of the midx file a |
| 84 |
given object id would be in (if it exists) with a single |
| 85 |
lookup. Thus, typical searches will only need to swap in |
| 86 |
two pages: one for the fanout table, and one for the object |
| 87 |
id. |
| 88 |
|
| 89 |
midx files are most useful when creating new backups, since |
| 90 |
searching for a nonexistent object in the repository |
| 91 |
necessarily requires searching through *all* the index |
| 92 |
files to ensure that it does not exist. (Searching for |
| 93 |
objects that *do* exist can be optimized; for example, |
| 94 |
consecutive objects are often stored in the same pack, so |
| 95 |
we can search that one first using an MRU algorithm.) |
| 96 |
|
| 97 |
|
| 98 |
# SEE ALSO |
| 99 |
|
| 100 |
`bup-save`(1), `bup-margin`(1), `bup-memtest`(1) |
| 101 |
|
| 102 |
# BUP |
| 103 |
|
| 104 |
Part of the `bup`(1) suite. |