| 1 |
% bup-memtest(1) Bup %BUP_VERSION% |
| 2 |
% Avery Pennarun <[email protected]> |
| 3 |
% %BUP_DATE% |
| 4 |
|
| 5 |
# NAME |
| 6 |
|
| 7 |
bup-memtest - test bup memory usage statistics |
| 8 |
|
| 9 |
# SYNOPSIS |
| 10 |
|
| 11 |
bup memtest [options...] |
| 12 |
|
| 13 |
# DESCRIPTION |
| 14 |
|
| 15 |
`bup memtest` opens the list of pack indexes in your bup |
| 16 |
repository, then searches the list for a series of |
| 17 |
nonexistent objects, printing memory usage statistics after |
| 18 |
each cycle. |
| 19 |
|
| 20 |
Because of the way Unix systems work, the output will |
| 21 |
usually show a large (and unchanging) value in the VmSize |
| 22 |
column, because mapping the index files in the first place |
| 23 |
takes a certain amount of virtual address space. However, this |
| 24 |
virtual memory usage is entirely virtual; it doesn't take |
| 25 |
any of your RAM. Over time, bup uses *parts* of the |
| 26 |
indexes, which need to be loaded from disk, and this is |
| 27 |
what causes an increase in the VmRSS column. |
| 28 |
|
| 29 |
# OPTIONS |
| 30 |
|
| 31 |
-n, \--number=*number* |
| 32 |
: set the number of objects to search for during each |
| 33 |
cycle (ie. before printing a line of output) |
| 34 |
|
| 35 |
-c, \--cycles=*cycles* |
| 36 |
: set the number of cycles (ie. the number of lines of |
| 37 |
output after the first). The first line of output is |
| 38 |
always 0 (ie. the baseline before searching for any |
| 39 |
objects). |
| 40 |
|
| 41 |
\--ignore-midx |
| 42 |
: ignore any `.midx` files created by `bup midx`. This |
| 43 |
allows you to compare memory performance with and |
| 44 |
without using midx. |
| 45 |
|
| 46 |
\--existing |
| 47 |
: search for existing objects instead of searching for |
| 48 |
random nonexistent ones. This can greatly affect |
| 49 |
memory usage and performance. Note that most of the |
| 50 |
time, `bup save` spends most of its time searching for |
| 51 |
nonexistent objects, since existing ones are probably |
| 52 |
in unmodified files that we won't be trying to back up |
| 53 |
anyway. So the default behaviour reflects real bup |
| 54 |
performance more accurately. But you might want this |
| 55 |
option anyway just to make sure you haven't made |
| 56 |
searching for existing objects much worse than before. |
| 57 |
|
| 58 |
|
| 59 |
# EXAMPLES |
| 60 |
$ bup memtest -n300 -c5 |
| 61 |
PackIdxList: using 1 index. |
| 62 |
VmSize VmRSS VmData VmStk |
| 63 |
0 20824 kB 4528 kB 1980 kB 84 kB |
| 64 |
300 20828 kB 5828 kB 1984 kB 84 kB |
| 65 |
600 20828 kB 6844 kB 1984 kB 84 kB |
| 66 |
900 20828 kB 7836 kB 1984 kB 84 kB |
| 67 |
1200 20828 kB 8736 kB 1984 kB 84 kB |
| 68 |
1500 20828 kB 9452 kB 1984 kB 84 kB |
| 69 |
|
| 70 |
$ bup memtest -n300 -c5 --ignore-midx |
| 71 |
PackIdxList: using 361 indexes. |
| 72 |
VmSize VmRSS VmData VmStk |
| 73 |
0 27444 kB 6552 kB 2516 kB 84 kB |
| 74 |
300 27448 kB 15832 kB 2520 kB 84 kB |
| 75 |
600 27448 kB 17220 kB 2520 kB 84 kB |
| 76 |
900 27448 kB 18012 kB 2520 kB 84 kB |
| 77 |
1200 27448 kB 18388 kB 2520 kB 84 kB |
| 78 |
1500 27448 kB 18556 kB 2520 kB 84 kB |
| 79 |
|
| 80 |
|
| 81 |
# DISCUSSION |
| 82 |
|
| 83 |
When optimizing bup indexing, the first goal is to keep the |
| 84 |
VmRSS reasonably low. However, it might eventually be |
| 85 |
necessary to swap in all the indexes, simply because |
| 86 |
you're searching for a lot of objects, and this will cause |
| 87 |
your RSS to grow as large as VmSize eventually. |
| 88 |
|
| 89 |
The key word here is *eventually*. As long as VmRSS grows |
| 90 |
reasonably slowly, the amount of disk activity caused by |
| 91 |
accessing pack indexes is reasonably small. If it grows |
| 92 |
quickly, bup will probably spend most of its time swapping |
| 93 |
index data from disk instead of actually running your |
| 94 |
backup, so backups will run very slowly. |
| 95 |
|
| 96 |
The purpose of `bup memtest` is to give you an idea of how |
| 97 |
fast your memory usage is growing, and to help in |
| 98 |
optimizing bup for better memory use. If you have memory |
| 99 |
problems you might be asked to send the output of `bup |
| 100 |
memtest` to help diagnose the problems. |
| 101 |
|
| 102 |
Tip: try using `bup midx -a` or `bup midx -f` to see if it |
| 103 |
helps reduce your memory usage. |
| 104 |
|
| 105 |
Trivia: index memory usage in bup (or git) is only really a |
| 106 |
problem when adding a large number of previously unseen |
| 107 |
objects. This is because for each object, we need to |
| 108 |
absolutely confirm that it isn't already in the database, |
| 109 |
which requires us to search through *all* the existing pack |
| 110 |
indexes to ensure that none of them contain the object in |
| 111 |
question. In the more obvious case of searching for |
| 112 |
objects that *do* exist, the objects being searched for are |
| 113 |
typically related in some way, which means they probably |
| 114 |
all exist in a small number of packfiles, so memory usage |
| 115 |
will be constrained to just those packfile indexes. |
| 116 |
|
| 117 |
Since git users typically don't add a lot of files in a |
| 118 |
single run, git doesn't really need a program like `bup |
| 119 |
midx`. bup, on the other hand, spends most of its time |
| 120 |
backing up files it hasn't seen before, so its memory usage |
| 121 |
patterns are different. |
| 122 |
|
| 123 |
|
| 124 |
# SEE ALSO |
| 125 |
|
| 126 |
`bup-midx`(1) |
| 127 |
|
| 128 |
# BUP |
| 129 |
|
| 130 |
Part of the `bup`(1) suite. |