Debian Codegraph @

manuals/en/main/pools.tex

raw
1 \chapter{Automated Disk Backup}
2 \label{blb:PoolsChapter}
3 \index[general]{Volumes!Using Pools to Manage}
4 \index[general]{Disk!Automated Backup}
5 \index[general]{Using Pools to Manage Volumes}
6 \index[general]{Automated Disk Backup}
7
8 If you manage five or ten machines and have a nice tape backup, you don't need
9 Pools, and you may wonder what they are good for. In this chapter, you will
10 see that Pools can help you optimize disk storage space. The same techniques
11 can be applied to a shop that has multiple tape drives, or that wants to mount
12 various different Volumes to meet their needs.
13
14 The rest of this chapter will give an example involving backup to disk
15 Volumes, but most of the information applies equally well to tape Volumes.
16
17 \label{blb:TheProblem}
18 \section{The Problem}
19 \index[general]{Problem}
20
21 A site that I administer (a charitable organization) had a tape DDS-3 tape
22 drive that was failing. The exact reason for the failure is still unknown.
23 Worse yet, their full backup size is about 15GB whereas the capacity of their
24 broken DDS-3 was at best 8GB (rated 6/12). A new DDS-4 tape drive and the
25 necessary cassettes was more expensive than their budget could handle.
26
27 \label{blb:TheSolution}
28 \section{The Solution}
29 \index[general]{Solution}
30
31 They want to maintain six months of backup data, and be able to access the old
32 files on a daily basis for a week, a weekly basis for a month, then monthly
33 for six months. In addition, offsite capability was not needed (well perhaps
34 it really is, but it was never used). Their daily changes amount to about
35 300MB on the average, or about 2GB per week.
36
37 As a consequence, the total volume of data they need to keep to meet their
38 needs is about 100GB (15GB x 6 + 2GB x 5 + 0.3 x 7) = 102.1GB.
39
40 The chosen solution was to buy a 120GB hard disk for next to nothing -- far
41 less than 1/10th the price of a tape drive and the cassettes to handle the
42 same amount of data, and to have \bacula{} write to disk files.
43
44 The rest of this chapter will explain how to setup \bacula{} so that it would
45 automatically manage a set of disk files with the minimum sysadmin
46 intervention. The system has been running since 22 January 2004 until today
47 (23 June 2007) with no intervention, with the exception of adding
48 a second 120GB hard disk after a year because their needs grew
49 over that time to more than the 120GB (168GB to be exact). The only other
50 intervention I have made is a periodic (about once a year) \bacula{} upgrade.
51
52 \label{blb:OverallDesign}
53 \section{Overall Design}
54 \index[general]{Overall Design}
55 \index[general]{Design!Overall}
56
57 Getting \bacula{} to write to disk rather than tape in the simplest case is
58 rather easy, and is documented in the previous chapter. In addition, all the
59 directives discussed here are explained in that chapter. We'll leave it to you
60 to look at the details there. If you haven't read it and are not familiar with
61 Pools, you probably should at least read it once quickly for the ideas before
62 continuing here.
63
64 One needs to consider about what happens if we have only a single large \bacula{}
65 Volume defined on our hard disk. Everything works fine until the Volume fills,
66 then \bacula{} will ask you to mount a new Volume. This same problem applies to
67 the use of tape Volumes if your tape fills. Being a hard disk and the only one
68 you have, this will be a bit of a problem. It should be obvious that it is
69 better to use a number of smaller Volumes and arrange for \bacula{} to
70 automatically recycle them so that the disk storage space can be reused. The
71 other problem with a single Volume, is that until version 2.0.0,
72 \bacula{} did not seek within a disk Volume, so restoring a single file can take
73 more time than one would expect.
74
75 As mentioned, the solution is to have multiple Volumes, or files on the disk.
76 To do so, we need to limit the use and thus the size of a single Volume, by
77 time, by number of jobs, or by size. Any of these would work, but we chose to
78 limit the use of a single Volume by putting a single job in each Volume with
79 the exception of Volumes containing Incremental backup where there will be 6
80 jobs (a week's worth of data) per volume. The details of this will be
81 discussed shortly. This is a single client backup, so if you have multiple
82 clients you will need to multiply those numbers by the number of clients,
83 or use a different system for switching volumes, such as limiting the
84 volume size.
85
86 The next problem to resolve is recycling of Volumes. As you noted from above,
87 the requirements are to be able to restore monthly for 6 months, weekly for a
88 month, and daily for a week. So to simplify things, why not do a Full save
89 once a month, a Differential save once a week, and Incremental saves daily.
90 Now since each of these different kinds of saves needs to remain valid for
91 differing periods, the simplest way to do this (and possibly the only) is to
92 have a separate Pool for each backup type.
93
94 The decision was to use three Pools: one for Full saves, one for Differential
95 saves, and one for Incremental saves, and each would have a different number
96 of volumes and a different Retention period to accomplish the requirements.
97
98 \label{blb:FullPool}
99 \subsection{Full Pool}
100 \index[general]{Pool!Full}
101 \index[general]{Full Pool}
102
103 Putting a single Full backup on each Volume, will require six Full save
104 Volumes, and a retention period of six months. The Pool needed to do that is:
105
106 \begin{bVerbatim}
107 Pool {
108 Name = Full-Pool
109 Pool Type = Backup
110 Recycle = yes
111 AutoPrune = yes
112 Volume Retention = 6 months
113 Maximum Volume Jobs = 1
114 Label Format = Full-
115 Maximum Volumes = 9
116 }
117 \end{bVerbatim}
118
119 Since these are disk Volumes, no space is lost by having separate Volumes for
120 each backup (done once a month in this case). The items to note are the
121 retention period of six months (i.e. they are recycled after six months), that
122 there is one job per volume (Maximum Volume Jobs = 1), the volumes will be
123 labeled Full-0001, \ldots{} Full-0006 automatically. One could have labeled these
124 manually from the start, but why not use the features of \bacula{}.
125
126 Six months after the first volume is used, it will be subject to pruning
127 and thus recycling, so with a maximum of 9 volumes, there should always be
128 3 volumes available (note, they may all be marked used, but they will be
129 marked purged and recycled as needed).
130
131 If you have two clients, you would want to set \textbf{Maximum Volume Jobs} to
132 2 instead of one, or set a limit on the size of the Volumes, and possibly
133 increase the maximum number of Volumes.
134
135
136 \label{blb:DiffPool}
137 \subsection{Differential Pool}
138 \index[general]{Pool!Differential}
139 \index[general]{Differential Pool}
140
141 For the Differential backup Pool, we choose a retention period of a bit longer
142 than a month and ensure that there is at least one Volume for each of the
143 maximum of five weeks in a month. So the following works:
144
145 \begin{bVerbatim}
146 Pool {
147 Name = Diff-Pool
148 Pool Type = Backup
149 Recycle = yes
150 AutoPrune = yes
151 Volume Retention = 40 days
152 Maximum Volume Jobs = 1
153 Label Format = Diff-
154 Maximum Volumes = 10
155 }
156 \end{bVerbatim}
157
158 As you can see, the Differential Pool can grow to a maximum of 9 volumes,
159 and the Volumes are retained 40 days and thereafter they can be recycled. Finally
160 there is one job per volume. This, of course, could be tightened up a lot, but
161 the expense here is a few GB which is not too serious.
162
163 If a new volume is used every week, after 40 days, one will have used 7
164 volumes, and there should then always be 3 volumes that can be purged and
165 recycled.
166
167 See the discussion above concering the Full pool for how to handle multiple
168 clients.
169
170 \label{blb:IncPool}
171 \subsection{Incremental Pool}
172 \index[general]{Incremental Pool}
173 \index[general]{Pool!Incremental}
174
175 Finally, here is the resource for the Incremental Pool:
176
177 \begin{bVerbatim}
178 Pool {
179 Name = Inc-Pool
180 Pool Type = Backup
181 Recycle = yes
182 AutoPrune = yes
183 Volume Retention = 20 days
184 Maximum Volume Jobs = 6
185 Label Format = Inc-
186 Maximum Volumes = 7
187 }
188 \end{bVerbatim}
189
190 We keep the data for 20 days rather than just a week as the needs require. To
191 reduce the proliferation of volume names, we keep a week's worth of data (6
192 incremental backups) in each Volume. In practice, the retention period should
193 be set to just a bit more than a week and keep only two or three volumes
194 instead of five. Again, the lost is very little and as the system reaches the
195 full steady state, we can adjust these values so that the total disk usage
196 doesn't exceed the disk capacity.
197
198 If you have two clients, the simplest thing to do is to increase the
199 maximum volume jobs from 6 to 12. As mentioned above, it is also possible
200 limit the size of the volumes. However, in that case, you will need to
201 have a better idea of the volume or add sufficient volumes to the pool so
202 that you will be assured that in the next cycle (after 20 days) there is
203 at least one volume that is pruned and can be recycled.
204
205
206 \label{blb:Example}
207 \section{The Actual Conf Files}
208 \index[general]{Files!Actual Conf}
209 \index[general]{Actual Conf Files}
210
211 The following example shows you the actual files used, with only a few minor
212 modifications to simplify things.
213
214 The Director's configuration file is as follows:
215
216 \begin{bVerbatim}
217 Director { # define myself
218 Name = bacula-dir
219 DIRport = 9101
220 QueryFile = "/opt/bacula/bin/query.sql"
221 WorkingDirectory = "/opt/bacula/working"
222 PidDirectory = "/opt/bacula/working"
223 Maximum Concurrent Jobs = 1
224 Password = " *** CHANGE ME ***"
225 Messages = Standard
226 }
227 # By default, this job will back up to disk in /tmp
228 Job {
229 Name = client
230 Type = Backup
231 Client = client-fd
232 FileSet = "Full Set"
233 Schedule = "WeeklyCycle"
234 Storage = File
235 Messages = Standard
236 Pool = Default
237 Full Backup Pool = Full-Pool
238 Incremental Backup Pool = Inc-Pool
239 Differential Backup Pool = Diff-Pool
240 Write Bootstrap = "/opt/bacula/working/client.bsr"
241 Priority = 10
242 }
243
244 # Backup the catalog database (after the nightly save)
245 Job {
246 Name = "BackupCatalog"
247 Type = Backup
248 Client = client-fd
249 FileSet="Catalog"
250 Schedule = "WeeklyCycleAfterBackup"
251 Storage = File
252 Messages = Standard
253 Pool = Default
254 # This creates an ASCII copy of the catalog
255 # WARNING!!! Passing the password via the command line is insecure.
256 # see comments in make_catalog_backup.pl for details.
257 RunBeforeJob = "/opt/bacula/bin/make_catalog_backup.pl MyCatalog"
258 # This deletes the copy of the catalog
259 RunAfterJob = "/opt/bacula/bin/delete_catalog_backup"
260 Write Bootstrap = "/opt/bacula/working/BackupCatalog.bsr"
261 Priority = 11 # run after main backup
262 }
263
264 # Standard Restore template, to be changed by Console program
265 Job {
266 Name = "RestoreFiles"
267 Type = Restore
268 Client = havana-fd
269 FileSet="Full Set"
270 Storage = File
271 Messages = Standard
272 Pool = Default
273 Where = /tmp/bacula-restores
274 }
275
276
277
278 # List of files to be backed up
279 FileSet {
280 Name = "Full Set"
281 Include = { Options { signature=SHA1; compression=GZIP9 }
282 File = /
283 File = /usr
284 File = /home
285 File = /boot
286 File = /var
287 File = /opt
288 }
289 Exclude = {
290 File = /proc
291 File = /tmp
292 File = /.journal
293 File = /.fsck
294 ...
295 }
296 }
297 Schedule {
298 Name = "WeeklyCycle"
299 Run = Level=Full 1st sun at 2:05
300 Run = Level=Differential 2nd-5th sun at 2:05
301 Run = Level=Incremental mon-sat at 2:05
302 }
303
304 # This schedule does the catalog. It starts after the WeeklyCycle
305 Schedule {
306 Name = "WeeklyCycleAfterBackup"
307 Run = Level=Full sun-sat at 2:10
308 }
309
310 # This is the backup of the catalog
311 FileSet {
312 Name = "Catalog"
313 Include { Options { signature=MD5 }
314 File = /opt/bacula/working/bacula.sql
315 }
316 }
317
318 Client {
319 Name = client-fd
320 Address = client
321 FDPort = 9102
322 Catalog = MyCatalog
323 Password = " *** CHANGE ME ***"
324 AutoPrune = yes # Prune expired Jobs/Files
325 Job Retention = 6 months
326 File Retention = 60 days
327 }
328
329 Storage {
330 Name = File
331 Address = localhost
332 SDPort = 9103
333 Password = " *** CHANGE ME ***"
334 Device = FileStorage
335 Media Type = File
336 }
337
338 Catalog {
339 Name = MyCatalog
340 dbname = bacula; user = bacula; password = ""
341 }
342
343 Pool {
344 Name = Full-Pool
345 Pool Type = Backup
346 Recycle = yes # automatically recycle Volumes
347 AutoPrune = yes # Prune expired volumes
348 Volume Retention = 6 months
349 Maximum Volume Jobs = 1
350 Label Format = Full-
351 Maximum Volumes = 9
352 }
353
354 Pool {
355 Name = Inc-Pool
356 Pool Type = Backup
357 Recycle = yes # automatically recycle Volumes
358 AutoPrune = yes # Prune expired volumes
359 Volume Retention = 20 days
360 Maximum Volume Jobs = 6
361 Label Format = Inc-
362 Maximum Volumes = 7
363 }
364
365 Pool {
366 Name = Diff-Pool
367 Pool Type = Backup
368 Recycle = yes
369 AutoPrune = yes
370 Volume Retention = 40 days
371 Maximum Volume Jobs = 1
372 Label Format = Diff-
373 Maximum Volumes = 10
374 }
375
376 Messages {
377 Name = Standard
378 mailcommand = "bsmtp -h mail.domain.com -f \"\(Bacula\) %r\"
379 -s \"Bacula: %t %e of %c %l\" %r"
380 operatorcommand = "bsmtp -h mail.domain.com -f \"\(Bacula\) %r\"
381 -s \"Bacula: Intervention needed for %j\" %r"
382 mail = [email protected] = all, !skipped
383 operator = [email protected] = mount
384 console = all, !skipped, !saved
385 append = "/opt/bacula/bin/log" = all, !skipped
386 }
387 \end{bVerbatim}
388
389 and the Storage daemon's configuration file is:
390
391 \begin{bVerbatim}
392 Storage { # definition of myself
393 Name = bacula-sd
394 SDPort = 9103 # Director's port
395 WorkingDirectory = "/opt/bacula/working"
396 Pid Directory = "/opt/bacula/working"
397 }
398 Director {
399 Name = bacula-dir
400 Password = " *** CHANGE ME ***"
401 }
402 Device {
403 Name = FileStorage
404 Media Type = File
405 Archive Device = /files/bacula
406 LabelMedia = yes; # lets Bacula label unlabeled media
407 Random Access = Yes;
408 AutomaticMount = yes; # when device opened, read it
409 RemovableMedia = no;
410 AlwaysOpen = no;
411 }
412 Messages {
413 Name = Standard
414 director = bacula-dir = all
415 }
416 \end{bVerbatim}