| 1 |
\chapter{Automated Disk Backup} |
| 2 |
\label{blb:PoolsChapter} |
| 3 |
\index[general]{Volumes!Using Pools to Manage} |
| 4 |
\index[general]{Disk!Automated Backup} |
| 5 |
\index[general]{Using Pools to Manage Volumes} |
| 6 |
\index[general]{Automated Disk Backup} |
| 7 |
|
| 8 |
If you manage five or ten machines and have a nice tape backup, you don't need |
| 9 |
Pools, and you may wonder what they are good for. In this chapter, you will |
| 10 |
see that Pools can help you optimize disk storage space. The same techniques |
| 11 |
can be applied to a shop that has multiple tape drives, or that wants to mount |
| 12 |
various different Volumes to meet their needs. |
| 13 |
|
| 14 |
The rest of this chapter will give an example involving backup to disk |
| 15 |
Volumes, but most of the information applies equally well to tape Volumes. |
| 16 |
|
| 17 |
\label{blb:TheProblem} |
| 18 |
\section{The Problem} |
| 19 |
\index[general]{Problem} |
| 20 |
|
| 21 |
A site that I administer (a charitable organization) had a tape DDS-3 tape |
| 22 |
drive that was failing. The exact reason for the failure is still unknown. |
| 23 |
Worse yet, their full backup size is about 15GB whereas the capacity of their |
| 24 |
broken DDS-3 was at best 8GB (rated 6/12). A new DDS-4 tape drive and the |
| 25 |
necessary cassettes was more expensive than their budget could handle. |
| 26 |
|
| 27 |
\label{blb:TheSolution} |
| 28 |
\section{The Solution} |
| 29 |
\index[general]{Solution} |
| 30 |
|
| 31 |
They want to maintain six months of backup data, and be able to access the old |
| 32 |
files on a daily basis for a week, a weekly basis for a month, then monthly |
| 33 |
for six months. In addition, offsite capability was not needed (well perhaps |
| 34 |
it really is, but it was never used). Their daily changes amount to about |
| 35 |
300MB on the average, or about 2GB per week. |
| 36 |
|
| 37 |
As a consequence, the total volume of data they need to keep to meet their |
| 38 |
needs is about 100GB (15GB x 6 + 2GB x 5 + 0.3 x 7) = 102.1GB. |
| 39 |
|
| 40 |
The chosen solution was to buy a 120GB hard disk for next to nothing -- far |
| 41 |
less than 1/10th the price of a tape drive and the cassettes to handle the |
| 42 |
same amount of data, and to have \bacula{} write to disk files. |
| 43 |
|
| 44 |
The rest of this chapter will explain how to setup \bacula{} so that it would |
| 45 |
automatically manage a set of disk files with the minimum sysadmin |
| 46 |
intervention. The system has been running since 22 January 2004 until today |
| 47 |
(23 June 2007) with no intervention, with the exception of adding |
| 48 |
a second 120GB hard disk after a year because their needs grew |
| 49 |
over that time to more than the 120GB (168GB to be exact). The only other |
| 50 |
intervention I have made is a periodic (about once a year) \bacula{} upgrade. |
| 51 |
|
| 52 |
\label{blb:OverallDesign} |
| 53 |
\section{Overall Design} |
| 54 |
\index[general]{Overall Design} |
| 55 |
\index[general]{Design!Overall} |
| 56 |
|
| 57 |
Getting \bacula{} to write to disk rather than tape in the simplest case is |
| 58 |
rather easy, and is documented in the previous chapter. In addition, all the |
| 59 |
directives discussed here are explained in that chapter. We'll leave it to you |
| 60 |
to look at the details there. If you haven't read it and are not familiar with |
| 61 |
Pools, you probably should at least read it once quickly for the ideas before |
| 62 |
continuing here. |
| 63 |
|
| 64 |
One needs to consider about what happens if we have only a single large \bacula{} |
| 65 |
Volume defined on our hard disk. Everything works fine until the Volume fills, |
| 66 |
then \bacula{} will ask you to mount a new Volume. This same problem applies to |
| 67 |
the use of tape Volumes if your tape fills. Being a hard disk and the only one |
| 68 |
you have, this will be a bit of a problem. It should be obvious that it is |
| 69 |
better to use a number of smaller Volumes and arrange for \bacula{} to |
| 70 |
automatically recycle them so that the disk storage space can be reused. The |
| 71 |
other problem with a single Volume, is that until version 2.0.0, |
| 72 |
\bacula{} did not seek within a disk Volume, so restoring a single file can take |
| 73 |
more time than one would expect. |
| 74 |
|
| 75 |
As mentioned, the solution is to have multiple Volumes, or files on the disk. |
| 76 |
To do so, we need to limit the use and thus the size of a single Volume, by |
| 77 |
time, by number of jobs, or by size. Any of these would work, but we chose to |
| 78 |
limit the use of a single Volume by putting a single job in each Volume with |
| 79 |
the exception of Volumes containing Incremental backup where there will be 6 |
| 80 |
jobs (a week's worth of data) per volume. The details of this will be |
| 81 |
discussed shortly. This is a single client backup, so if you have multiple |
| 82 |
clients you will need to multiply those numbers by the number of clients, |
| 83 |
or use a different system for switching volumes, such as limiting the |
| 84 |
volume size. |
| 85 |
|
| 86 |
The next problem to resolve is recycling of Volumes. As you noted from above, |
| 87 |
the requirements are to be able to restore monthly for 6 months, weekly for a |
| 88 |
month, and daily for a week. So to simplify things, why not do a Full save |
| 89 |
once a month, a Differential save once a week, and Incremental saves daily. |
| 90 |
Now since each of these different kinds of saves needs to remain valid for |
| 91 |
differing periods, the simplest way to do this (and possibly the only) is to |
| 92 |
have a separate Pool for each backup type. |
| 93 |
|
| 94 |
The decision was to use three Pools: one for Full saves, one for Differential |
| 95 |
saves, and one for Incremental saves, and each would have a different number |
| 96 |
of volumes and a different Retention period to accomplish the requirements. |
| 97 |
|
| 98 |
\label{blb:FullPool} |
| 99 |
\subsection{Full Pool} |
| 100 |
\index[general]{Pool!Full} |
| 101 |
\index[general]{Full Pool} |
| 102 |
|
| 103 |
Putting a single Full backup on each Volume, will require six Full save |
| 104 |
Volumes, and a retention period of six months. The Pool needed to do that is: |
| 105 |
|
| 106 |
\begin{bVerbatim} |
| 107 |
Pool { |
| 108 |
Name = Full-Pool |
| 109 |
Pool Type = Backup |
| 110 |
Recycle = yes |
| 111 |
AutoPrune = yes |
| 112 |
Volume Retention = 6 months |
| 113 |
Maximum Volume Jobs = 1 |
| 114 |
Label Format = Full- |
| 115 |
Maximum Volumes = 9 |
| 116 |
} |
| 117 |
\end{bVerbatim} |
| 118 |
|
| 119 |
Since these are disk Volumes, no space is lost by having separate Volumes for |
| 120 |
each backup (done once a month in this case). The items to note are the |
| 121 |
retention period of six months (i.e. they are recycled after six months), that |
| 122 |
there is one job per volume (Maximum Volume Jobs = 1), the volumes will be |
| 123 |
labeled Full-0001, \ldots{} Full-0006 automatically. One could have labeled these |
| 124 |
manually from the start, but why not use the features of \bacula{}. |
| 125 |
|
| 126 |
Six months after the first volume is used, it will be subject to pruning |
| 127 |
and thus recycling, so with a maximum of 9 volumes, there should always be |
| 128 |
3 volumes available (note, they may all be marked used, but they will be |
| 129 |
marked purged and recycled as needed). |
| 130 |
|
| 131 |
If you have two clients, you would want to set \textbf{Maximum Volume Jobs} to |
| 132 |
2 instead of one, or set a limit on the size of the Volumes, and possibly |
| 133 |
increase the maximum number of Volumes. |
| 134 |
|
| 135 |
|
| 136 |
\label{blb:DiffPool} |
| 137 |
\subsection{Differential Pool} |
| 138 |
\index[general]{Pool!Differential} |
| 139 |
\index[general]{Differential Pool} |
| 140 |
|
| 141 |
For the Differential backup Pool, we choose a retention period of a bit longer |
| 142 |
than a month and ensure that there is at least one Volume for each of the |
| 143 |
maximum of five weeks in a month. So the following works: |
| 144 |
|
| 145 |
\begin{bVerbatim} |
| 146 |
Pool { |
| 147 |
Name = Diff-Pool |
| 148 |
Pool Type = Backup |
| 149 |
Recycle = yes |
| 150 |
AutoPrune = yes |
| 151 |
Volume Retention = 40 days |
| 152 |
Maximum Volume Jobs = 1 |
| 153 |
Label Format = Diff- |
| 154 |
Maximum Volumes = 10 |
| 155 |
} |
| 156 |
\end{bVerbatim} |
| 157 |
|
| 158 |
As you can see, the Differential Pool can grow to a maximum of 9 volumes, |
| 159 |
and the Volumes are retained 40 days and thereafter they can be recycled. Finally |
| 160 |
there is one job per volume. This, of course, could be tightened up a lot, but |
| 161 |
the expense here is a few GB which is not too serious. |
| 162 |
|
| 163 |
If a new volume is used every week, after 40 days, one will have used 7 |
| 164 |
volumes, and there should then always be 3 volumes that can be purged and |
| 165 |
recycled. |
| 166 |
|
| 167 |
See the discussion above concering the Full pool for how to handle multiple |
| 168 |
clients. |
| 169 |
|
| 170 |
\label{blb:IncPool} |
| 171 |
\subsection{Incremental Pool} |
| 172 |
\index[general]{Incremental Pool} |
| 173 |
\index[general]{Pool!Incremental} |
| 174 |
|
| 175 |
Finally, here is the resource for the Incremental Pool: |
| 176 |
|
| 177 |
\begin{bVerbatim} |
| 178 |
Pool { |
| 179 |
Name = Inc-Pool |
| 180 |
Pool Type = Backup |
| 181 |
Recycle = yes |
| 182 |
AutoPrune = yes |
| 183 |
Volume Retention = 20 days |
| 184 |
Maximum Volume Jobs = 6 |
| 185 |
Label Format = Inc- |
| 186 |
Maximum Volumes = 7 |
| 187 |
} |
| 188 |
\end{bVerbatim} |
| 189 |
|
| 190 |
We keep the data for 20 days rather than just a week as the needs require. To |
| 191 |
reduce the proliferation of volume names, we keep a week's worth of data (6 |
| 192 |
incremental backups) in each Volume. In practice, the retention period should |
| 193 |
be set to just a bit more than a week and keep only two or three volumes |
| 194 |
instead of five. Again, the lost is very little and as the system reaches the |
| 195 |
full steady state, we can adjust these values so that the total disk usage |
| 196 |
doesn't exceed the disk capacity. |
| 197 |
|
| 198 |
If you have two clients, the simplest thing to do is to increase the |
| 199 |
maximum volume jobs from 6 to 12. As mentioned above, it is also possible |
| 200 |
limit the size of the volumes. However, in that case, you will need to |
| 201 |
have a better idea of the volume or add sufficient volumes to the pool so |
| 202 |
that you will be assured that in the next cycle (after 20 days) there is |
| 203 |
at least one volume that is pruned and can be recycled. |
| 204 |
|
| 205 |
|
| 206 |
\label{blb:Example} |
| 207 |
\section{The Actual Conf Files} |
| 208 |
\index[general]{Files!Actual Conf} |
| 209 |
\index[general]{Actual Conf Files} |
| 210 |
|
| 211 |
The following example shows you the actual files used, with only a few minor |
| 212 |
modifications to simplify things. |
| 213 |
|
| 214 |
The Director's configuration file is as follows: |
| 215 |
|
| 216 |
\begin{bVerbatim} |
| 217 |
Director { # define myself |
| 218 |
Name = bacula-dir |
| 219 |
DIRport = 9101 |
| 220 |
QueryFile = "/opt/bacula/bin/query.sql" |
| 221 |
WorkingDirectory = "/opt/bacula/working" |
| 222 |
PidDirectory = "/opt/bacula/working" |
| 223 |
Maximum Concurrent Jobs = 1 |
| 224 |
Password = " *** CHANGE ME ***" |
| 225 |
Messages = Standard |
| 226 |
} |
| 227 |
# By default, this job will back up to disk in /tmp |
| 228 |
Job { |
| 229 |
Name = client |
| 230 |
Type = Backup |
| 231 |
Client = client-fd |
| 232 |
FileSet = "Full Set" |
| 233 |
Schedule = "WeeklyCycle" |
| 234 |
Storage = File |
| 235 |
Messages = Standard |
| 236 |
Pool = Default |
| 237 |
Full Backup Pool = Full-Pool |
| 238 |
Incremental Backup Pool = Inc-Pool |
| 239 |
Differential Backup Pool = Diff-Pool |
| 240 |
Write Bootstrap = "/opt/bacula/working/client.bsr" |
| 241 |
Priority = 10 |
| 242 |
} |
| 243 |
|
| 244 |
# Backup the catalog database (after the nightly save) |
| 245 |
Job { |
| 246 |
Name = "BackupCatalog" |
| 247 |
Type = Backup |
| 248 |
Client = client-fd |
| 249 |
FileSet="Catalog" |
| 250 |
Schedule = "WeeklyCycleAfterBackup" |
| 251 |
Storage = File |
| 252 |
Messages = Standard |
| 253 |
Pool = Default |
| 254 |
# This creates an ASCII copy of the catalog |
| 255 |
# WARNING!!! Passing the password via the command line is insecure. |
| 256 |
# see comments in make_catalog_backup.pl for details. |
| 257 |
RunBeforeJob = "/opt/bacula/bin/make_catalog_backup.pl MyCatalog" |
| 258 |
# This deletes the copy of the catalog |
| 259 |
RunAfterJob = "/opt/bacula/bin/delete_catalog_backup" |
| 260 |
Write Bootstrap = "/opt/bacula/working/BackupCatalog.bsr" |
| 261 |
Priority = 11 # run after main backup |
| 262 |
} |
| 263 |
|
| 264 |
# Standard Restore template, to be changed by Console program |
| 265 |
Job { |
| 266 |
Name = "RestoreFiles" |
| 267 |
Type = Restore |
| 268 |
Client = havana-fd |
| 269 |
FileSet="Full Set" |
| 270 |
Storage = File |
| 271 |
Messages = Standard |
| 272 |
Pool = Default |
| 273 |
Where = /tmp/bacula-restores |
| 274 |
} |
| 275 |
|
| 276 |
|
| 277 |
|
| 278 |
# List of files to be backed up |
| 279 |
FileSet { |
| 280 |
Name = "Full Set" |
| 281 |
Include = { Options { signature=SHA1; compression=GZIP9 } |
| 282 |
File = / |
| 283 |
File = /usr |
| 284 |
File = /home |
| 285 |
File = /boot |
| 286 |
File = /var |
| 287 |
File = /opt |
| 288 |
} |
| 289 |
Exclude = { |
| 290 |
File = /proc |
| 291 |
File = /tmp |
| 292 |
File = /.journal |
| 293 |
File = /.fsck |
| 294 |
... |
| 295 |
} |
| 296 |
} |
| 297 |
Schedule { |
| 298 |
Name = "WeeklyCycle" |
| 299 |
Run = Level=Full 1st sun at 2:05 |
| 300 |
Run = Level=Differential 2nd-5th sun at 2:05 |
| 301 |
Run = Level=Incremental mon-sat at 2:05 |
| 302 |
} |
| 303 |
|
| 304 |
# This schedule does the catalog. It starts after the WeeklyCycle |
| 305 |
Schedule { |
| 306 |
Name = "WeeklyCycleAfterBackup" |
| 307 |
Run = Level=Full sun-sat at 2:10 |
| 308 |
} |
| 309 |
|
| 310 |
# This is the backup of the catalog |
| 311 |
FileSet { |
| 312 |
Name = "Catalog" |
| 313 |
Include { Options { signature=MD5 } |
| 314 |
File = /opt/bacula/working/bacula.sql |
| 315 |
} |
| 316 |
} |
| 317 |
|
| 318 |
Client { |
| 319 |
Name = client-fd |
| 320 |
Address = client |
| 321 |
FDPort = 9102 |
| 322 |
Catalog = MyCatalog |
| 323 |
Password = " *** CHANGE ME ***" |
| 324 |
AutoPrune = yes # Prune expired Jobs/Files |
| 325 |
Job Retention = 6 months |
| 326 |
File Retention = 60 days |
| 327 |
} |
| 328 |
|
| 329 |
Storage { |
| 330 |
Name = File |
| 331 |
Address = localhost |
| 332 |
SDPort = 9103 |
| 333 |
Password = " *** CHANGE ME ***" |
| 334 |
Device = FileStorage |
| 335 |
Media Type = File |
| 336 |
} |
| 337 |
|
| 338 |
Catalog { |
| 339 |
Name = MyCatalog |
| 340 |
dbname = bacula; user = bacula; password = "" |
| 341 |
} |
| 342 |
|
| 343 |
Pool { |
| 344 |
Name = Full-Pool |
| 345 |
Pool Type = Backup |
| 346 |
Recycle = yes # automatically recycle Volumes |
| 347 |
AutoPrune = yes # Prune expired volumes |
| 348 |
Volume Retention = 6 months |
| 349 |
Maximum Volume Jobs = 1 |
| 350 |
Label Format = Full- |
| 351 |
Maximum Volumes = 9 |
| 352 |
} |
| 353 |
|
| 354 |
Pool { |
| 355 |
Name = Inc-Pool |
| 356 |
Pool Type = Backup |
| 357 |
Recycle = yes # automatically recycle Volumes |
| 358 |
AutoPrune = yes # Prune expired volumes |
| 359 |
Volume Retention = 20 days |
| 360 |
Maximum Volume Jobs = 6 |
| 361 |
Label Format = Inc- |
| 362 |
Maximum Volumes = 7 |
| 363 |
} |
| 364 |
|
| 365 |
Pool { |
| 366 |
Name = Diff-Pool |
| 367 |
Pool Type = Backup |
| 368 |
Recycle = yes |
| 369 |
AutoPrune = yes |
| 370 |
Volume Retention = 40 days |
| 371 |
Maximum Volume Jobs = 1 |
| 372 |
Label Format = Diff- |
| 373 |
Maximum Volumes = 10 |
| 374 |
} |
| 375 |
|
| 376 |
Messages { |
| 377 |
Name = Standard |
| 378 |
mailcommand = "bsmtp -h mail.domain.com -f \"\(Bacula\) %r\" |
| 379 |
-s \"Bacula: %t %e of %c %l\" %r" |
| 380 |
operatorcommand = "bsmtp -h mail.domain.com -f \"\(Bacula\) %r\" |
| 381 |
-s \"Bacula: Intervention needed for %j\" %r" |
| 382 |
mail = [email protected] = all, !skipped |
| 383 |
operator = [email protected] = mount |
| 384 |
console = all, !skipped, !saved |
| 385 |
append = "/opt/bacula/bin/log" = all, !skipped |
| 386 |
} |
| 387 |
\end{bVerbatim} |
| 388 |
|
| 389 |
and the Storage daemon's configuration file is: |
| 390 |
|
| 391 |
\begin{bVerbatim} |
| 392 |
Storage { # definition of myself |
| 393 |
Name = bacula-sd |
| 394 |
SDPort = 9103 # Director's port |
| 395 |
WorkingDirectory = "/opt/bacula/working" |
| 396 |
Pid Directory = "/opt/bacula/working" |
| 397 |
} |
| 398 |
Director { |
| 399 |
Name = bacula-dir |
| 400 |
Password = " *** CHANGE ME ***" |
| 401 |
} |
| 402 |
Device { |
| 403 |
Name = FileStorage |
| 404 |
Media Type = File |
| 405 |
Archive Device = /files/bacula |
| 406 |
LabelMedia = yes; # lets Bacula label unlabeled media |
| 407 |
Random Access = Yes; |
| 408 |
AutomaticMount = yes; # when device opened, read it |
| 409 |
RemovableMedia = no; |
| 410 |
AlwaysOpen = no; |
| 411 |
} |
| 412 |
Messages { |
| 413 |
Name = Standard |
| 414 |
director = bacula-dir = all |
| 415 |
} |
| 416 |
\end{bVerbatim} |