| 1 |
\chapter{Migration and Copy} |
| 2 |
\label{blb:MigrationChapter} |
| 3 |
\index[general]{Migration} |
| 4 |
\index[general]{Copy} |
| 5 |
|
| 6 |
The term Migration, as used in the context of \bacula{}, means moving data from |
| 7 |
one Volume to another. In particular it refers to a Job (similar to a backup |
| 8 |
job) that reads data that was previously backed up to a Volume and writes |
| 9 |
it to another Volume. As part of this process, the File catalog records |
| 10 |
associated with the first backup job are purged. In other words, Migration |
| 11 |
moves \bacula{} Job data from one Volume to another by reading the Job data |
| 12 |
from the Volume it is stored on, writing it to a different Volume in a |
| 13 |
different Pool, and then purging the database records for the first Job. |
| 14 |
|
| 15 |
The Copy process is essentially identical to the Migration feature with the |
| 16 |
exception that the Job that is copied is left unchanged. This essentially |
| 17 |
creates two identical copies of the same backup. However, the copy is treated |
| 18 |
as a copy rather than a backup job, and hence is not directly available for |
| 19 |
restore. If bacula founds a copy when a job record is purged (deleted) from the |
| 20 |
catalog, it will promote the copy as \textsl{real} backup and will make it |
| 21 |
available for automatic restore. Note: in the text below, to simplify it, |
| 22 |
we usually speak of a migration job. This, in fact, means either a |
| 23 |
migration job or a copy job. |
| 24 |
|
| 25 |
The Copy and the Migration jobs run without using the File daemon by copying |
| 26 |
the data from the old backup Volume to a different Volume in a different |
| 27 |
Pool. It is not possible to run commands on the defined Client via a RunScript |
| 28 |
from within the Migration or Copy Job. |
| 29 |
|
| 30 |
The selection process for which Job or Jobs are migrated |
| 31 |
can be based on quite a number of different criteria such as: |
| 32 |
\begin{bitemize} |
| 33 |
\item a single previous Job |
| 34 |
\item a Volume |
| 35 |
\item a Client |
| 36 |
\item a regular expression matching a Job, Volume, or Client name |
| 37 |
\item the time a Job has been on a Volume |
| 38 |
\item high and low water marks (usage or occupation) of a Pool |
| 39 |
\item Volume size |
| 40 |
\end{bitemize} |
| 41 |
|
| 42 |
The details of these selection criteria will be defined below. |
| 43 |
|
| 44 |
To run a Migration job, you must first define a Job resource very similar |
| 45 |
to a Backup Job but with \textbf{Type = Migrate} instead of |
| 46 |
\textbf{Type = Backup}. One of the key points to remember is that the Pool that is |
| 47 |
specified for the migration job is the only pool from which jobs will |
| 48 |
be migrated, with one exception noted below. In addition, the Pool to |
| 49 |
which the selected Job or Jobs will be migrated is defined by the |
| 50 |
\textbf{Next Pool = \ldots{}} in the Pool resource specified for the Migration Job. |
| 51 |
|
| 52 |
\bacula{} permits Pools to contain Volumes with different Media Types. |
| 53 |
However, when doing migration, this is a very undesirable condition. For |
| 54 |
migration to work properly, you should use Pools containing only Volumes of |
| 55 |
the same Media Type for all migration jobs. |
| 56 |
|
| 57 |
The migration job normally is either manually started or starts |
| 58 |
from a Schedule much like a backup job. It searches |
| 59 |
for a previous backup Job or Jobs that match the parameters you have |
| 60 |
specified in the migration Job resource, primarily a \textbf{Selection Type} |
| 61 |
(detailed a bit later). Then for |
| 62 |
each previous backup JobId found, the Migration Job will run a new Job which |
| 63 |
copies the old Job data from the previous Volume to a new Volume in |
| 64 |
the Migration Pool. It is possible that no prior Jobs are found for |
| 65 |
migration, in which case, the Migration job will simply terminate having |
| 66 |
done nothing, but normally at a minimum, three jobs are involved during a |
| 67 |
migration: |
| 68 |
|
| 69 |
\begin{bitemize} |
| 70 |
\item The currently running Migration control Job. This is only |
| 71 |
a control job for starting the migration child jobs. |
| 72 |
\item The previous Backup Job (already run). The File records |
| 73 |
for this Job are purged if the Migration job successfully |
| 74 |
terminates. The original data remains on the Volume until |
| 75 |
it is recycled and rewritten. |
| 76 |
\item A new Migration Backup Job that moves the data from the |
| 77 |
previous Backup job to the new Volume. If you subsequently |
| 78 |
do a restore, the data will be read from this Job. |
| 79 |
\end{bitemize} |
| 80 |
|
| 81 |
If the Migration control job finds a number of JobIds to migrate (e.g. |
| 82 |
it is asked to migrate one or more Volumes), it will start one new |
| 83 |
migration backup job for each JobId found on the specified Volumes. |
| 84 |
Please note that Migration doesn't scale too well since Migrations are |
| 85 |
done on a Job by Job basis. This if you select a very large volume or |
| 86 |
a number of volumes for migration, you may have a large number of |
| 87 |
Jobs that start. Because each job must read the same Volume, they will |
| 88 |
run consecutively (not simultaneously). |
| 89 |
|
| 90 |
\section{Migration and Copy Job Resource Directives} |
| 91 |
|
| 92 |
The following directives can appear in a Director's Job resource, and they |
| 93 |
are used to define a Migration job. |
| 94 |
|
| 95 |
\begin{bdescription} |
| 96 |
\item [Pool = \bbracket{Pool-name}] The Pool specified in the Migration |
| 97 |
control Job is not a new directive for the Job resource, but it is |
| 98 |
particularly important because it determines what Pool will be examined |
| 99 |
for finding JobIds to migrate. The exception to this is when |
| 100 |
\textbf{Selection Type = SQLQuery}, and although a Pool directive must still be |
| 101 |
specified, no Pool is used, unless you specifically include it in the |
| 102 |
SQL query. Note, in any case, the Pool resource defined by the Pool |
| 103 |
directive must contain a \textbf{Next Pool = \ldots{}} directive to define the |
| 104 |
Pool to which the data will be migrated. |
| 105 |
|
| 106 |
\item [Type = Migrate] |
| 107 |
\textbf{Migrate} is a new type that defines the job that is run as being a |
| 108 |
Migration Job. A Migration Job is a sort of control job and does not have |
| 109 |
any Files associated with it, and in that sense they are more or less like |
| 110 |
an Admin job. Migration jobs simply check to see if there is anything to |
| 111 |
Migrate then possibly start and control new Backup jobs to migrate the data |
| 112 |
from the specified Pool to another Pool. Note, any original JobId that |
| 113 |
is migrated will be marked as having been migrated, and the original |
| 114 |
JobId can nolonger be used for restores; all restores will be done from |
| 115 |
the new migrated Job. |
| 116 |
|
| 117 |
|
| 118 |
\item [Type = Copy] |
| 119 |
\textbf{Copy} is a new type that defines the job that is run as being a |
| 120 |
Copy Job. A Copy Job is a sort of control job and does not have |
| 121 |
any Files associated with it, and in that sense they are more or less like |
| 122 |
an Admin job. Copy jobs simply check to see if there is anything to |
| 123 |
Copy then possibly start and control new Backup jobs to copy the data |
| 124 |
from the specified Pool to another Pool. Note that when a copy is |
| 125 |
made, the original JobIds are left unchanged. The new copies can not |
| 126 |
be used for restoration unless you specifically choose them by JobId. |
| 127 |
Also you subsequently delete a JobId that has a copy, the copy will be |
| 128 |
automatically upgraded to a Backup rather than a Copy, and it will |
| 129 |
subsequently be used for restoration. |
| 130 |
|
| 131 |
\smallskip |
| 132 |
Once again, a Copy Job cannot be used to restore unless you explicitly |
| 133 |
specify the Copy JobId during the restore command. If the original |
| 134 |
backup Job is deleted and there is a Copy of that backup Job, |
| 135 |
the Copy JobIds will be changed to be backup Jobs that can then |
| 136 |
be restored. |
| 137 |
|
| 138 |
|
| 139 |
\bwebdoc{Director:Job:SelectionType} |
| 140 |
\item [Selection Type = \bbracket{Selection-type-keyword}] |
| 141 |
The \bbracket{Selection-type-keyword} determines how the migration job |
| 142 |
will go about selecting what JobIds to migrate. In most cases, it is |
| 143 |
used in conjunction with a \textbf{Selection Pattern} to give you fine |
| 144 |
control over exactly what JobIds are selected. The possible values |
| 145 |
for \bbracket{Selection-type-keyword} are: |
| 146 |
\begin{bdescription} |
| 147 |
\item [SmallestVolume] This selection keyword selects the volume with the |
| 148 |
fewest bytes from the Pool to be migrated. The Pool to be migrated |
| 149 |
is the Pool defined in the Migration Job resource. The migration |
| 150 |
control job will then start and run one migration backup job for |
| 151 |
each of the Jobs found on this Volume. The Selection Pattern, if |
| 152 |
specified, is not used. |
| 153 |
|
| 154 |
\item [OldestVolume] This selection keyword selects the volume with the |
| 155 |
oldest last write time in the Pool to be migrated. The Pool to be |
| 156 |
migrated is the Pool defined in the Migration Job resource. The |
| 157 |
migration control job will then start and run one migration backup |
| 158 |
job for each of the Jobs found on this Volume. The Selection |
| 159 |
Pattern, if specified, is not used. |
| 160 |
|
| 161 |
\item [Client] The Client selection type, first selects all the Clients |
| 162 |
that have been backed up in the Pool specified by the Migration |
| 163 |
Job resource, then it applies the \textbf{Selection Pattern} (defined |
| 164 |
below) as a regular expression to the list of Client names, giving |
| 165 |
a filtered Client name list. All jobs that were backed up for those |
| 166 |
filtered (regexed) Clients will be migrated. |
| 167 |
The migration control job will then start and run one migration |
| 168 |
backup job for each of the JobIds found for those filtered Clients. |
| 169 |
|
| 170 |
\item [Volume] The Volume selection type, first selects all the Volumes |
| 171 |
that have been backed up in the Pool specified by the Migration |
| 172 |
Job resource, then it applies the \textbf{Selection Pattern} (defined |
| 173 |
below) as a regular expression to the list of Volume names, giving |
| 174 |
a filtered Volume list. All JobIds that were backed up for those |
| 175 |
filtered (regexed) Volumes will be migrated. |
| 176 |
The migration control job will then start and run one migration |
| 177 |
backup job for each of the JobIds found on those filtered Volumes. |
| 178 |
|
| 179 |
Jobs on Volumes will be considered for Migration only if the Volume |
| 180 |
is marked, Full, Used, or Error. Volumes that are still marked |
| 181 |
Append will not be considered for migration. This prevents \bacula{} |
| 182 |
from attempting to read the Volume at the same time it is writing it. |
| 183 |
It also reduces other deadlock situations, as well as avoids the |
| 184 |
problem that you migrate a Volume and later find new files appended |
| 185 |
to that Volume. |
| 186 |
|
| 187 |
|
| 188 |
\item [Job] The Job selection type, first selects all the Jobs (as |
| 189 |
defined on the \textbf{Name} directive in a Job resource) |
| 190 |
that have been backed up in the Pool specified by the Migration |
| 191 |
Job resource, then it applies the \textbf{Selection Pattern} (defined |
| 192 |
below) as a regular expression to the list of Job names, giving |
| 193 |
a filtered Job name list. All JobIds that were run for those |
| 194 |
filtered (regexed) Job names will be migrated. Note, for a given |
| 195 |
Job named, they can be many jobs (JobIds) that ran. |
| 196 |
The migration control job will then start and run one migration |
| 197 |
backup job for each of the Jobs found. |
| 198 |
|
| 199 |
\item [SQLQuery] The SQLQuery selection type, used the |
| 200 |
\textbf{Selection Pattern} as an SQL query to obtain the JobIds to be migrated. |
| 201 |
The Selection Pattern must be a valid SELECT SQL statement for your |
| 202 |
SQL engine, and it must return the JobId as the first field |
| 203 |
of the SELECT. |
| 204 |
|
| 205 |
\item [PoolOccupancy] This selection type will cause the Migration job |
| 206 |
to compute the total size of the specified pool for all Media Types |
| 207 |
combined. If it exceeds the \textbf{Migration High Bytes} defined in |
| 208 |
the Pool, the Migration job will migrate all JobIds beginning with |
| 209 |
the oldest Volume in the pool (determined by Last Write time) until |
| 210 |
the Pool bytes drop below the \textbf{Migration Low Bytes} defined in the |
| 211 |
Pool. This calculation should be consider rather approximate because |
| 212 |
it is made once by the Migration job before migration is begun, and |
| 213 |
thus does not take into account additional data written into the Pool |
| 214 |
during the migration. In addition, the calculation of the total Pool |
| 215 |
byte size is based on the Volume bytes saved in the Volume (Media) |
| 216 |
database entries. The bytes calculate for Migration is based on |
| 217 |
the value stored in the Job records of the Jobs to be migrated. |
| 218 |
These do not include the Storage daemon overhead as is in the total |
| 219 |
Pool size. As a consequence, normally, the migration will migrate |
| 220 |
more bytes than strictly necessary. |
| 221 |
|
| 222 |
\item [PoolTime] The PoolTime selection type will cause the Migration job to |
| 223 |
look at the time each JobId has been in the Pool since the job ended. |
| 224 |
All Jobs in the Pool longer than the time specified on \textbf{Migration Time} |
| 225 |
directive in the Pool resource will be migrated. |
| 226 |
|
| 227 |
\item [PoolUncopiedJobs] This selection which copies all jobs from a pool |
| 228 |
to an other pool which were not copied before is available only for copy Jobs. |
| 229 |
|
| 230 |
\end{bdescription} |
| 231 |
|
| 232 |
\bwebdoc{Director:Job:SelectionPattern} |
| 233 |
\item [Selection Pattern = \bbracket{Quoted-string}] |
| 234 |
The Selection Patterns permitted for each Selection-type-keyword are |
| 235 |
described above. |
| 236 |
|
| 237 |
For the OldestVolume and SmallestVolume, this |
| 238 |
Selection pattern is not used (ignored). |
| 239 |
|
| 240 |
\smallskip |
| 241 |
For the Client, Volume, and Job |
| 242 |
keywords, this pattern must be a valid regular expression that will filter |
| 243 |
the appropriate item names found in the Pool. |
| 244 |
|
| 245 |
\smallskip |
| 246 |
For the SQLQuery keyword, this pattern must be a valid SELECT SQL statement |
| 247 |
that returns JobIds. |
| 248 |
|
| 249 |
\bwebdoc{Director:Job:PurgeMigrationJob} |
| 250 |
\item [Purge Migration Job = \byesorno{}] |
| 251 |
This directive may be added to the Migration Job definition in the Director |
| 252 |
configuration file to purge the job migrated at the end of a migration. |
| 253 |
\end{bdescription} |
| 254 |
|
| 255 |
\section{Migration Pool Resource Directives} |
| 256 |
|
| 257 |
The following directives can appear in a Director's Pool resource, and they |
| 258 |
are used to define a Migration job. |
| 259 |
|
| 260 |
\begin{bdescription} |
| 261 |
\bwebdoc{Director:Pool:MigrationTime} |
| 262 |
\item [Migration Time = \bbracket{time-specification}] |
| 263 |
If a PoolTime migration is done, the time specified here in seconds (time |
| 264 |
modifiers are permitted -- e.g. hours, \ldots{}) will be used. If the |
| 265 |
previous Backup Job or Jobs selected have been in the Pool longer than |
| 266 |
the specified PoolTime, then they will be migrated. |
| 267 |
|
| 268 |
\bwebdoc{Director:Pool:MigrationHighBytes} |
| 269 |
\item [Migration High Bytes = \bbracket{byte-specification}] |
| 270 |
This directive specifies the number of bytes in the Pool which will |
| 271 |
trigger a migration if a \textbf{PoolOccupancy} migration selection |
| 272 |
type has been specified. The fact that the Pool |
| 273 |
usage goes above this level does not automatically trigger a migration |
| 274 |
job. However, if a migration job runs and has the PoolOccupancy selection |
| 275 |
type set, the Migration High Bytes will be applied. \bacula{} does not |
| 276 |
currently restrict a pool to have only a single Media Type, so you |
| 277 |
must keep in mind that if you mix Media Types in a Pool, the results |
| 278 |
may not be what you want, as the Pool count of all bytes will be |
| 279 |
for all Media Types combined. |
| 280 |
|
| 281 |
\bwebdoc{Director:Pool:MigrationLowBytes} |
| 282 |
\item [Migration Low Bytes = \bbracket{byte-specification}] |
| 283 |
This directive specifies the number of bytes in the Pool which will |
| 284 |
stop a migration if a \textbf{PoolOccupancy} migration selection |
| 285 |
type has been specified and triggered by more than Migration High |
| 286 |
Bytes being in the pool. In other words, once a migration job |
| 287 |
is started with \textbf{PoolOccupancy} migration selection and it |
| 288 |
determines that there are more than Migration High Bytes, the |
| 289 |
migration job will continue to run jobs until the number of |
| 290 |
bytes in the Pool drop to or below Migration Low Bytes. |
| 291 |
|
| 292 |
\bwebdoc{Director:Pool:NextPool} |
| 293 |
\item [Next Pool = \bbracket{pool-specification}] |
| 294 |
The Next Pool directive specifies the pool to which Jobs will be |
| 295 |
migrated. This directive is required to define the Pool into which |
| 296 |
the data will be migrated. Without this directive, the migration job |
| 297 |
will terminate in error. |
| 298 |
|
| 299 |
The Next Pool directive may also be specified |
| 300 |
in the Job resource or on a Run directive in the Schedule resource. |
| 301 |
Any Next Pool directive in the Job resource will take precedence over |
| 302 |
the Pool definition, and any Next Pool specification on the Run |
| 303 |
directive in a Schedule resource will take ultimate precedence. |
| 304 |
|
| 305 |
\bwebdoc{Director:Pool:End} |
| 306 |
\item [Storage = \bbracket{storage-specification}] |
| 307 |
The Storage directive specifies what Storage resource will be used |
| 308 |
for all Jobs that use this Pool. It takes precedence over any other |
| 309 |
Storage specifications that may have been given such as in the |
| 310 |
Schedule Run directive, or in the Job resource. We highly recommend |
| 311 |
that you define the Storage resource to be used in the Pool rather |
| 312 |
than elsewhere (job, schedule run, \ldots{}). |
| 313 |
\end{bdescription} |
| 314 |
|
| 315 |
\section{Important Migration Considerations} |
| 316 |
\index[general]{Important Migration Considerations} |
| 317 |
\begin{bitemize} |
| 318 |
\item Each Pool into which you migrate Jobs or Volumes \textbf{must} |
| 319 |
contain Volumes of only one Media Type. |
| 320 |
|
| 321 |
\item Migration takes place on a JobId by JobId basis. That is |
| 322 |
each JobId is migrated in its entirety and independently |
| 323 |
of other JobIds. Once the Job is migrated, it will be |
| 324 |
on the new medium in the new Pool, but for the most part, |
| 325 |
aside from having a new JobId, it will appear with all the |
| 326 |
same characteristics of the original job (start, end time, \ldots{}). |
| 327 |
The column RealEndTime in the catalog Job table will contain the |
| 328 |
time and date that the Migration terminated, and by comparing |
| 329 |
it with the EndTime column you can tell whether or not the |
| 330 |
job was migrated. The original job is purged of its File |
| 331 |
records, and its Type field is changed from "B" to "M" to |
| 332 |
indicate that the job was migrated. |
| 333 |
|
| 334 |
\item As noted above, for the Migration High Bytes, the calculation |
| 335 |
of the bytes to migrate is somewhat approximate. |
| 336 |
|
| 337 |
\item If you keep Volumes of different Media Types in the same Pool, |
| 338 |
it is not clear how well migration will work. We recommend only |
| 339 |
one Media Type per pool. |
| 340 |
|
| 341 |
\item It is possible to get into a resource deadlock where \bacula{} does |
| 342 |
not find enough drives to simultaneously read and write all the |
| 343 |
Volumes needed to do Migrations. For the moment, you must take |
| 344 |
care as all the resource deadlock algorithms are not yet implemented. |
| 345 |
You can also use the \bdirectivename{Maximum Concurrent Read Jobs} directive to |
| 346 |
control the drive assignment behavior to reduce resource deadlocks. |
| 347 |
The MaximumConcurrentReadJobs should have a maximum of one less |
| 348 |
than one half the number of drives that are available for reading |
| 349 |
and writing. Doing so, will assure that for each read drive there |
| 350 |
is one that can write so that the Migration jobs will not stall |
| 351 |
due to a deadlock. |
| 352 |
|
| 353 |
\item Migration is done only when you run a Migration job. If you set a |
| 354 |
Migration High Bytes and that number of bytes is exceeded in the Pool |
| 355 |
no migration job will automatically start. You must schedule the |
| 356 |
migration jobs, and they must run for any migration to take place. |
| 357 |
|
| 358 |
\item If you migrate a number of Volumes, a very large number of Migration |
| 359 |
jobs may start. |
| 360 |
|
| 361 |
\item Figuring out what jobs will actually be migrated can be a bit complicated |
| 362 |
due to the flexibility provided by the regex patterns and the number of |
| 363 |
different options. Turning on a debug level of 100 or more will provide |
| 364 |
a limited amount of debug information about the migration selection |
| 365 |
process. |
| 366 |
|
| 367 |
\item \bacula{} currently does only minimal Storage conflict resolution, so you |
| 368 |
must take care to ensure that you don't try to read and write to the |
| 369 |
same device or \bacula{} may block waiting to reserve a drive that it |
| 370 |
will never find. In general, ensure that all your migration |
| 371 |
pools contain only one Media Type, and that you always |
| 372 |
migrate to pools with different Media Types. |
| 373 |
|
| 374 |
\item The \textbf{Next Pool = \ldots{}} directive must be defined in the Pool |
| 375 |
referenced in the Migration Job to define the Pool into which the |
| 376 |
data will be migrated. |
| 377 |
|
| 378 |
\item Pay particular attention to the fact that data is migrated on a Job |
| 379 |
by Job basis, and for any particular tape Volume, only one Job can read |
| 380 |
that Volume at a time (no simultaneous read), so migration jobs that |
| 381 |
all reference the same Volume will run sequentially. This can be a |
| 382 |
potential bottle neck and does not scale very well to large numbers |
| 383 |
of jobs. |
| 384 |
\smallskip |
| 385 |
For disk Volumes, multiple simultaneous Jobs can read the same |
| 386 |
Volume at the same time, so the above restriction does not apply. |
| 387 |
|
| 388 |
|
| 389 |
\item Only migration of Selection Types of Job and Volume have |
| 390 |
been carefully tested. All the other migration methods (time, |
| 391 |
occupancy, smallest, oldest, \ldots{}) need additional testing. |
| 392 |
|
| 393 |
\item Plugins are not compatible with Copy/Migration/VirtualFull jobs by |
| 394 |
default. Please contact \bacula{} Systems support team to know if your strategy |
| 395 |
will be possible with your setup. |
| 396 |
|
| 397 |
\item Windows backups using \acs{VSS} writers (normally plugins) are not |
| 398 |
compatible with Migration and Copy. The reason is that \acs{VSS} writers |
| 399 |
require that certain \acs{XML} files generated at the end of the Backup, |
| 400 |
but they must be presented to the \acs{VSS} writer at the beginning of a |
| 401 |
restore Job. This is very difficult for a backup program such as |
| 402 |
\bacula{}. Just the same \bacula{} accomplishes this by saving the \acs{VSS} |
| 403 |
writer's \acs{XML} files in Restore Objects in the catalog. However, |
| 404 |
during Migration and Copy Jobs these Restore Objects are not currently |
| 405 |
copied to the new Job. |
| 406 |
\end{bitemize} |
| 407 |
|
| 408 |
|
| 409 |
\section{Example Migration Jobs} |
| 410 |
\index[general]{Example Migration Jobs} |
| 411 |
|
| 412 |
When you specify a Migration Job, you must specify all the standard |
| 413 |
directives as for a Job. However, certain such as the Level, Client, and |
| 414 |
FileSet, though they must be defined, are ignored by the Migration job |
| 415 |
because the values from the original job used instead. |
| 416 |
|
| 417 |
\smallskip |
| 418 |
As an example, suppose you have the following Job that |
| 419 |
you run every night. To note: there is no Storage directive in the |
| 420 |
Job resource; there is a Storage directive in each of the Pool |
| 421 |
resources; the Pool to be migrated (File) contains a Next Pool |
| 422 |
directive that defines the output Pool (where the data is written |
| 423 |
by the migration job). |
| 424 |
|
| 425 |
\begin{bVerbatim} |
| 426 |
# Define the backup Job |
| 427 |
Job { |
| 428 |
Name = "NightlySave" |
| 429 |
Type = Backup |
| 430 |
Level = Incremental # default |
| 431 |
Client=rufus-fd |
| 432 |
FileSet="Full Set" |
| 433 |
Schedule = "WeeklyCycle" |
| 434 |
Messages = Standard |
| 435 |
Pool = Default |
| 436 |
} |
| 437 |
|
| 438 |
# Default pool definition |
| 439 |
Pool { |
| 440 |
Name = Default |
| 441 |
Pool Type = Backup |
| 442 |
AutoPrune = yes |
| 443 |
Recycle = yes |
| 444 |
Next Pool = Tape |
| 445 |
Storage = File |
| 446 |
LabelFormat = "File" |
| 447 |
} |
| 448 |
|
| 449 |
# Tape pool definition |
| 450 |
Pool { |
| 451 |
Name = Tape |
| 452 |
Pool Type = Backup |
| 453 |
AutoPrune = yes |
| 454 |
Recycle = yes |
| 455 |
Storage = DLTDrive |
| 456 |
} |
| 457 |
|
| 458 |
# Definition of File storage device |
| 459 |
Storage { |
| 460 |
Name = File |
| 461 |
Address = rufus |
| 462 |
Password = "ccV3lVTsQRsdIUGyab0N4sMDavui2hOBkmpBU0aQKOr9" |
| 463 |
Device = "File" # same as Device in Storage daemon |
| 464 |
Media Type = File # same as MediaType in Storage daemon |
| 465 |
} |
| 466 |
|
| 467 |
# Definition of DLT tape storage device |
| 468 |
Storage { |
| 469 |
Name = DLTDrive |
| 470 |
Address = rufus |
| 471 |
Password = "ccV3lVTsQRsdIUGyab0N4sMDavui2hOBkmpBU0aQKOr9" |
| 472 |
Device = "HP DLT 80" # same as Device in Storage daemon |
| 473 |
Media Type = DLT8000 # same as MediaType in Storage daemon |
| 474 |
} |
| 475 |
|
| 476 |
\end{bVerbatim} |
| 477 |
|
| 478 |
Where we have included only the essential information -- i.e. the |
| 479 |
Director, FileSet, Catalog, Client, Schedule, and Messages resources are |
| 480 |
omitted. |
| 481 |
|
| 482 |
As you can see, by running the NightlySave Job, the data will be backed up |
| 483 |
to File storage using the Default pool to specify the Storage as File. |
| 484 |
|
| 485 |
Now, if we add the following Job resource to this conf file. |
| 486 |
|
| 487 |
\begin{bVerbatim} |
| 488 |
Job { |
| 489 |
Name = "migrate-volume" |
| 490 |
Type = Migrate |
| 491 |
Level = Full |
| 492 |
Client = rufus-fd |
| 493 |
FileSet = "Full Set" |
| 494 |
Messages = Standard |
| 495 |
Pool = Default |
| 496 |
Maximum Concurrent Jobs = 4 |
| 497 |
Selection Type = Volume |
| 498 |
Selection Pattern = "File" |
| 499 |
} |
| 500 |
\end{bVerbatim} |
| 501 |
|
| 502 |
and then run the job named \textbf{migrate-volume}, all volumes in the Pool |
| 503 |
named Default (as specified in the migrate-volume Job that match the |
| 504 |
regular expression pattern \textbf{File} will be migrated to tape storage |
| 505 |
DLTDrive because the \textbf{Next Pool} in the Default Pool specifies that |
| 506 |
Migrations should go to the pool named \textbf{Tape}, which uses |
| 507 |
Storage \textbf{DLTDrive}. |
| 508 |
|
| 509 |
If instead, we use a Job resource as follows: |
| 510 |
|
| 511 |
\begin{bVerbatim} |
| 512 |
Job { |
| 513 |
Name = "migrate" |
| 514 |
Type = Migrate |
| 515 |
Level = Full |
| 516 |
Client = rufus-fd |
| 517 |
FileSet="Full Set" |
| 518 |
Messages = Standard |
| 519 |
Pool = Default |
| 520 |
Maximum Concurrent Jobs = 4 |
| 521 |
Selection Type = Job |
| 522 |
Selection Pattern = ".*Save" |
| 523 |
} |
| 524 |
\end{bVerbatim} |
| 525 |
|
| 526 |
All jobs ending with the name Save will be migrated from the File Default to |
| 527 |
the Tape Pool, or from File storage to Tape storage. |
| 528 |
|
| 529 |
\section{Virtual Backup Consolidation} |
| 530 |
\label{blb:VirtualBackupChapter} |
| 531 |
|
| 532 |
When the Job Level is set to VirtualFull, it permits you to consolidate the |
| 533 |
previous Full backup plus the most recent Differential backup and any |
| 534 |
subsequent Incremental backups into a new Full backup. This new Full backup |
| 535 |
will then be considered as the most recent Full for any future Incremental or |
| 536 |
Differential backups. The VirtualFull backup is accomplished without |
| 537 |
contacting the client by reading the previous backup data and writing it to a |
| 538 |
volume in a different pool. |
| 539 |
|
| 540 |
\smallskip |
| 541 |
\bacula{}'s virtual backup feature is often called Synthetic Backup or |
| 542 |
Consolidation in other backup products. |
| 543 |
|
| 544 |
\smallskip |
| 545 |
In some respects the Virtual Backup feature works similar to a Migration job, |
| 546 |
in that \bacula{} normally reads the data from the pool specified in the Job |
| 547 |
resource, and writes it to the \textbf{Next Pool} specified in the Job |
| 548 |
resource. Note, this means that usually the output from the Virtual Backup is |
| 549 |
written into a different pool from where your prior backups are saved. Doing it |
| 550 |
this way guarantees that you will not get a deadlock situation attempting to |
| 551 |
read and write to the same volume in the Storage daemon. If you then want to do |
| 552 |
subsequent backups, you may need to move the Virtual Full Volume back to your |
| 553 |
normal backup pool. Alternatively, you can set your \textbf{Next Pool} to point |
| 554 |
to the current pool. This will cause \bacula{} to read and write to Volumes in |
| 555 |
the current pool. In general, this will work, because \bacula{} will not allow |
| 556 |
reading and writing on the same Volume. In any case, once a VirtualFull has |
| 557 |
been created, and a restore is done involving the most current Full, it will |
| 558 |
read the Volume or Volumes by the VirtualFull regardless of in which Pool the |
| 559 |
Volume is found. |
| 560 |
|
| 561 |
\smallskip |
| 562 |
A typical Job resource definition might look like the following: |
| 563 |
|
| 564 |
\begin{bVerbatim} |
| 565 |
Job { |
| 566 |
Name = "MyBackup" |
| 567 |
Type = Backup |
| 568 |
Client=localhost-fd |
| 569 |
FileSet = "Full Set" |
| 570 |
Storage = File |
| 571 |
Messages = Standard |
| 572 |
Pool = Default |
| 573 |
SpoolData = yes |
| 574 |
} |
| 575 |
|
| 576 |
# Default pool definition |
| 577 |
Pool { |
| 578 |
Name = Default |
| 579 |
Pool Type = Backup |
| 580 |
Volume Retention = 365d # one year |
| 581 |
NextPool = Full |
| 582 |
Storage = File |
| 583 |
} |
| 584 |
|
| 585 |
Pool { |
| 586 |
Name = Full |
| 587 |
Pool Type = Backup |
| 588 |
Volume Retention = 365d # one year |
| 589 |
Storage = DiskChanger |
| 590 |
} |
| 591 |
|
| 592 |
# Definition of file storage device |
| 593 |
Storage { |
| 594 |
Name = File |
| 595 |
Address = localhost |
| 596 |
Password = "xxx" |
| 597 |
Device = FileStorage |
| 598 |
Media Type = File |
| 599 |
Maximum Concurrent Jobs = 5 |
| 600 |
} |
| 601 |
|
| 602 |
# Definition of DDS Virtual tape disk storage device |
| 603 |
Storage { |
| 604 |
Name = DiskChanger |
| 605 |
Address = localhost # N.B. Use a fully qualified name here |
| 606 |
Password = "yyy" |
| 607 |
Device = DiskChanger |
| 608 |
Media Type = DiskChangerMedia |
| 609 |
Maximum Concurrent Jobs = 4 |
| 610 |
Autochanger = yes |
| 611 |
} |
| 612 |
\end{bVerbatim} |
| 613 |
|
| 614 |
Then in bconsole or via a Run schedule, you would run the job as: |
| 615 |
|
| 616 |
\begin{bVerbatim} |
| 617 |
run job=MyBackup level=Full |
| 618 |
run job=MyBackup level=Incremental |
| 619 |
run job=MyBackup level=Differential |
| 620 |
run job=MyBackup level=Incremental |
| 621 |
run job=MyBackup level=Incremental |
| 622 |
\end{bVerbatim} |
| 623 |
|
| 624 |
So providing there were changes between each of those jobs, you would end up |
| 625 |
with a Full backup, a Differential, which includes the first Incremental |
| 626 |
backup, then two Incremental backups. All the above jobs would be written to |
| 627 |
the \textbf{Default} pool. |
| 628 |
|
| 629 |
\smallskip |
| 630 |
To consolidate those backups into a new Full backup, you would run the |
| 631 |
following: |
| 632 |
|
| 633 |
\begin{bVerbatim} |
| 634 |
run job=MyBackup level=VirtualFull |
| 635 |
\end{bVerbatim} |
| 636 |
|
| 637 |
And it would produce a new Full backup without using the client, and the output |
| 638 |
would be written to the \textbf{Full} Pool which uses the Diskchanger Storage. |
| 639 |
|
| 640 |
\smallskip |
| 641 |
If the Virtual Full is run, and there are no prior Jobs, the Virtual Full will |
| 642 |
fail with an error. |
| 643 |
|
| 644 |
Note, the Start and End time of the Virtual Full backup is set to the |
| 645 |
values for the last job included in the Virtual Full (in the above example, |
| 646 |
it is an Increment). This is so that if another incremental is done, which |
| 647 |
will be based on the Virtual Full, it will backup all files from the |
| 648 |
last Job included in the Virtual Full rather than from the time the Virtual |
| 649 |
Full was actually run. |
| 650 |
|
| 651 |
\subsection{Manually Specify Jobs to Consolidate} |
| 652 |
By default \bacula{} is selecting jobs automatically, however, you may want to |
| 653 |
choose any point in time to create the Virtual backup. |
| 654 |
|
| 655 |
For example, if you have the following Jobs in your catalog: |
| 656 |
\begin{bVerbatim} |
| 657 |
+-------+---------+-------+----------+----------+-----------+ |
| 658 |
| JobId | Name | Level | JobFiles | JobBytes | JobStatus | |
| 659 |
+-------+---------+-------+----------+----------+-----------+ |
| 660 |
| 1 | Vbackup | F | 1754 | 50118554 | T | |
| 661 |
| 2 | Vbackup | I | 1 | 4 | T | |
| 662 |
| 3 | Vbackup | I | 1 | 4 | T | |
| 663 |
| 4 | Vbackup | D | 2 | 8 | T | |
| 664 |
| 5 | Vbackup | I | 1 | 6 | T | |
| 665 |
| 6 | Vbackup | I | 10 | 60 | T | |
| 666 |
| 7 | Vbackup | I | 11 | 65 | T | |
| 667 |
| 8 | Save | F | 1758 | 50118564 | T | |
| 668 |
+-------+---------+-------+----------+----------+-----------+ |
| 669 |
\end{bVerbatim} |
| 670 |
|
| 671 |
If you want to consolidate only the first 3 jobs and create a virtual backup |
| 672 |
equivalent to Job 1 + Job 2 + Job 3, you will use \texttt{jobid=3} in the |
| 673 |
\bcommandname{run} command, then \bacula{} will select the previous Full backup, the |
| 674 |
previous Differential (if any) and all subsequent Incremental jobs. |
| 675 |
|
| 676 |
\begin{bVerbatim} |
| 677 |
run job=Vbackup jobid=3 level=VirtualFull |
| 678 |
\end{bVerbatim} |
| 679 |
|
| 680 |
If you want to consolidate a specific job list, you must specify the exact |
| 681 |
list of jobs to merge in the \bcommandname{run} command line. For example, to consolidate |
| 682 |
the last Differential and all subsequent Incremental, you will use |
| 683 |
\texttt{jobid=4,5,6,7} or \texttt{jobid=4-7} in the \bcommandname{run} command line. As one |
| 684 |
of the Job in the list is a Differential backup, \bacula{} will set the new job |
| 685 |
level to Differential. If the list is composed only with Incremental jobs, |
| 686 |
the new job will have a level set to Incremental. |
| 687 |
|
| 688 |
\begin{bVerbatim} |
| 689 |
run job=Vbackup jobid=4-7 level=VirtualFull |
| 690 |
\end{bVerbatim} |
| 691 |
|
| 692 |
When using this feature, \bacula{} will automatically discard jobs that are not |
| 693 |
related to the current Job. For example, specifying \texttt{jobid=7,8}, |
| 694 |
\bacula{} will discard the jobid 8. |
| 695 |
|
| 696 |
If you know what you are doing and still want to consolidate jobs that have |
| 697 |
different names (so probably different clients, filesets, etc\ldots{}), you must |
| 698 |
use \texttt{alljobid=} keyword instead of \texttt{jobid=}. |
| 699 |
|
| 700 |
\begin{bVerbatim} |
| 701 |
run job=Vbackup alljobid=1-3,6-8 level=VirtualFull |
| 702 |
\end{bVerbatim} |
| 703 |
|