| 1 |
\chapter{The Job}\label{chap:jobs} |
| 2 |
\section{Definition} |
| 3 |
The Job is the basic unit in Bacula and is run by the Director. It ties together the following items: |
| 4 |
\begin{bitemize} |
| 5 |
\item \textbf{Who}: The \emph{Client}, the machine to backup |
| 6 |
\item \textbf{What}: The \emph{File Set}, which files to backup and not to backup |
| 7 |
\item \textbf{Where}: |
| 8 |
\begin{bitemize} |
| 9 |
\item \emph{Storage}: what physical device to backup to |
| 10 |
\item \emph{Pool}: which set of Volumes to use |
| 11 |
\item \emph{Catalog}: where to keep track of files |
| 12 |
\end{bitemize} |
| 13 |
\item \textbf{When}: \emph{Schedule}: when to run the Job |
| 14 |
\end{bitemize} |
| 15 |
|
| 16 |
\section{Types} |
| 17 |
There are several types of jobs in Bacula: |
| 18 |
%I think we do not need the character codes from the catalog - ARL |
| 19 |
\begin{bitemize} |
| 20 |
\item Backup |
| 21 |
\item Restore |
| 22 |
\item Admin |
| 23 |
\item Verify |
| 24 |
\item Copy |
| 25 |
\item Migration |
| 26 |
\item Archive |
| 27 |
%\item Scan |
| 28 |
\item Console connection |
| 29 |
\item Internal system job |
| 30 |
\end{bitemize} |
| 31 |
%\begin{bitemize} |
| 32 |
%\item Backup (B) |
| 33 |
%\item Restore (R) |
| 34 |
%\item Admin (D) |
| 35 |
%\item Verify (V) |
| 36 |
%\item Copy (C/c) |
| 37 |
%\item Migration (M/g) |
| 38 |
%\item Archive (A) |
| 39 |
%\item Scan (S) |
| 40 |
%\item Console btool (U) |
| 41 |
%\item Internal system ”job” (I) |
| 42 |
%\end{bitemize} |
| 43 |
|
| 44 |
\section{Backup Job Description} |
| 45 |
As shown in figure~\vref{fig:back-up-job-diagram}, when running a backup job: |
| 46 |
\begin{bitemize} |
| 47 |
\item the Bacula Director will connect to the File Daemon and Storage Daemon, |
| 48 |
\item the Director will then send to the File Daemon the necessary information to actually realize the backup job, like the Level, the File Set, the Storage Daemon to contact, etc. |
| 49 |
\item then the File Daemon will contact the Storage Daemon, identify the Data to be sent and send them as well as the related meta data (file attributes) to the Storage Daemon |
| 50 |
\item the Storage Daemon will put them into one or more Bacula Volumes, according to the Pool Resource chosen for the Job |
| 51 |
\item the Storage Daemon will send metadata back to the Director. |
| 52 |
\end{bitemize} |
| 53 |
\bimageH[0.75\linewidth]{bee8-back-up-job-diagram}{The backup job diagram}{fig:back-up-job-diagram} |
| 54 |
%% \begin{figure} |
| 55 |
%% \begin{center} |
| 56 |
%% \includegraphics[width=0.75\linewidth]{bee8-back-up-job-diagram} |
| 57 |
%% \caption{The backup job diagram}\label{fig:back-up-job-diagram} |
| 58 |
%% \end{center} |
| 59 |
%% \end{figure} |
| 60 |
The moment Bacula will run a Backup Job depends on how this job is started: |
| 61 |
\begin{bitemize} |
| 62 |
\item manually through a \emph{Console}, e.\,g with \btool{bconsole}: \bcommand{run job=name-of-the-job} |
| 63 |
\item automatically by a defined and referenced Schedule |
| 64 |
\item automatically by an external script or command like\\ |
| 65 |
\bcommand{bconsole -c bconsole.conf <{}< EOF\\run job=name-of-the-job \ldots\\EOF} |
| 66 |
\end{bitemize} |
| 67 |
|
| 68 |
\subsection{Admin Jobs} |
| 69 |
An Admin Job is a job which does not backup or move any data, but which can be used to execute scripts through the \emph{Run Script} resource and \emph{Console} or \emph{Command} directives. |
| 70 |
The following definition will launch the script ``/opt/bacula/scripts/my-script.pl'' located on the Director machine according to the Schedule ``AdminSchedule'' |
| 71 |
\begin{bVerbatim} |
| 72 |
Job { |
| 73 |
Type = Admin |
| 74 |
Run Script { |
| 75 |
Runs When = Before |
| 76 |
Runs on Client = No |
| 77 |
Command = "/opt/bacula/scripts/my-script.pl" |
| 78 |
} |
| 79 |
Schedule = AdminSchedule |
| 80 |
... |
| 81 |
} |
| 82 |
\end{bVerbatim} |
| 83 |
|
| 84 |
\subsection{Migration, Copy and Virtual Full Jobs} |
| 85 |
Bacula provides three kinds of Jobs that do not actually backup data, but move data around. They are doing very similar things: |
| 86 |
\begin{bitemize} |
| 87 |
\item Migration Jobs allow the migration of job data from one volume to another one; This is usually interesting when implementing a Disk-to-Disk-to-Tape strategy, or, more general, any sort of multi tiered backup storage system. |
| 88 |
\item Copy Jobs copy a Job's data to another volume. This is usually used when the requirements include at least two places where backup data is being kept. |
| 89 |
\item Virtual Full Job which provide a way to consolidate several jobs into one, without requesting any data from the client, based only on existing backup data. |
| 90 |
\end{bitemize} |
| 91 |
All these kind of Jobs rely on a \emph{Next Pool} directive. |
| 92 |
|
| 93 |
\subsubsection{Virtual Full Consideration} |
| 94 |
Virtual Full backups are a way to do backups in an ``Incremental forever'' style while continuing to provide Full backups at the same time. |
| 95 |
|
| 96 |
When backing up data at incremental or differential levels, Bacula (by default) does not do anything regarding removed or moved files or directories. Which implies that the result of a restoration could be different than the latest state of the machine. For that reason, we advise using ``Accurate'' mode which is enabled by the directive of the same name. When set to ``yes'' in a Job, Bacula will record removed missing files or directories, and depending on additional configuration, Bacula will also consider more criteria than just time stamps to determine if a file needs to be backed up. In this case, Bacula will restore the machine to the exact same state (from a backup content point of view) that it was in during the backups. |
| 97 |
|
| 98 |
Administrators will understand that the ``Accurate'' mode takes additional resources and time when running backups. |
| 99 |
|
| 100 |
To improve performance, you may use the ``Accurate'' directive when using Virtual Full backups only for the last incremental before the Virtual Full itself. Activating this directive for every incremental backup would be even better but could increase the backup time. |
| 101 |
\begin{bVerbatim} |
| 102 |
# |
| 103 |
# A usual job definition |
| 104 |
Job { |
| 105 |
Name = "j-bacula" |
| 106 |
JobDefs = "DefaultJob" |
| 107 |
FileSet = "BaculaFileSet" |
| 108 |
Client = client-fd |
| 109 |
Schedule = s-Data2Disk |
| 110 |
Max Full interval = 19 days |
| 111 |
Run Script { |
| 112 |
Runs When = after |
| 113 |
Runs on Client = no |
| 114 |
Runs on Failure = yes |
| 115 |
Command = "/opt/bacula/scripts/run_copyjob.pl %l %i %b t-rdx j-copy-full" ; |
| 116 |
# Launching the copy job as soon as the backup is done |
| 117 |
# %l is job level |
| 118 |
# %i is job id |
| 119 |
# %b is job bytes |
| 120 |
# j-copy-full is the job name (see below) |
| 121 |
# The script "run_copyjob.pl" issues a shell command like |
| 122 |
# bconsole -c bconsole.conf << END_OF_DATA |
| 123 |
# run jobid=%i job=j-copy-full storage=t-rdx yes |
| 124 |
# quit |
| 125 |
# END_OF_DATA |
| 126 |
} |
| 127 |
} |
| 128 |
\end{bVerbatim} |
| 129 |
And here is the ``Copy'' job to copy the job from Disk to RDX |
| 130 |
\begin{bVerbatim} |
| 131 |
Job { |
| 132 |
Name = "j-copy-full" |
| 133 |
Type = Copy |
| 134 |
Level = Full |
| 135 |
Client = client-fd |
| 136 |
File Set = "Empty Set" |
| 137 |
Messages = Standard |
| 138 |
Pool = PoolVFull |
| 139 |
Maximum Concurrent Jobs = 1 |
| 140 |
} |
| 141 |
\end{bVerbatim} |
| 142 |
And the Schedule used to run the several levels |
| 143 |
\begin{bVerbatim} |
| 144 |
Schedule { |
| 145 |
Name = s-Data2Disk |
| 146 |
Run = Level=incremental monday-thursday,saturday at 21:00 |
| 147 |
Run = Level=incremental accurate=yes friday at 12:30 |
| 148 |
Run = Level=VirtualFull priority=15 friday at 12:35 |
| 149 |
} |
| 150 |
\end{bVerbatim} |
| 151 |
|
| 152 |
|
| 153 |
\section{Pruning} |
| 154 |
Pruning by default occurs at the end of a Job. When doing some tests or if you only have few jobs a day, this could be fine. But as soon as the number of jobs is growing, you would prefer to manage pruning another way to let your Catalog do its best for the backup jobs, not for the database administrative tasks. |
| 155 |
|
| 156 |
In such a situation, you will configure your clients not to activate the pruning algorithm, using the ``Auto Prune'' Directive such as the following: |
| 157 |
\begin{bVerbatim} |
| 158 |
Client { |
| 159 |
Name = client-fd |
| 160 |
Address = bacula.example.com |
| 161 |
FDPort = 9102 |
| 162 |
Catalog = Catalog |
| 163 |
Password = "do-you-want-a-very-strong-password?" |
| 164 |
File Retention = 15 days |
| 165 |
Job Retention = 50 days |
| 166 |
AutoPrune = no |
| 167 |
# No automatic pruning at the end of a job for this client |
| 168 |
} |
| 169 |
\end{bVerbatim} |
| 170 |
If you don't do anything more, your Catalog will grow infinitely. To keep it at its best, you should define an ``Admin'' job, like the following: |
| 171 |
\begin{bVerbatim} |
| 172 |
Job { |
| 173 |
Name = "admin-manual-pruning" |
| 174 |
Type = Admin |
| 175 |
JobDefs = "DefaultJob" |
| 176 |
RunScript { |
| 177 |
Runs When = Before |
| 178 |
# below command relies on proper PATH! |
| 179 |
Command = "/bin/sh -c \"echo prune expired volume yes\" | bconsole" |
| 180 |
Runs On Client = no |
| 181 |
} |
| 182 |
Schedule = s-Prune |
| 183 |
} |
| 184 |
\end{bVerbatim} |
| 185 |
As a Bacula volume can contain one or more jobs (or parts of jobs) and a job contains one or more files, the pruning process will have side effects: |
| 186 |
\begin{bitemize} |
| 187 |
\item when pruning a Volume, all the jobs related to the Volume are pruned |
| 188 |
\item when pruning a Job, all the files related to this jobs are pruned |
| 189 |
\end{bitemize} |
| 190 |
That is the reason why you should have the following inequality: |
| 191 |
\smallskip{} |
| 192 |
|
| 193 |
\begin{center} |
| 194 |
\bhighlight{File Retention $\leqslant{}$ Job Retention $\leqslant{}$ Volume Retention} |
| 195 |
\end{center} |
| 196 |
%% R_f \leqslant R_j \leqslant R_v |
| 197 |
%%\end{equation} |
| 198 |
%% Where: |
| 199 |
%% \begin{bitemize} |
| 200 |
%% \item $$ R_f $$ is the File retention period |
| 201 |
%% \item $R_j$ is the Job retention period |
| 202 |
%% \item $R_v$ is the Volume retention period |
| 203 |
%% \end{bitemize} |