| 1 |
\chapter{Catalog Services} |
| 2 |
\label{blb:TheChapterStart30} |
| 3 |
\label{blb:developers:catalog:service} |
| 4 |
\index[general]{Services!Catalog} |
| 5 |
\index[general]{Catalog Services} |
| 6 |
|
| 7 |
\section{General} |
| 8 |
\index[general]{General} |
| 9 |
|
| 10 |
This chapter is intended to be a technical discussion of the Catalog services |
| 11 |
and as such is not targeted at end users but rather at developers and system |
| 12 |
administrators that want or need to know more of the working details of \textbf{\bacula{}}. |
| 13 |
|
| 14 |
The \textbf{\bacula{} Catalog} services consist of the programs that provide the \acs{SQL} |
| 15 |
database engine for storage and retrieval of all information concerning files |
| 16 |
that were backed up and their locations on the storage media. |
| 17 |
|
| 18 |
We have investigated the possibility of using the following \acs{SQL} engines for |
| 19 |
\bacula{}: Beagle, mSQL, \acs{GNU} \acs{SQL}, \postgresql{}, Oracle, and \mysql{}. Each |
| 20 |
presents certain problems with either licensing or maturity. At present, we |
| 21 |
have chosen for development purposes to use \mysql{} and \postgresql{}. |
| 22 |
\mysql{} was chosen because it is fast, proven to be reliable, widely used, and |
| 23 |
actively being developed. \mysql{} is released under the \acs{GNU} \acs{GPL} license. |
| 24 |
\postgresql{} was chosen because it is a full-featured, very mature database, and |
| 25 |
because Dan Langille did the \bacula{} driver for it. \postgresql{} is distributed |
| 26 |
under the BSD license. |
| 27 |
|
| 28 |
The \bacula{} \acs{SQL} code has been written in a manner that will allow it to be |
| 29 |
easily modified to support any of the current \acs{SQL} database systems on the |
| 30 |
market (for example: mSQL, iODBC, unixODBC, Solid, OpenLink ODBC, EasySoft |
| 31 |
ODBC, InterBase, Oracle8, Oracle7, and DB2). |
| 32 |
|
| 33 |
If you do not specify either \texttt{{-}{-}with-mysql} or \texttt{{-}{-}with-postgresq} on the \btool{./configure} line, \bacula{} will use its minimalist |
| 34 |
internal database. This database is kept for build reasons but is no longer |
| 35 |
supported. \bacula{} \textbf{requires} one of the three databases (\mysql{}, |
| 36 |
\postgresql{}, or \sqlite{}) to run. |
| 37 |
|
| 38 |
\subsection{Filenames and Maximum Filename Length} |
| 39 |
\index[general]{Filenames and Maximum Filename Length} |
| 40 |
\index[general]{Length!Filenames and Maximum Filename} |
| 41 |
|
| 42 |
In general, either \mysql{} or \postgresql{} permit storing arbitrary long |
| 43 |
path names and file names in the catalog database. In practice, there still |
| 44 |
may be one or two places in the Catalog interface code that restrict the |
| 45 |
maximum path length to 512 characters and the maximum file name length to 512 |
| 46 |
characters. These restrictions are believed to have been removed. Please note, |
| 47 |
these restrictions apply only to the Catalog database and thus to your ability |
| 48 |
to list online the files saved during any job. All information received and |
| 49 |
stored by the Storage daemon (normally on tape) allows and handles arbitrarily |
| 50 |
long path and filenames. |
| 51 |
|
| 52 |
\subsection{Installing and Configuring \mysql{}} |
| 53 |
\index[general]{\mysql{}!Installing and Configuring} |
| 54 |
\index[general]{Installing and Configuring \mysql{}} |
| 55 |
|
| 56 |
For the details of installing and configuring \mysql{}, please see the |
| 57 |
\bxlink{Installing and Configuring \mysql{}}{blb:MySqlChapter}{main}{chapter} of |
| 58 |
the \bmainman{}. |
| 59 |
|
| 60 |
\subsection{Installing and Configuring \postgresql{}} |
| 61 |
\index[general]{\postgresql{}!Installing and Configuring} |
| 62 |
\index[general]{Installing and Configuring \postgresql{}} |
| 63 |
|
| 64 |
For the details of installing and configuring \postgresql{}, please see the |
| 65 |
\bxlink{Installing and Configuring \postgresql{}}{blb:PostgreSqlChapter}{main}{chapter} |
| 66 |
of the \bmainman{}. |
| 67 |
|
| 68 |
\subsection{Internal \bacula{} Catalog} |
| 69 |
\index[general]{Catalog!Internal \bacula{}} |
| 70 |
\index[general]{Internal \bacula{} Catalog} |
| 71 |
|
| 72 |
Please see the \bxlink{Internal \bacula{} Database}{blb:chap:InternalBaculaDatabase}{misc}{chapter} |
| 73 |
of the \bmiscman{} for more details. |
| 74 |
|
| 75 |
\subsection{Database Table Design} |
| 76 |
\index[general]{Design!Database Table} |
| 77 |
\index[general]{Database Table Design} |
| 78 |
|
| 79 |
All discussions that follow pertain to the \mysql{} database. The details for the |
| 80 |
\postgresql{} databases are quite similar. |
| 81 |
|
| 82 |
Because the Catalog database may contain very large amounts of data for large |
| 83 |
sites, we have made a modest attempt to normalize the data tables to reduce |
| 84 |
redundant information. While reducing the size of the database significantly, |
| 85 |
it does, unfortunately, add some complications to the structures. |
| 86 |
|
| 87 |
In simple terms, the Catalog database must contain a record of all Jobs run by |
| 88 |
\bacula{}, and for each Job, it must maintain a list of all files saved, with |
| 89 |
their File Attributes (permissions, create date, ...), and the location and |
| 90 |
Media on which the file is stored. This is seemingly a simple task, but it |
| 91 |
represents a huge amount interlinked data. Note: the list of files and their |
| 92 |
attributes is not maintained when using the internal \bacula{} database. The data |
| 93 |
stored in the File records, which allows the user or administrator to obtain a |
| 94 |
list of all files backed up during a job, is by far the largest volume of |
| 95 |
information put into the Catalog database. |
| 96 |
|
| 97 |
Although the Catalog database has been designed to handle backup data for |
| 98 |
multiple clients, some users may want to maintain multiple databases, one for |
| 99 |
each machine to be backed up. This reduces the risk of confusion of accidental |
| 100 |
restoring a file to the wrong machine as well as reducing the amount of data |
| 101 |
in a single database, thus increasing efficiency and reducing the impact of a |
| 102 |
lost or damaged database. |
| 103 |
|
| 104 |
\section{Sequence of Creation of Records for a Save Job} |
| 105 |
\index[general]{Sequence of Creation of Records for a Save Job} |
| 106 |
\index[general]{Job!Sequence of Creation of Records for a Save} |
| 107 |
|
| 108 |
Start with StartDate, ClientName, Filename, Path, Attributes, MediaName, |
| 109 |
MediaCoordinates. (PartNumber, NumParts). In the steps below, ``Create new'' |
| 110 |
means to create a new record whether or not it is unique. ``Create unique'' |
| 111 |
means each record in the database should be unique. Thus, one must first |
| 112 |
search to see if the record exists, and only if not should a new one be |
| 113 |
created, otherwise the existing RecordId should be used. |
| 114 |
|
| 115 |
\begin{benumerate} |
| 116 |
\item Create new Job record with StartDate; save JobId |
| 117 |
\item Create unique Media record; save MediaId |
| 118 |
\item Create unique Client record; save ClientId |
| 119 |
\item Create unique Filename record; save FilenameId |
| 120 |
\item Create unique Path record; save PathId |
| 121 |
\item Create unique Attribute record; save AttributeId |
| 122 |
store ClientId, FilenameId, PathId, and Attributes |
| 123 |
\item Create new File record |
| 124 |
store JobId, AttributeId, MediaCoordinates, etc |
| 125 |
\item Repeat steps 4 through 8 for each file |
| 126 |
\item Create a JobMedia record; save MediaId |
| 127 |
\item Update Job record filling in EndDate and other Job statistics |
| 128 |
\end{benumerate} |
| 129 |
|
| 130 |
\section{Database Tables} |
| 131 |
\index[general]{Database Tables} |
| 132 |
\index[general]{Tables!Database} |
| 133 |
\LTXtable{\linewidth}{t-dev-dbpath} |
| 134 |
|
| 135 |
The \bilink{\btable{Path} table}{blb:dev:path:table:layout} contains the path or directory names of all |
| 136 |
directories on the system or systems. The filename and any MSDOS disk name are |
| 137 |
stripped off. As with the filename, only one copy of each directory name is |
| 138 |
kept regardless of how many machines or drives have the same directory. These |
| 139 |
path names should be stored in Unix path name format. |
| 140 |
|
| 141 |
Some simple testing on a Linux file system indicates that separating the |
| 142 |
filename and the path may be more complication than is warranted by the space |
| 143 |
savings. For example, this system has a total of 89,097 files, 60,467 of which |
| 144 |
have unique filenames, and there are 4,374 unique paths. |
| 145 |
|
| 146 |
Finding all those files and doing two stats() per file takes an average wall |
| 147 |
clock time of 1 min 35 seconds on a 400MHz machine running RedHat 6.1 Linux. |
| 148 |
|
| 149 |
Finding all those files and putting them directly into a \mysql{} database with |
| 150 |
the path and filename defined as TEXT, which is variable length up to 65,535 |
| 151 |
characters takes 19 mins 31 seconds and creates a 27.6 MByte database. |
| 152 |
|
| 153 |
Doing the same thing, but inserting them into Blob fields with the filename |
| 154 |
indexed on the first 30 characters and the path name indexed on the 255 (max) |
| 155 |
characters takes 5 mins 18 seconds and creates a 5.24 MB database. Rerunning |
| 156 |
the job (with the database already created) takes about 2 mins 50 seconds. |
| 157 |
|
| 158 |
Running the same as the last one (Path and Filename Blob), but Filename |
| 159 |
indexed on the first 30 characters and the Path on the first 50 characters |
| 160 |
(linear search done there after) takes 5 mins on the average and creates a 3.4 |
| 161 |
MB database. Rerunning with the data already in the DB takes 3 mins 35 |
| 162 |
seconds. |
| 163 |
|
| 164 |
Finally, saving only the full path name rather than splitting the path and the |
| 165 |
file, and indexing it on the first 50 characters takes 6 mins 43 seconds and |
| 166 |
creates a 7.35 MB database. |
| 167 |
|
| 168 |
|
| 169 |
\LTXtable{\linewidth}{t-dev-dbfile} |
| 170 |
|
| 171 |
The \bilink{\btable{File} table}{blb:dev:file:table:layout} contains one entry for each file backed up by |
| 172 |
\bacula{}. Thus a file that is backed up multiple times (as is normal) will have |
| 173 |
multiple entries in the File table. This will probably be the table with the |
| 174 |
most number of records. Consequently, it is essential to keep the size of this |
| 175 |
record to an absolute minimum. At the same time, this table must contain all |
| 176 |
the information (or pointers to the information) about the file and where it |
| 177 |
is backed up. Since a file may be backed up many times without having changed, |
| 178 |
the path is stored in separate tables. |
| 179 |
|
| 180 |
This table contains by far the largest amount of information in the Catalog |
| 181 |
database, both from the stand point of number of records, and the stand point |
| 182 |
of total database size. As a consequence, the user must take care to |
| 183 |
periodically reduce the number of File records using the \bcommandname{retention} |
| 184 |
command in the Console program. |
| 185 |
|
| 186 |
\LTXtable{\linewidth}{t-dev-dbjob} |
| 187 |
|
| 188 |
The \bilink{\btable{Job} table}{blb:dev:job:table:layout} contains one record for each Job run by \bacula{}. Thus |
| 189 |
normally, there will be one per day per machine added to the database. Note, |
| 190 |
the JobId is used to index Job records in the database, and it often is shown |
| 191 |
to the user in the Console program. However, care must be taken with its use |
| 192 |
as it is not unique from database to database. For example, the user may have |
| 193 |
a database for Client data saved on machine Rufus and another database for |
| 194 |
Client data saved on machine Roxie. In this case, the two database will each |
| 195 |
have JobIds that match those in another database. For a unique reference to a |
| 196 |
Job, see Job below. |
| 197 |
|
| 198 |
The Name field of the Job record corresponds to the Name resource record given |
| 199 |
in the Director's configuration file. Thus it is a generic name, and it will |
| 200 |
be normal to find many Jobs (or even all Jobs) with the same Name. |
| 201 |
|
| 202 |
The Job field contains a combination of the Name and the schedule time of the |
| 203 |
Job by the Director. Thus for a given Director, even with multiple Catalog |
| 204 |
databases, the Job will contain a unique name that represents the Job. |
| 205 |
|
| 206 |
For a given Storage daemon, the VolSessionId and VolSessionTime form a unique |
| 207 |
identification of the Job. This will be the case even if multiple Directors |
| 208 |
are using the same Storage daemon. |
| 209 |
|
| 210 |
The Job Type (or simply Type) can have one of the following values: |
| 211 |
|
| 212 |
\LTXtable{\linewidth}{t-dev-dbjobtypes} |
| 213 |
|
| 214 |
Note, the Job Type values in table \vref{blb:dev:jobtypes} noted above are not kept in an \acs{SQL} table. |
| 215 |
|
| 216 |
|
| 217 |
The JobStatus field specifies how the job terminated, and can be one of the |
| 218 |
following: |
| 219 |
\LTXtable{\linewidth}{t-dev-dbjobstatuses} |
| 220 |
|
| 221 |
|
| 222 |
\LTXtable{\linewidth}{t-dev-dbfileset} |
| 223 |
|
| 224 |
The \bilink{\btable{FileSet} table}{blb:dev:filesets:table:layout} contains one entry for each FileSet that is used. The |
| 225 |
MD5 signature is kept to ensure that if the user changes anything inside the |
| 226 |
FileSet, it will be detected and the new FileSet will be used. This is |
| 227 |
particularly important when doing an incremental update. If the user deletes a |
| 228 |
file or adds a file, we need to ensure that a Full backup is done prior to the |
| 229 |
next incremental. |
| 230 |
|
| 231 |
|
| 232 |
\LTXtable{\linewidth}{t-dev-dbjobmedia} |
| 233 |
|
| 234 |
The \bilink{\btable{JobMedia} table}{blb:dev:jobmedia:table:layout} contains one entry at the following: start of |
| 235 |
the job, start of each new tape file, start of each new tape, end of the |
| 236 |
job. Since by default, a new tape file is written every 2GB, in general, |
| 237 |
you will have more than 2 JobMedia records per Job. The number can be |
| 238 |
varied by changing the "Maximum File Size" specified in the Device |
| 239 |
resource. This record allows \bacula{} to efficiently position close to |
| 240 |
(within 2GB) any given file in a backup. For restoring a full Job, |
| 241 |
these records are not very important, but if you want to retrieve |
| 242 |
a single file that was written near the end of a 100GB backup, the |
| 243 |
JobMedia records can speed it up by orders of magnitude by permitting |
| 244 |
forward spacing files and blocks rather than reading the whole 100GB |
| 245 |
backup. |
| 246 |
|
| 247 |
|
| 248 |
|
| 249 |
|
| 250 |
\LTXtable{\linewidth}{t-dev-dbmedia} |
| 251 |
|
| 252 |
The \bilink{\btable{Volume} table}{blb:dev:media:table:layout}\footnote{Internally referred to as the Media table} contains |
| 253 |
one entry for each volume, that is each tape, cassette (8mm, DLT, DAT, ...), |
| 254 |
or file on which information is or was backed up. There is one Volume record |
| 255 |
created for each of the NumVols specified in the Pool resource record. |
| 256 |
|
| 257 |
\LTXtable{\linewidth}{t-dev-dbpool} |
| 258 |
|
| 259 |
The \bilink{\btable{Pool} table}{blb:dev:pool:table:layout} contains one entry for each media pool controlled by |
| 260 |
\bacula{} in this database. One media record exists for each of the NumVols |
| 261 |
contained in the Pool. The PoolType is a \bacula{} defined keyword. The MediaType |
| 262 |
is defined by the administrator, and corresponds to the MediaType specified in |
| 263 |
the Director's Storage definition record. The CurrentVol is the sequence |
| 264 |
number of the Media record for the current volume. |
| 265 |
|
| 266 |
|
| 267 |
\LTXtable{\linewidth}{t-dev-dbclient} |
| 268 |
|
| 269 |
The \bilink{\btable{Client} table}{blb:dev:client:table:layout} contains one entry for each machine backed up by \bacula{} |
| 270 |
in this database. Normally the Name is a fully qualified domain name. |
| 271 |
|
| 272 |
|
| 273 |
\LTXtable{\linewidth}{t-dev-dbstorage} |
| 274 |
|
| 275 |
The \bilink{\btable{Storage} table}{blb:dev:storage:table:layout} contains one entry for each Storage used. |
| 276 |
|
| 277 |
|
| 278 |
\LTXtable{\linewidth}{t-dev-dbcounter} |
| 279 |
|
| 280 |
The \bilink{\btable{Counter} table}{blb:dev:counter:table:layout} contains one entry for each permanent counter defined |
| 281 |
by the user. |
| 282 |
|
| 283 |
\LTXtable{\linewidth}{t-dev-dbjobhistory} |
| 284 |
|
| 285 |
The \bilink{\btable{JobHisto} table}{blb:dev:jobhistory:table:layout} is the same as the Job table, but it keeps |
| 286 |
long term statistics (i.e. it is not pruned with the Job). |
| 287 |
|
| 288 |
|
| 289 |
\LTXtable{\linewidth}{t-dev-dblog} |
| 290 |
|
| 291 |
The \bilink{\btable{Log} table}{blb:dev:log:table:layout} contains a log of all Job output. |
| 292 |
|
| 293 |
\LTXtable{\linewidth}{t-dev-dblocation} |
| 294 |
|
| 295 |
The \bilink{\btable{Location} table}{blb:dev:location:table:layout} defines where a Volume is physically. |
| 296 |
|
| 297 |
|
| 298 |
\LTXtable{\linewidth}{t-dev-dblocationlog} |
| 299 |
|
| 300 |
The \bilink{\btable{Location Log} table}{blb:dev:locationlog:table:layout} contains a log of all Job output. |
| 301 |
|
| 302 |
|
| 303 |
\LTXtable{\linewidth}{t-dev-dbversion} |
| 304 |
|
| 305 |
The \bilink{\btable{Version} table}{blb:dev:version:table:layout} defines the \bacula{} database version number. \bacula{} |
| 306 |
checks this number before reading the database to ensure that it is compatible |
| 307 |
with the \bacula{} binary file. |
| 308 |
|
| 309 |
|
| 310 |
\LTXtable{\linewidth}{t-dev-dbbasefiles} |
| 311 |
|
| 312 |
The \bilink{\btable{BaseFiles} table}{blb:dev:basefiles:table:layout} contains all the File references for a particular |
| 313 |
JobId that point to a Base file -- i.e. they were previously saved and hence |
| 314 |
were not saved in the current JobId but in BaseJobId under FileId. FileIndex |
| 315 |
is the index of the file, and is used for optimization of Restore jobs to |
| 316 |
prevent the need to read the FileId record when creating the in memory tree. |
| 317 |
This record is not yet implemented. |