Debian Codegraph @

manuals/en/developers/catalog.tex

raw
1 \chapter{Catalog Services}
2 \label{blb:TheChapterStart30}
3 \label{blb:developers:catalog:service}
4 \index[general]{Services!Catalog}
5 \index[general]{Catalog Services}
6
7 \section{General}
8 \index[general]{General}
9
10 This chapter is intended to be a technical discussion of the Catalog services
11 and as such is not targeted at end users but rather at developers and system
12 administrators that want or need to know more of the working details of \textbf{\bacula{}}.
13
14 The \textbf{\bacula{} Catalog} services consist of the programs that provide the \acs{SQL}
15 database engine for storage and retrieval of all information concerning files
16 that were backed up and their locations on the storage media.
17
18 We have investigated the possibility of using the following \acs{SQL} engines for
19 \bacula{}: Beagle, mSQL, \acs{GNU} \acs{SQL}, \postgresql{}, Oracle, and \mysql{}. Each
20 presents certain problems with either licensing or maturity. At present, we
21 have chosen for development purposes to use \mysql{} and \postgresql{}.
22 \mysql{} was chosen because it is fast, proven to be reliable, widely used, and
23 actively being developed. \mysql{} is released under the \acs{GNU} \acs{GPL} license.
24 \postgresql{} was chosen because it is a full-featured, very mature database, and
25 because Dan Langille did the \bacula{} driver for it. \postgresql{} is distributed
26 under the BSD license.
27
28 The \bacula{} \acs{SQL} code has been written in a manner that will allow it to be
29 easily modified to support any of the current \acs{SQL} database systems on the
30 market (for example: mSQL, iODBC, unixODBC, Solid, OpenLink ODBC, EasySoft
31 ODBC, InterBase, Oracle8, Oracle7, and DB2).
32
33 If you do not specify either \texttt{{-}{-}with-mysql} or \texttt{{-}{-}with-postgresq} on the \btool{./configure} line, \bacula{} will use its minimalist
34 internal database. This database is kept for build reasons but is no longer
35 supported. \bacula{} \textbf{requires} one of the three databases (\mysql{},
36 \postgresql{}, or \sqlite{}) to run.
37
38 \subsection{Filenames and Maximum Filename Length}
39 \index[general]{Filenames and Maximum Filename Length}
40 \index[general]{Length!Filenames and Maximum Filename}
41
42 In general, either \mysql{} or \postgresql{} permit storing arbitrary long
43 path names and file names in the catalog database. In practice, there still
44 may be one or two places in the Catalog interface code that restrict the
45 maximum path length to 512 characters and the maximum file name length to 512
46 characters. These restrictions are believed to have been removed. Please note,
47 these restrictions apply only to the Catalog database and thus to your ability
48 to list online the files saved during any job. All information received and
49 stored by the Storage daemon (normally on tape) allows and handles arbitrarily
50 long path and filenames.
51
52 \subsection{Installing and Configuring \mysql{}}
53 \index[general]{\mysql{}!Installing and Configuring}
54 \index[general]{Installing and Configuring \mysql{}}
55
56 For the details of installing and configuring \mysql{}, please see the
57 \bxlink{Installing and Configuring \mysql{}}{blb:MySqlChapter}{main}{chapter} of
58 the \bmainman{}.
59
60 \subsection{Installing and Configuring \postgresql{}}
61 \index[general]{\postgresql{}!Installing and Configuring}
62 \index[general]{Installing and Configuring \postgresql{}}
63
64 For the details of installing and configuring \postgresql{}, please see the
65 \bxlink{Installing and Configuring \postgresql{}}{blb:PostgreSqlChapter}{main}{chapter}
66 of the \bmainman{}.
67
68 \subsection{Internal \bacula{} Catalog}
69 \index[general]{Catalog!Internal \bacula{}}
70 \index[general]{Internal \bacula{} Catalog}
71
72 Please see the \bxlink{Internal \bacula{} Database}{blb:chap:InternalBaculaDatabase}{misc}{chapter}
73 of the \bmiscman{} for more details.
74
75 \subsection{Database Table Design}
76 \index[general]{Design!Database Table}
77 \index[general]{Database Table Design}
78
79 All discussions that follow pertain to the \mysql{} database. The details for the
80 \postgresql{} databases are quite similar.
81
82 Because the Catalog database may contain very large amounts of data for large
83 sites, we have made a modest attempt to normalize the data tables to reduce
84 redundant information. While reducing the size of the database significantly,
85 it does, unfortunately, add some complications to the structures.
86
87 In simple terms, the Catalog database must contain a record of all Jobs run by
88 \bacula{}, and for each Job, it must maintain a list of all files saved, with
89 their File Attributes (permissions, create date, ...), and the location and
90 Media on which the file is stored. This is seemingly a simple task, but it
91 represents a huge amount interlinked data. Note: the list of files and their
92 attributes is not maintained when using the internal \bacula{} database. The data
93 stored in the File records, which allows the user or administrator to obtain a
94 list of all files backed up during a job, is by far the largest volume of
95 information put into the Catalog database.
96
97 Although the Catalog database has been designed to handle backup data for
98 multiple clients, some users may want to maintain multiple databases, one for
99 each machine to be backed up. This reduces the risk of confusion of accidental
100 restoring a file to the wrong machine as well as reducing the amount of data
101 in a single database, thus increasing efficiency and reducing the impact of a
102 lost or damaged database.
103
104 \section{Sequence of Creation of Records for a Save Job}
105 \index[general]{Sequence of Creation of Records for a Save Job}
106 \index[general]{Job!Sequence of Creation of Records for a Save}
107
108 Start with StartDate, ClientName, Filename, Path, Attributes, MediaName,
109 MediaCoordinates. (PartNumber, NumParts). In the steps below, ``Create new''
110 means to create a new record whether or not it is unique. ``Create unique''
111 means each record in the database should be unique. Thus, one must first
112 search to see if the record exists, and only if not should a new one be
113 created, otherwise the existing RecordId should be used.
114
115 \begin{benumerate}
116 \item Create new Job record with StartDate; save JobId
117 \item Create unique Media record; save MediaId
118 \item Create unique Client record; save ClientId
119 \item Create unique Filename record; save FilenameId
120 \item Create unique Path record; save PathId
121 \item Create unique Attribute record; save AttributeId
122 store ClientId, FilenameId, PathId, and Attributes
123 \item Create new File record
124 store JobId, AttributeId, MediaCoordinates, etc
125 \item Repeat steps 4 through 8 for each file
126 \item Create a JobMedia record; save MediaId
127 \item Update Job record filling in EndDate and other Job statistics
128 \end{benumerate}
129
130 \section{Database Tables}
131 \index[general]{Database Tables}
132 \index[general]{Tables!Database}
133 \LTXtable{\linewidth}{t-dev-dbpath}
134
135 The \bilink{\btable{Path} table}{blb:dev:path:table:layout} contains the path or directory names of all
136 directories on the system or systems. The filename and any MSDOS disk name are
137 stripped off. As with the filename, only one copy of each directory name is
138 kept regardless of how many machines or drives have the same directory. These
139 path names should be stored in Unix path name format.
140
141 Some simple testing on a Linux file system indicates that separating the
142 filename and the path may be more complication than is warranted by the space
143 savings. For example, this system has a total of 89,097 files, 60,467 of which
144 have unique filenames, and there are 4,374 unique paths.
145
146 Finding all those files and doing two stats() per file takes an average wall
147 clock time of 1 min 35 seconds on a 400MHz machine running RedHat 6.1 Linux.
148
149 Finding all those files and putting them directly into a \mysql{} database with
150 the path and filename defined as TEXT, which is variable length up to 65,535
151 characters takes 19 mins 31 seconds and creates a 27.6 MByte database.
152
153 Doing the same thing, but inserting them into Blob fields with the filename
154 indexed on the first 30 characters and the path name indexed on the 255 (max)
155 characters takes 5 mins 18 seconds and creates a 5.24 MB database. Rerunning
156 the job (with the database already created) takes about 2 mins 50 seconds.
157
158 Running the same as the last one (Path and Filename Blob), but Filename
159 indexed on the first 30 characters and the Path on the first 50 characters
160 (linear search done there after) takes 5 mins on the average and creates a 3.4
161 MB database. Rerunning with the data already in the DB takes 3 mins 35
162 seconds.
163
164 Finally, saving only the full path name rather than splitting the path and the
165 file, and indexing it on the first 50 characters takes 6 mins 43 seconds and
166 creates a 7.35 MB database.
167
168
169 \LTXtable{\linewidth}{t-dev-dbfile}
170
171 The \bilink{\btable{File} table}{blb:dev:file:table:layout} contains one entry for each file backed up by
172 \bacula{}. Thus a file that is backed up multiple times (as is normal) will have
173 multiple entries in the File table. This will probably be the table with the
174 most number of records. Consequently, it is essential to keep the size of this
175 record to an absolute minimum. At the same time, this table must contain all
176 the information (or pointers to the information) about the file and where it
177 is backed up. Since a file may be backed up many times without having changed,
178 the path is stored in separate tables.
179
180 This table contains by far the largest amount of information in the Catalog
181 database, both from the stand point of number of records, and the stand point
182 of total database size. As a consequence, the user must take care to
183 periodically reduce the number of File records using the \bcommandname{retention}
184 command in the Console program.
185
186 \LTXtable{\linewidth}{t-dev-dbjob}
187
188 The \bilink{\btable{Job} table}{blb:dev:job:table:layout} contains one record for each Job run by \bacula{}. Thus
189 normally, there will be one per day per machine added to the database. Note,
190 the JobId is used to index Job records in the database, and it often is shown
191 to the user in the Console program. However, care must be taken with its use
192 as it is not unique from database to database. For example, the user may have
193 a database for Client data saved on machine Rufus and another database for
194 Client data saved on machine Roxie. In this case, the two database will each
195 have JobIds that match those in another database. For a unique reference to a
196 Job, see Job below.
197
198 The Name field of the Job record corresponds to the Name resource record given
199 in the Director's configuration file. Thus it is a generic name, and it will
200 be normal to find many Jobs (or even all Jobs) with the same Name.
201
202 The Job field contains a combination of the Name and the schedule time of the
203 Job by the Director. Thus for a given Director, even with multiple Catalog
204 databases, the Job will contain a unique name that represents the Job.
205
206 For a given Storage daemon, the VolSessionId and VolSessionTime form a unique
207 identification of the Job. This will be the case even if multiple Directors
208 are using the same Storage daemon.
209
210 The Job Type (or simply Type) can have one of the following values:
211
212 \LTXtable{\linewidth}{t-dev-dbjobtypes}
213
214 Note, the Job Type values in table \vref{blb:dev:jobtypes} noted above are not kept in an \acs{SQL} table.
215
216
217 The JobStatus field specifies how the job terminated, and can be one of the
218 following:
219 \LTXtable{\linewidth}{t-dev-dbjobstatuses}
220
221
222 \LTXtable{\linewidth}{t-dev-dbfileset}
223
224 The \bilink{\btable{FileSet} table}{blb:dev:filesets:table:layout} contains one entry for each FileSet that is used. The
225 MD5 signature is kept to ensure that if the user changes anything inside the
226 FileSet, it will be detected and the new FileSet will be used. This is
227 particularly important when doing an incremental update. If the user deletes a
228 file or adds a file, we need to ensure that a Full backup is done prior to the
229 next incremental.
230
231
232 \LTXtable{\linewidth}{t-dev-dbjobmedia}
233
234 The \bilink{\btable{JobMedia} table}{blb:dev:jobmedia:table:layout} contains one entry at the following: start of
235 the job, start of each new tape file, start of each new tape, end of the
236 job. Since by default, a new tape file is written every 2GB, in general,
237 you will have more than 2 JobMedia records per Job. The number can be
238 varied by changing the "Maximum File Size" specified in the Device
239 resource. This record allows \bacula{} to efficiently position close to
240 (within 2GB) any given file in a backup. For restoring a full Job,
241 these records are not very important, but if you want to retrieve
242 a single file that was written near the end of a 100GB backup, the
243 JobMedia records can speed it up by orders of magnitude by permitting
244 forward spacing files and blocks rather than reading the whole 100GB
245 backup.
246
247
248
249
250 \LTXtable{\linewidth}{t-dev-dbmedia}
251
252 The \bilink{\btable{Volume} table}{blb:dev:media:table:layout}\footnote{Internally referred to as the Media table} contains
253 one entry for each volume, that is each tape, cassette (8mm, DLT, DAT, ...),
254 or file on which information is or was backed up. There is one Volume record
255 created for each of the NumVols specified in the Pool resource record.
256
257 \LTXtable{\linewidth}{t-dev-dbpool}
258
259 The \bilink{\btable{Pool} table}{blb:dev:pool:table:layout} contains one entry for each media pool controlled by
260 \bacula{} in this database. One media record exists for each of the NumVols
261 contained in the Pool. The PoolType is a \bacula{} defined keyword. The MediaType
262 is defined by the administrator, and corresponds to the MediaType specified in
263 the Director's Storage definition record. The CurrentVol is the sequence
264 number of the Media record for the current volume.
265
266
267 \LTXtable{\linewidth}{t-dev-dbclient}
268
269 The \bilink{\btable{Client} table}{blb:dev:client:table:layout} contains one entry for each machine backed up by \bacula{}
270 in this database. Normally the Name is a fully qualified domain name.
271
272
273 \LTXtable{\linewidth}{t-dev-dbstorage}
274
275 The \bilink{\btable{Storage} table}{blb:dev:storage:table:layout} contains one entry for each Storage used.
276
277
278 \LTXtable{\linewidth}{t-dev-dbcounter}
279
280 The \bilink{\btable{Counter} table}{blb:dev:counter:table:layout} contains one entry for each permanent counter defined
281 by the user.
282
283 \LTXtable{\linewidth}{t-dev-dbjobhistory}
284
285 The \bilink{\btable{JobHisto} table}{blb:dev:jobhistory:table:layout} is the same as the Job table, but it keeps
286 long term statistics (i.e. it is not pruned with the Job).
287
288
289 \LTXtable{\linewidth}{t-dev-dblog}
290
291 The \bilink{\btable{Log} table}{blb:dev:log:table:layout} contains a log of all Job output.
292
293 \LTXtable{\linewidth}{t-dev-dblocation}
294
295 The \bilink{\btable{Location} table}{blb:dev:location:table:layout} defines where a Volume is physically.
296
297
298 \LTXtable{\linewidth}{t-dev-dblocationlog}
299
300 The \bilink{\btable{Location Log} table}{blb:dev:locationlog:table:layout} contains a log of all Job output.
301
302
303 \LTXtable{\linewidth}{t-dev-dbversion}
304
305 The \bilink{\btable{Version} table}{blb:dev:version:table:layout} defines the \bacula{} database version number. \bacula{}
306 checks this number before reading the database to ensure that it is compatible
307 with the \bacula{} binary file.
308
309
310 \LTXtable{\linewidth}{t-dev-dbbasefiles}
311
312 The \bilink{\btable{BaseFiles} table}{blb:dev:basefiles:table:layout} contains all the File references for a particular
313 JobId that point to a Base file -- i.e. they were previously saved and hence
314 were not saved in the current JobId but in BaseJobId under FileId. FileIndex
315 is the index of the file, and is used for optimization of Restore jobs to
316 prevent the need to read the FileId record when creating the in memory tree.
317 This record is not yet implemented.