Debian Codegraph @

manuals/en/developers/mediaformat.tex

raw
1 %%
2 %%
3
4 \chapter{Storage Media Output Format}
5 \label{blb:TheChapterStart9}
6 \index{Format!Storage Media Output}
7 \index{Storage Media Output Format}
8
9 \section{General}
10 \index{General}
11
12 This document describes the media format written by the Storage daemon. The
13 Storage daemon reads and writes in units of blocks. Blocks contain records.
14 Each block has a block header followed by records, and each record has a
15 record header followed by record data.
16
17 This chapter is intended to be a technical discussion of the Media Format and
18 as such is not targeted at end users but rather at developers and system
19 administrators that want or need to know more of the working details of {\bf
20 Bacula}.
21
22 \section{Definitions}
23 \index{Definitions}
24
25 \begin{bdescription}
26
27 \item [Block]
28 \index{Block}
29 A block represents the primitive unit of information that the Storage daemon
30 reads and writes to a physical device. Normally, for a tape device, it will
31 be the same as a tape block. The Storage daemon always reads and writes
32 blocks. A block consists of block header information followed by records.
33 Clients of the Storage daemon (the File daemon) normally never see blocks.
34 However, some of the Storage tools (bls, bscan, bextract, ...) may be use
35 block header information. In older Bacula tape versions, a block could
36 contain records (see record definition below) from multiple jobs. However,
37 all blocks currently written by Bacula are block level BB02, and a given
38 block contains records for only a single job. Different jobs simply have
39 their own private blocks that are intermingled with the other blocks from
40 other jobs on the Volume (previously the records were intermingled within
41 the blocks). Having only records from a single job in any give block
42 permitted moving the VolumeSessionId and VolumeSessionTime (see below) from
43 each record heading to the Block header. This has two advantages: 1. a block
44 can be quickly rejected based on the contents of the header without reading
45 all the records. 2. because there is on the average more than one record per
46 block, less data is written to the Volume for each job.
47
48 \item [Record]
49 \index{Record}
50 A record consists of a Record Header, which is managed by the Storage daemon
51 and Record Data, which is the data received from the Client. A record is the
52 primitive unit of information sent to and from the Storage daemon by the
53 Client (File daemon) programs. The details are described below.
54
55 \item [JobId]
56 \index{JobId}
57 A number assigned by the Director daemon for a particular job. This number
58 will be unique for that particular Director (Catalog). The daemons use this
59 number to keep track of individual jobs. Within the Storage daemon, the JobId
60 may not be unique if several Directors are accessing the Storage daemon
61 simultaneously.
62
63 \item [Session]
64 \index{Session}
65 A Session is a concept used in the Storage daemon corresponds one to one to a
66 Job with the exception that each session is uniquely identified within the
67 Storage daemon by a unique SessionId/SessionTime pair (see below).
68
69 \item [VolSessionId]
70 \index{VolSessionId}
71 A unique number assigned by the Storage daemon to a particular session (Job)
72 it is having with a File daemon. This number by itself is not unique to the
73 given Volume, but with the VolSessionTime, it is unique.
74
75 \item [VolSessionTime]
76 \index{VolSessionTime}
77 A unique number assigned by the Storage daemon to a particular Storage daemon
78 execution. It is actually the Unix time\_t value of when the Storage daemon
79 began execution cast to a 32 bit unsigned integer. The combination of the
80 {\bf VolSessionId} and the {\bf VolSessionTime} for a given Storage daemon is
81 guaranteed to be unique for each Job (or session).
82
83 \item [FileIndex]
84 \index{FileIndex}
85 A sequential number beginning at one assigned by the File daemon to the files
86 within a job that are sent to the Storage daemon for backup. The Storage
87 daemon ensures that this number is greater than zero and sequential. Note,
88 the Storage daemon uses negative FileIndexes to flag Session Start and End
89 Labels as well as End of Volume Labels. Thus, the combination of
90 VolSessionId, VolSessionTime, and FileIndex uniquely identifies the records
91 for a single file written to a Volume.
92
93 \item [Stream]
94 \index{Stream}
95 While writing the information for any particular file to the Volume, there
96 can be any number of distinct pieces of information about that file, e.g. the
97 attributes, the file data, ... The Stream indicates what piece of data it
98 is, and it is an arbitrary number assigned by the File daemon to the parts
99 (Unix attributes, Win32 attributes, data, compressed data,\ ...) of a file
100 that are sent to the Storage daemon. The Storage daemon has no knowledge of
101 the details of a Stream; it simply represents a numbered stream of bytes. The
102 data for a given stream may be passed to the Storage daemon in single record,
103 or in multiple records.
104
105 \item [Block Header]
106 \index{Block Header}
107 A block header consists of a block identification (``BB02''), a block length
108 in bytes (typically 64,512) a checksum, and sequential block number. Each
109 block starts with a Block Header and is followed by Records. Current block
110 headers also contain the VolSessionId and VolSessionTime for the records
111 written to that block.
112
113 \item [Record Header]
114 \index{Record Header}
115 A record header contains the Volume Session Id, the Volume Session Time, the
116 FileIndex, the Stream, and the size of the data record which follows. The
117 Record Header is always immediately followed by a Data Record if the size
118 given in the Header is greater than zero. Note, for Block headers of level
119 BB02 (version 1.27 and later), the Record header as written to tape does not
120 contain the Volume Session Id and the Volume Session Time as these two
121 fields are stored in the BB02 Block header. The in-memory record header does
122 have those fields for convenience.
123
124 \item [Data Record]
125 \index{Data Record}
126 A data record consists of a binary stream of bytes and is always preceded by
127 a Record Header. The details of the meaning of the binary stream of bytes are
128 unknown to the Storage daemon, but the Client programs (File daemon) defines
129 and thus knows the details of each record type.
130
131 \item [Volume Label]
132 \index{Volume Label}
133 A label placed by the Storage daemon at the beginning of each storage volume.
134 It contains general information about the volume. It is written in Record
135 format. The Storage daemon manages Volume Labels, and if the client wants, he
136 may also read them.
137
138 \item [Begin Session Label]
139 \index{Begin Session Label}
140 The Begin Session Label is a special record placed by the Storage daemon on
141 the storage medium as the first record of an append session job with a File
142 daemon. This record is useful for finding the beginning of a particular
143 session (Job), since no records with the same VolSessionId and VolSessionTime
144 will precede this record. This record is not normally visible outside of the
145 Storage daemon. The Begin Session Label is similar to the Volume Label except
146 that it contains additional information pertaining to the Session.
147
148 \item [End Session Label]
149 \index{End Session Label}
150 The End Session Label is a special record placed by the Storage daemon on the
151 storage medium as the last record of an append session job with a File
152 daemon. The End Session Record is distinguished by a FileIndex with a value
153 of minus two (-2). This record is useful for detecting the end of a
154 particular session since no records with the same VolSessionId and
155 VolSessionTime will follow this record. This record is not normally visible
156 outside of the Storage daemon. The End Session Label is similar to the Volume
157 Label except that it contains additional information pertaining to the
158 Session.
159 \end{bdescription}
160
161 \section{Storage Daemon File Output Format}
162 \index{Format!Storage Daemon File Output}
163 \index{Storage Daemon File Output Format}
164
165 The file storage and tape storage formats are identical except that tape
166 records are by default blocked into blocks of 64,512 bytes, except for the
167 last block, which is the actual number of bytes written rounded up to a
168 multiple of 1024 whereas the last record of file storage is not rounded up.
169 The default block size of 64,512 bytes may be overridden by the user (some
170 older tape drives only support block sizes of 32K). Each Session written to
171 tape is terminated with an End of File mark (this will be removed later).
172 Sessions written to file are simply appended to the end of the file.
173
174 \section{Overall Format}
175 \index{Format!Overall}
176 \index{Overall Format}
177
178 A Bacula output file consists of Blocks of data. Each block contains a block
179 header followed by records. Each record consists of a record header followed
180 by the record data. The first record on a tape will always be the Volume Label
181 Record.
182
183 No Record Header will be split across Bacula blocks. However, Record Data may
184 be split across any number of Bacula blocks. Obviously this will not be the
185 case for the Volume Label which will always be smaller than the Bacula Block
186 size.
187
188 To simplify reading tapes, the Start of Session (SOS) and End of Session (EOS)
189 records are never split across blocks. If this is about to happen, Bacula will
190 write a short block before writing the session record (actually, the SOS
191 record should always be the first record in a block, excepting perhaps the
192 Volume label).
193
194 Due to hardware limitations, the last block written to the tape may not be
195 fully written. If your drive permits backspace record, Bacula will backup over
196 the last record written on the tape, re-read it and verify that it was
197 correctly written.
198
199 When a new tape is mounted Bacula will write the full contents of the
200 partially written block to the new tape ensuring that there is no loss of
201 data. When reading a tape, Bacula will discard any block that is not totally
202 written, thus ensuring that there is no duplication of data. In addition,
203 since Bacula blocks are sequentially numbered within a Job, it is easy to
204 ensure that no block is missing or duplicated.
205
206 \section{Serialization}
207 \index{Serialization}
208
209 All Block Headers, Record Headers, and Label Records are written using
210 Bacula's serialization routines. These routines guarantee that the data is
211 written to the output volume in a machine independent format.
212
213 \section{Block Header}
214 \index{Header!Block}
215 \index{Block Header}
216
217 The format of the Block Header (version 1.27 and later) is:
218
219 \footnotesize
220 \begin{bVerbatim}
221 uint32_t CheckSum; /* Block check sum */
222 uint32_t BlockSize; /* Block byte size including the header */
223 uint32_t BlockNumber; /* Block number */
224 char ID[4] = "BB02"; /* Identification and block level */
225 uint32_t VolSessionId; /* Session Id for Job */
226 uint32_t VolSessionTime; /* Session Time for Job */
227 \end{bVerbatim}
228 \normalsize
229
230 The Block header is a fixed length and fixed format and is followed by Record
231 Headers and Record Data. The CheckSum field is a 32 bit checksum of the block
232 data and the block header but not including the CheckSum field. The Block
233 Header is always immediately followed by a Record Header. If the tape is
234 damaged, a Bacula utility will be able to recover as much information as
235 possible from the tape by recovering blocks which are valid. The Block header
236 is written using the Bacula serialization routines and thus is guaranteed to
237 be in machine independent format. See below for version 2 of the block header.
238
239
240 \section{Record Header}
241 \index{Header!Record}
242 \index{Record Header}
243
244 Each binary data record is preceded by a Record Header. The Record Header is
245 fixed length and fixed format, whereas the binary data record is of variable
246 length. The Record Header is written using the Bacula serialization routines
247 and thus is guaranteed to be in machine independent format.
248
249 The format of the Record Header (version 1.27 or later) is:
250
251 \footnotesize
252 \begin{bVerbatim}
253 int32_t FileIndex; /* File index supplied by File daemon */
254 int32_t Stream; /* Stream number supplied by File daemon */
255 uint32_t DataSize; /* size of following data record in bytes */
256 \end{bVerbatim}
257 \normalsize
258
259 This record is followed by the binary Stream data of DataSize bytes, followed
260 by another Record Header record and the binary stream data. For the definitive
261 definition of this record, see record.h in the src/stored directory.
262
263 Additional notes on the above:
264
265 \begin{bdescription}
266
267 \item [The {\bf VolSessionId} ]
268 \index{VolSessionId}
269 is a unique sequential number that is assigned by the Storage Daemon to a
270 particular Job. This number is sequential since the start of execution of the
271 daemon.
272
273 \item [The {\bf VolSessionTime} ]
274 \index{VolSessionTime}
275 is the time/date that the current execution of the Storage Daemon started. It
276 assures that the combination of VolSessionId and VolSessionTime is unique for
277 every jobs written to the tape, even if there was a machine crash between two
278 writes.
279
280 \item [The {\bf FileIndex} ]
281 \index{FileIndex}
282 is a sequential file number within a job. The Storage daemon requires this
283 index to be greater than zero and sequential. Note, however, that the File
284 daemon may send multiple Streams for the same FileIndex. In addition, the
285 Storage daemon uses negative FileIndices to hold the Begin Session Label, the
286 End Session Label, and the End of Volume Label.
287
288 \item [The {\bf Stream} ]
289 \index{Stream}
290 is defined by the File daemon and is used to identify separate parts of the
291 data saved for each file (Unix attributes, Win32 attributes, file data,
292 compressed file data, sparse file data, ...). The Storage Daemon has no idea
293 of what a Stream is or what it contains except that the Stream is required to
294 be a positive integer. Negative Stream numbers are used internally by the
295 Storage daemon to indicate that the record is a continuation of the previous
296 record (the previous record would not entirely fit in the block).
297
298 For Start Session and End Session Labels (where the FileIndex is negative),
299 the Storage daemon uses the Stream field to contain the JobId. The current
300 stream definitions are:
301
302 \footnotesize
303 \begin{bVerbatim}
304 #define STREAM_UNIX_ATTRIBUTES 1 /* Generic Unix attributes */
305 #define STREAM_FILE_DATA 2 /* Standard uncompressed data */
306 #define STREAM_MD5_SIGNATURE 3 /* MD5 signature for the file */
307 #define STREAM_GZIP_DATA 4 /* GZip compressed file data */
308 /* Extended Unix attributes with Win32 Extended data. Deprecated. */
309 #define STREAM_UNIX_ATTRIBUTES_EX 5 /* Extended Unix attr for Win32 EX */
310 #define STREAM_SPARSE_DATA 6 /* Sparse data stream */
311 #define STREAM_SPARSE_GZIP_DATA 7
312 #define STREAM_PROGRAM_NAMES 8 /* program names for program data */
313 #define STREAM_PROGRAM_DATA 9 /* Data needing program */
314 #define STREAM_SHA1_SIGNATURE 10 /* SHA1 signature for the file */
315 #define STREAM_WIN32_DATA 11 /* Win32 BackupRead data */
316 #define STREAM_WIN32_GZIP_DATA 12 /* Gzipped Win32 BackupRead data */
317 #define STREAM_MACOS_FORK_DATA 13 /* Mac resource fork */
318 #define STREAM_HFSPLUS_ATTRIBUTES 14 /* Mac OS extra attributes */
319 #define STREAM_UNIX_ATTRIBUTES_ACCESS_ACL 15 /* Standard ACL attributes on UNIX */
320 #define STREAM_UNIX_ATTRIBUTES_DEFAULT_ACL 16 /* Default ACL attributes on UNIX */
321 \end{bVerbatim}
322 \normalsize
323
324 \item [The {\bf DataSize} ]
325 \index{DataSize}
326 is the size in bytes of the binary data record that follows the Session
327 Record header. The Storage Daemon has no idea of the actual contents of the
328 binary data record. For standard Unix files, the data record typically
329 contains the file attributes or the file data. For a sparse file the first
330 64 bits of the file data contains the storage address for the data block.
331 \end{bdescription}
332
333 The Record Header is never split across two blocks. If there is not enough
334 room in a block for the full Record Header, the block is padded to the end
335 with zeros and the Record Header begins in the next block. The data record, on
336 the other hand, may be split across multiple blocks and even multiple physical
337 volumes. When a data record is split, the second (and possibly subsequent)
338 piece of the data is preceded by a new Record Header. Thus each piece of data
339 is always immediately preceded by a Record Header. When reading a record, if
340 Bacula finds only part of the data in the first record, it will automatically
341 read the next record and concatenate the data record to form a full data
342 record.
343
344 \section{Version BB02 Block Header}
345 \index{Version BB02 Block Header}
346 \index{Header!Version BB02 Block}
347
348 Each session or Job has its own private block. As a consequence, the SessionId
349 and SessionTime are written once in each Block Header and not in the Record
350 Header. So, the second and current version of the Block Header BB02 is:
351
352 \footnotesize
353 \begin{bVerbatim}
354 uint32_t CheckSum; /* Block check sum */
355 uint32_t BlockSize; /* Block byte size including the header */
356 uint32_t BlockNumber; /* Block number */
357 char ID[4] = "BB02"; /* Identification and block level */
358 uint32_t VolSessionId; /* Applies to all records */
359 uint32_t VolSessionTime; /* contained in this block */
360 \end{bVerbatim}
361 \normalsize
362
363 As with the previous version, the BB02 Block header is a fixed length and
364 fixed format and is followed by Record Headers and Record Data. The CheckSum
365 field is a 32 bit CRC checksum of the block data and the block header but not
366 including the CheckSum field. The Block Header is always immediately followed
367 by a Record Header. If the tape is damaged, a Bacula utility will be able to
368 recover as much information as possible from the tape by recovering blocks
369 which are valid. The Block header is written using the Bacula serialization
370 routines and thus is guaranteed to be in machine independent format.
371
372 \section{Version 2 Record Header}
373 \index{Version 2 Record Header}
374 \index{Header!Version 2 Record}
375
376 Version 2 Record Header is written to the medium when using Version BB02 Block
377 Headers. The memory representation of the record is identical to the old BB01
378 Record Header, but on the storage medium, the first two fields, namely
379 VolSessionId and VolSessionTime are not written. The Block Header is filled
380 with these values when the First user record is written (i.e. non label
381 record) so that when the block is written, it will have the current and unique
382 VolSessionId and VolSessionTime. On reading each record from the Block, the
383 VolSessionId and VolSessionTime is filled in the Record Header from the Block
384 Header.
385
386 \section{Volume Label Format}
387 \index{Volume Label Format}
388 \index{Format!Volume Label}
389
390 Tape volume labels are created by the Storage daemon in response to a {\bf
391 label} command given to the Console program, or alternatively by the {\bf
392 btape} program. created. Each volume is labeled with the following information
393 using the Bacula serialization routines, which guarantee machine byte order
394 independence.
395
396 For Bacula versions 1.27 and later, the Volume Label Format is:
397
398 \footnotesize
399 \begin{bVerbatim}
400 char Id[32]; /* Bacula 1.0 Immortal\n */
401 uint32_t VerNum; /* Label version number */
402 /* VerNum 11 and greater Bacula 1.27 and later */
403 btime_t label_btime; /* Time/date tape labeled */
404 btime_t write_btime; /* Time/date tape first written */
405 /* The following are 0 in VerNum 11 and greater */
406 float64_t write_date; /* Date this label written */
407 float64_t write_time; /* Time this label written */
408 char VolName[128]; /* Volume name */
409 char PrevVolName[128]; /* Previous Volume Name */
410 char PoolName[128]; /* Pool name */
411 char PoolType[128]; /* Pool type */
412 char MediaType[128]; /* Type of this media */
413 char HostName[128]; /* Host name of writing computer */
414 char LabelProg[32]; /* Label program name */
415 char ProgVersion[32]; /* Program version */
416 char ProgDate[32]; /* Program build date/time */
417 \end{bVerbatim}
418 \normalsize
419
420 Note, the LabelType (Volume Label, Volume PreLabel, Session Start Label, ...)
421 is stored in the record FileIndex field of the Record Header and does not
422 appear in the data part of the record.
423
424 \section{Session Label}
425 \index{Label!Session}
426 \index{Session Label}
427
428 The Session Label is written at the beginning and end of each session as well
429 as the last record on the physical medium. It has the following binary format:
430
431
432 \footnotesize
433 \begin{bVerbatim}
434 char Id[32]; /* Bacula Immortal ... */
435 uint32_t VerNum; /* Label version number */
436 uint32_t JobId; /* Job id */
437 uint32_t VolumeIndex; /* sequence no of vol */
438 /* Prior to VerNum 11 */
439 float64_t write_date; /* Date this label written */
440 /* VerNum 11 and greater */
441 btime_t write_btime; /* time/date record written */
442 /* The following is zero VerNum 11 and greater */
443 float64_t write_time; /* Time this label written */
444 char PoolName[128]; /* Pool name */
445 char PoolType[128]; /* Pool type */
446 char JobName[128]; /* base Job name */
447 char ClientName[128];
448 /* Added in VerNum 10 */
449 char Job[128]; /* Unique Job name */
450 char FileSetName[128]; /* FileSet name */
451 uint32_t JobType;
452 uint32_t JobLevel;
453 \end{bVerbatim}
454 \normalsize
455
456 In addition, the EOS label contains:
457
458 \footnotesize
459 \begin{bVerbatim}
460 /* The remainder are part of EOS label only */
461 uint32_t JobFiles;
462 uint64_t JobBytes;
463 uint32_t start_block;
464 uint32_t end_block;
465 uint32_t start_file;
466 uint32_t end_file;
467 uint32_t JobErrors;
468 \end{bVerbatim}
469 \normalsize
470
471 In addition, for VerNum greater than 10, the EOS label contains (in addition
472 to the above):
473
474 \footnotesize
475 \begin{bVerbatim}
476 uint32_t JobStatus /* Job termination code */
477 \end{bVerbatim}
478 \normalsize
479
480 : Note, the LabelType (Volume Label, Volume PreLabel, Session Start Label,
481 ...) is stored in the record FileIndex field and does not appear in the data
482 part of the record. Also, the Stream field of the Record Header contains the
483 JobId. This permits quick filtering without actually reading all the session
484 data in many cases.
485
486 \section{Overall Storage Format}
487 \index{Format!Overall Storage}
488 \index{Overall Storage Format}
489
490 \footnotesize
491 \begin{bVerbatim}
492 Current Bacula Tape Format
493 6 June 2001
494 Version BB02 added 28 September 2002
495 Version BB01 is the old deprecated format.
496 A Bacula tape is composed of tape Blocks. Each block
497 has a Block header followed by the block data. Block
498 Data consists of Records. Records consist of Record
499 Headers followed by Record Data.
500 :=======================================================:
501 | |
502 | Block Header (24 bytes) |
503 | |
504 |-------------------------------------------------------|
505 | |
506 | Record Header (12 bytes) |
507 | |
508 |-------------------------------------------------------|
509 | |
510 | Record Data |
511 | |
512 |-------------------------------------------------------|
513 | |
514 | Record Header (12 bytes) |
515 | |
516 |-------------------------------------------------------|
517 | |
518 | ... |
519 Block Header: the first item in each block. The format is
520 shown below.
521 Partial Data block: occurs if the data from a previous
522 block spills over to this block (the normal case except
523 for the first block on a tape). However, this partial
524 data block is always preceded by a record header.
525 Record Header: identifies the Volume Session, the Stream
526 and the following Record Data size. See below for format.
527 Record data: arbitrary binary data.
528 Block Header Format BB02
529 :=======================================================:
530 | CheckSum (uint32_t) |
531 |-------------------------------------------------------|
532 | BlockSize (uint32_t) |
533 |-------------------------------------------------------|
534 | BlockNumber (uint32_t) |
535 |-------------------------------------------------------|
536 | "BB02" (char [4]) |
537 |-------------------------------------------------------|
538 | VolSessionId (uint32_t) |
539 |-------------------------------------------------------|
540 | VolSessionTime (uint32_t) |
541 :=======================================================:
542 BBO2: Serves to identify the block as a
543 Bacula block and also servers as a block format identifier
544 should we ever need to change the format.
545 BlockSize: is the size in bytes of the block. When reading
546 back a block, if the BlockSize does not agree with the
547 actual size read, Bacula discards the block.
548 CheckSum: a checksum for the Block.
549 BlockNumber: is the sequential block number on the tape.
550 VolSessionId: a unique sequential number that is assigned
551 by the Storage Daemon to a particular Job.
552 This number is sequential since the start
553 of execution of the daemon.
554 VolSessionTime: the time/date that the current execution
555 of the Storage Daemon started. It assures
556 that the combination of VolSessionId and
557 VolSessionTime is unique for all jobs
558 written to the tape, even if there was a
559 machine crash between two writes.
560 Record Header Format BB02
561 :=======================================================:
562 | FileIndex (int32_t) |
563 |-------------------------------------------------------|
564 | Stream (int32_t) |
565 |-------------------------------------------------------|
566 | DataSize (uint32_t) |
567 :=======================================================:
568 FileIndex: a sequential file number within a job. The
569 Storage daemon enforces this index to be
570 greater than zero and sequential. Note,
571 however, that the File daemon may send
572 multiple Streams for the same FileIndex.
573 The Storage Daemon uses negative FileIndices
574 to identify Session Start and End labels
575 as well as the End of Volume labels.
576 Stream: defined by the File daemon and is intended to be
577 used to identify separate parts of the data
578 saved for each file (attributes, file data,
579 ...). The Storage Daemon has no idea of
580 what a Stream is or what it contains.
581 DataSize: the size in bytes of the binary data record
582 that follows the Session Record header.
583 The Storage Daemon has no idea of the
584 actual contents of the binary data record.
585 For standard Unix files, the data record
586 typically contains the file attributes or
587 the file data. For a sparse file
588 the first 64 bits of the data contains
589 the storage address for the data block.
590 Volume Label
591 :=======================================================:
592 | Id (32 bytes) |
593 |-------------------------------------------------------|
594 | VerNum (uint32_t) |
595 |-------------------------------------------------------|
596 | label_date (float64_t) |
597 | label_btime (btime_t VerNum 11 |
598 |-------------------------------------------------------|
599 | label_time (float64_t) |
600 | write_btime (btime_t VerNum 11 |
601 |-------------------------------------------------------|
602 | write_date (float64_t) |
603 | 0 (float64_t) VerNum 11 |
604 |-------------------------------------------------------|
605 | write_time (float64_t) |
606 | 0 (float64_t) VerNum 11 |
607 |-------------------------------------------------------|
608 | VolName (128 bytes) |
609 |-------------------------------------------------------|
610 | PrevVolName (128 bytes) |
611 |-------------------------------------------------------|
612 | PoolName (128 bytes) |
613 |-------------------------------------------------------|
614 | PoolType (128 bytes) |
615 |-------------------------------------------------------|
616 | MediaType (128 bytes) |
617 |-------------------------------------------------------|
618 | HostName (128 bytes) |
619 |-------------------------------------------------------|
620 | LabelProg (32 bytes) |
621 |-------------------------------------------------------|
622 | ProgVersion (32 bytes) |
623 |-------------------------------------------------------|
624 | ProgDate (32 bytes) |
625 |-------------------------------------------------------|
626 :=======================================================:
627
628 Id: 32 byte Bacula identifier "Bacula 1.0 immortal\n"
629 (old version also recognized:)
630 Id: 32 byte Bacula identifier "Bacula 0.9 mortal\n"
631 LabelType (Saved in the FileIndex of the Header record).
632 PRE_LABEL -1 Volume label on unwritten tape
633 VOL_LABEL -2 Volume label after tape written
634 EOM_LABEL -3 Label at EOM (not currently implemented)
635 SOS_LABEL -4 Start of Session label (format given below)
636 EOS_LABEL -5 End of Session label (format given below)
637 VerNum: 11
638 label_date: Julian day tape labeled
639 label_time: Julian time tape labeled
640 write_date: Julian date tape first used (data written)
641 write_time: Julian time tape first used (data written)
642 VolName: "Physical" Volume name
643 PrevVolName: The VolName of the previous tape (if this tape is
644 a continuation of the previous one).
645 PoolName: Pool Name
646 PoolType: Pool Type
647 MediaType: Media Type
648 HostName: Name of host that is first writing the tape
649 LabelProg: Name of the program that labeled the tape
650 ProgVersion: Version of the label program
651 ProgDate: Date Label program built
652 Session Label
653 :=======================================================:
654 | Id (32 bytes) |
655 |-------------------------------------------------------|
656 | VerNum (uint32_t) |
657 |-------------------------------------------------------|
658 | JobId (uint32_t) |
659 |-------------------------------------------------------|
660 | write_btime (btime_t) VerNum 11 |
661 |-------------------------------------------------------|
662 | 0 (float64_t) VerNum 11 |
663 |-------------------------------------------------------|
664 | PoolName (128 bytes) |
665 |-------------------------------------------------------|
666 | PoolType (128 bytes) |
667 |-------------------------------------------------------|
668 | JobName (128 bytes) |
669 |-------------------------------------------------------|
670 | ClientName (128 bytes) |
671 |-------------------------------------------------------|
672 | Job (128 bytes) |
673 |-------------------------------------------------------|
674 | FileSetName (128 bytes) |
675 |-------------------------------------------------------|
676 | JobType (uint32_t) |
677 |-------------------------------------------------------|
678 | JobLevel (uint32_t) |
679 |-------------------------------------------------------|
680 | FileSetMD5 (50 bytes) VerNum 11 |
681 |-------------------------------------------------------|
682 Additional fields in End Of Session Label
683 |-------------------------------------------------------|
684 | JobFiles (uint32_t) |
685 |-------------------------------------------------------|
686 | JobBytes (uint32_t) |
687 |-------------------------------------------------------|
688 | start_block (uint32_t) |
689 |-------------------------------------------------------|
690 | end_block (uint32_t) |
691 |-------------------------------------------------------|
692 | start_file (uint32_t) |
693 |-------------------------------------------------------|
694 | end_file (uint32_t) |
695 |-------------------------------------------------------|
696 | JobErrors (uint32_t) |
697 |-------------------------------------------------------|
698 | JobStatus (uint32_t) VerNum 11 |
699 :=======================================================:
700 * => fields deprecated
701 Id: 32 byte Bacula Identifier "Bacula 1.0 immortal\n"
702 LabelType (in FileIndex field of Header):
703 EOM_LABEL -3 Label at EOM
704 SOS_LABEL -4 Start of Session label
705 EOS_LABEL -5 End of Session label
706 VerNum: 11
707 JobId: JobId
708 write_btime: Bacula time/date this tape record written
709 write_date: Julian date tape this record written - deprecated
710 write_time: Julian time tape this record written - deprecated.
711 PoolName: Pool Name
712 PoolType: Pool Type
713 MediaType: Media Type
714 ClientName: Name of File daemon or Client writing this session
715 Not used for EOM_LABEL.
716 \end{bVerbatim}
717 \normalsize
718
719 \section{Unix File Attributes}
720 \index{Unix File Attributes}
721 \index{Attributes!Unix File}
722
723 The Unix File Attributes packet consists of the following:
724
725 \lt{}File-Index\gt{} \lt{}Type\gt{}
726 \lt{}Filename\gt{}@\lt{}File-Attributes\gt{}@\lt{}Link\gt{}
727 @\lt{}Extended-Attributes@\gt{} where
728
729 \begin{bdescription}
730
731 \item [@]
732 represents a byte containing a binary zero.
733
734 \item [FileIndex]
735 \index{FileIndex}
736 is the sequential file index starting from one assigned by the File daemon.
737
738 \item [Type]
739 \index{Type}
740 is one of the following:
741
742 \footnotesize
743 \begin{bVerbatim}
744 #define FT_LNKSAVED 1 /* hard link to file already saved */
745 #define FT_REGE 2 /* Regular file but empty */
746 #define FT_REG 3 /* Regular file */
747 #define FT_LNK 4 /* Soft Link */
748 #define FT_DIR 5 /* Directory */
749 #define FT_SPEC 6 /* Special file -- chr, blk, fifo, sock */
750 #define FT_NOACCESS 7 /* Not able to access */
751 #define FT_NOFOLLOW 8 /* Could not follow link */
752 #define FT_NOSTAT 9 /* Could not stat file */
753 #define FT_NOCHG 10 /* Incremental option, file not changed */
754 #define FT_DIRNOCHG 11 /* Incremental option, directory not changed */
755 #define FT_ISARCH 12 /* Trying to save archive file */
756 #define FT_NORECURSE 13 /* No recursion into directory */
757 #define FT_NOFSCHG 14 /* Different file system, prohibited */
758 #define FT_NOOPEN 15 /* Could not open directory */
759 #define FT_RAW 16 /* Raw block device */
760 #define FT_FIFO 17 /* Raw fifo device */
761 \end{bVerbatim}
762 \normalsize
763
764 \item [Filename]
765 \index{Filename}
766 is the fully qualified filename.
767
768 \item [File-Attributes]
769 \index{File-Attributes}
770 consists of the 13 fields of the stat() buffer in ASCII base64 format
771 separated by spaces. These fields and their meanings are shown below. This
772 stat() packet is in Unix format, and MUST be provided (constructed) for ALL
773 systems.
774
775 \item [Link]
776 \index{Link}
777 when the FT code is FT\_LNK or FT\_LNKSAVED, the item in question is a Unix
778 link, and this field contains the fully qualified link name. When the FT code
779 is not FT\_LNK or FT\_LNKSAVED, this field is null.
780
781 \item [Extended-Attributes]
782 \index{Extended-Attributes}
783 The exact format of this field is operating system dependent. It contains
784 additional or extended attributes of a system dependent nature. Currently,
785 this field is used only on WIN32 systems where it contains a ASCII base64
786 representation of the WIN32\_FILE\_ATTRIBUTE\_DATA structure as defined by
787 Windows. The fields in the base64 representation of this structure are like
788 the File-Attributes separated by spaces.
789 \end{bdescription}
790
791 The File-attributes consist of the following:
792
793 \LTXtable{\linewidth}{t-dev-fileattributes}
794
795 \section{Old Depreciated Tape Format}
796 \index{Old Depreciated Tape Format}
797 \index{Format!Old Depreciated Tape}
798
799 The format of the Block Header (version 1.26 and earlier) is:
800
801 \footnotesize
802 \begin{bVerbatim}
803 uint32_t CheckSum; /* Block check sum */
804 uint32_t BlockSize; /* Block byte size including the header */
805 uint32_t BlockNumber; /* Block number */
806 char ID[4] = "BB01"; /* Identification and block level */
807 \end{bVerbatim}
808 \normalsize
809
810 The format of the Record Header (version 1.26 or earlier) is:
811
812 \footnotesize
813 \begin{bVerbatim}
814 uint32_t VolSessionId; /* Unique ID for this session */
815 uint32_t VolSessionTime; /* Start time/date of session */
816 int32_t FileIndex; /* File index supplied by File daemon */
817 int32_t Stream; /* Stream number supplied by File daemon */
818 uint32_t DataSize; /* size of following data record in bytes */
819 \end{bVerbatim}
820 \normalsize
821
822 \footnotesize
823 \begin{bVerbatim}
824 Current Bacula Tape Format
825 6 June 2001
826 Version BB01 is the old deprecated format.
827 A Bacula tape is composed of tape Blocks. Each block
828 has a Block header followed by the block data. Block
829 Data consists of Records. Records consist of Record
830 Headers followed by Record Data.
831 :=======================================================:
832 | |
833 | Block Header |
834 | (16 bytes version BB01) |
835 |-------------------------------------------------------|
836 | |
837 | Record Header |
838 | (20 bytes version BB01) |
839 |-------------------------------------------------------|
840 | |
841 | Record Data |
842 | |
843 |-------------------------------------------------------|
844 | |
845 | Record Header |
846 | (20 bytes version BB01) |
847 |-------------------------------------------------------|
848 | |
849 | ... |
850 Block Header: the first item in each block. The format is
851 shown below.
852 Partial Data block: occurs if the data from a previous
853 block spills over to this block (the normal case except
854 for the first block on a tape). However, this partial
855 data block is always preceded by a record header.
856 Record Header: identifies the Volume Session, the Stream
857 and the following Record Data size. See below for format.
858 Record data: arbitrary binary data.
859 Block Header Format BB01 (deprecated)
860 :=======================================================:
861 | CheckSum (uint32_t) |
862 |-------------------------------------------------------|
863 | BlockSize (uint32_t) |
864 |-------------------------------------------------------|
865 | BlockNumber (uint32_t) |
866 |-------------------------------------------------------|
867 | "BB01" (char [4]) |
868 :=======================================================:
869 BBO1: Serves to identify the block as a
870 Bacula block and also servers as a block format identifier
871 should we ever need to change the format.
872 BlockSize: is the size in bytes of the block. When reading
873 back a block, if the BlockSize does not agree with the
874 actual size read, Bacula discards the block.
875 CheckSum: a checksum for the Block.
876 BlockNumber: is the sequential block number on the tape.
877 VolSessionId: a unique sequential number that is assigned
878 by the Storage Daemon to a particular Job.
879 This number is sequential since the start
880 of execution of the daemon.
881 VolSessionTime: the time/date that the current execution
882 of the Storage Daemon started. It assures
883 that the combination of VolSessionId and
884 VolSessionTime is unique for all jobs
885 written to the tape, even if there was a
886 machine crash between two writes.
887 Record Header Format BB01 (deprecated)
888 :=======================================================:
889 | VolSessionId (uint32_t) |
890 |-------------------------------------------------------|
891 | VolSessionTime (uint32_t) |
892 |-------------------------------------------------------|
893 | FileIndex (int32_t) |
894 |-------------------------------------------------------|
895 | Stream (int32_t) |
896 |-------------------------------------------------------|
897 | DataSize (uint32_t) |
898 :=======================================================:
899 VolSessionId: a unique sequential number that is assigned
900 by the Storage Daemon to a particular Job.
901 This number is sequential since the start
902 of execution of the daemon.
903 VolSessionTime: the time/date that the current execution
904 of the Storage Daemon started. It assures
905 that the combination of VolSessionId and
906 VolSessionTime is unique for all jobs
907 written to the tape, even if there was a
908 machine crash between two writes.
909 FileIndex: a sequential file number within a job. The
910 Storage daemon enforces this index to be
911 greater than zero and sequential. Note,
912 however, that the File daemon may send
913 multiple Streams for the same FileIndex.
914 The Storage Daemon uses negative FileIndices
915 to identify Session Start and End labels
916 as well as the End of Volume labels.
917 Stream: defined by the File daemon and is intended to be
918 used to identify separate parts of the data
919 saved for each file (attributes, file data,
920 ...). The Storage Daemon has no idea of
921 what a Stream is or what it contains.
922 DataSize: the size in bytes of the binary data record
923 that follows the Session Record header.
924 The Storage Daemon has no idea of the
925 actual contents of the binary data record.
926 For standard Unix files, the data record
927 typically contains the file attributes or
928 the file data. For a sparse file
929 the first 64 bits of the data contains
930 the storage address for the data block.
931 Volume Label
932 :=======================================================:
933 | Id (32 bytes) |
934 |-------------------------------------------------------|
935 | VerNum (uint32_t) |
936 |-------------------------------------------------------|
937 | label_date (float64_t) |
938 |-------------------------------------------------------|
939 | label_time (float64_t) |
940 |-------------------------------------------------------|
941 | write_date (float64_t) |
942 |-------------------------------------------------------|
943 | write_time (float64_t) |
944 |-------------------------------------------------------|
945 | VolName (128 bytes) |
946 |-------------------------------------------------------|
947 | PrevVolName (128 bytes) |
948 |-------------------------------------------------------|
949 | PoolName (128 bytes) |
950 |-------------------------------------------------------|
951 | PoolType (128 bytes) |
952 |-------------------------------------------------------|
953 | MediaType (128 bytes) |
954 |-------------------------------------------------------|
955 | HostName (128 bytes) |
956 |-------------------------------------------------------|
957 | LabelProg (32 bytes) |
958 |-------------------------------------------------------|
959 | ProgVersion (32 bytes) |
960 |-------------------------------------------------------|
961 | ProgDate (32 bytes) |
962 |-------------------------------------------------------|
963 :=======================================================:
964
965 Id: 32 byte Bacula identifier "Bacula 1.0 immortal\n"
966 (old version also recognized:)
967 Id: 32 byte Bacula identifier "Bacula 0.9 mortal\n"
968 LabelType (Saved in the FileIndex of the Header record).
969 PRE_LABEL -1 Volume label on unwritten tape
970 VOL_LABEL -2 Volume label after tape written
971 EOM_LABEL -3 Label at EOM (not currently implemented)
972 SOS_LABEL -4 Start of Session label (format given below)
973 EOS_LABEL -5 End of Session label (format given below)
974 label_date: Julian day tape labeled
975 label_time: Julian time tape labeled
976 write_date: Julian date tape first used (data written)
977 write_time: Julian time tape first used (data written)
978 VolName: "Physical" Volume name
979 PrevVolName: The VolName of the previous tape (if this tape is
980 a continuation of the previous one).
981 PoolName: Pool Name
982 PoolType: Pool Type
983 MediaType: Media Type
984 HostName: Name of host that is first writing the tape
985 LabelProg: Name of the program that labeled the tape
986 ProgVersion: Version of the label program
987 ProgDate: Date Label program built
988 Session Label
989 :=======================================================:
990 | Id (32 bytes) |
991 |-------------------------------------------------------|
992 | VerNum (uint32_t) |
993 |-------------------------------------------------------|
994 | JobId (uint32_t) |
995 |-------------------------------------------------------|
996 | *write_date (float64_t) VerNum 10 |
997 |-------------------------------------------------------|
998 | *write_time (float64_t) VerNum 10 |
999 |-------------------------------------------------------|
1000 | PoolName (128 bytes) |
1001 |-------------------------------------------------------|
1002 | PoolType (128 bytes) |
1003 |-------------------------------------------------------|
1004 | JobName (128 bytes) |
1005 |-------------------------------------------------------|
1006 | ClientName (128 bytes) |
1007 |-------------------------------------------------------|
1008 | Job (128 bytes) |
1009 |-------------------------------------------------------|
1010 | FileSetName (128 bytes) |
1011 |-------------------------------------------------------|
1012 | JobType (uint32_t) |
1013 |-------------------------------------------------------|
1014 | JobLevel (uint32_t) |
1015 |-------------------------------------------------------|
1016 | FileSetMD5 (50 bytes) VerNum 11 |
1017 |-------------------------------------------------------|
1018 Additional fields in End Of Session Label
1019 |-------------------------------------------------------|
1020 | JobFiles (uint32_t) |
1021 |-------------------------------------------------------|
1022 | JobBytes (uint32_t) |
1023 |-------------------------------------------------------|
1024 | start_block (uint32_t) |
1025 |-------------------------------------------------------|
1026 | end_block (uint32_t) |
1027 |-------------------------------------------------------|
1028 | start_file (uint32_t) |
1029 |-------------------------------------------------------|
1030 | end_file (uint32_t) |
1031 |-------------------------------------------------------|
1032 | JobErrors (uint32_t) |
1033 |-------------------------------------------------------|
1034 | JobStatus (uint32_t) VerNum 11 |
1035 :=======================================================:
1036 * => fields deprecated
1037 Id: 32 byte Bacula Identifier "Bacula 1.0 immortal\n"
1038 LabelType (in FileIndex field of Header):
1039 EOM_LABEL -3 Label at EOM
1040 SOS_LABEL -4 Start of Session label
1041 EOS_LABEL -5 End of Session label
1042 VerNum: 11
1043 JobId: JobId
1044 write_btime: Bacula time/date this tape record written
1045 write_date: Julian date tape this record written - deprecated
1046 write_time: Julian time tape this record written - deprecated.
1047 PoolName: Pool Name
1048 PoolType: Pool Type
1049 MediaType: Media Type
1050 ClientName: Name of File daemon or Client writing this session
1051 Not used for EOM_LABEL.
1052 \end{bVerbatim}
1053 \normalsize