Debian Codegraph @

manuals/en/developers/daemonprotocol.tex

raw
1 \chapter{Daemon Protocol}
2 \label{blb:TheChapterStart2}
3 \index{Protocol!Daemon }
4 \index{Daemon Protocol }
5
6 \section{General}
7 \index{General }
8
9 This document describes the protocols used between the various daemons. As
10 \bacula{} has developed, it has become quite out of date. The general idea still
11 holds true, but the details of the fields for each command, and indeed the
12 commands themselves have changed considerably.
13
14 It is intended to be a technical discussion of the general daemon protocols
15 and as such is not targeted at end users but rather at developers and system
16 administrators that want or need to know more of the working details of \textbf{\bacula{}}.
17
18 \section{Low Level Network Protocol}
19 \index{Protocol!Low Level Network }
20 \index{Low Level Network Protocol }
21
22 At the lowest level, the network protocol is handled by \textbf{BSOCK} packets
23 which contain a lot of information about the status of the network connection:
24 who is at the other end, etc. Each basic \textbf{\bacula{}} network read or write
25 actually consists of two low level network read/writes. The first write always
26 sends four bytes of data in machine independent byte order. If data is to
27 follow, the first four bytes are a positive non-zero integer indicating the
28 length of the data that follow in the subsequent write. If the four byte
29 integer is zero or negative, it indicates a special request, a sort of network
30 signaling capability. In this case, no data packet will follow. The low level
31 BSOCK routines expect that only a single thread is accessing the socket at a
32 time. It is advised that multiple threads do not read/write the same socket.
33 If you must do this, you must provide some sort of locking mechanism. It would
34 not be appropriate for efficiency reasons to make every call to the BSOCK
35 routines lock and unlock the packet.
36
37 \section{General Daemon Protocol}
38 \index{General Daemon Protocol }
39 \index{Protocol!General Daemon }
40
41 In general, all the daemons follow the following global rules. There may be
42 exceptions depending on the specific case. Normally, one daemon will be
43 sending commands to another daemon (specifically, the Director to the Storage
44 daemon and the Director to the File daemon).
45
46 \begin{bitemize}
47 \item Commands are always ASCII commands that are upper/lower case dependent
48 as well as space sensitive.
49 \item All binary data is converted into ASCII (either with printf statements
50 or using base64 encoding).
51 \item All responses to commands sent are always prefixed with a return
52 numeric code where codes in the 1000's are reserved for the Director, the
53 2000's are reserved for the File daemon, and the 3000's are reserved for the
54 Storage daemon.
55 \item Any response that is not prefixed with a numeric code is a command (or
56 subcommand if you like) coming from the other end. For example, while the
57 Director is corresponding with the Storage daemon, the Storage daemon can
58 request Catalog services from the Director. This convention permits each side
59 to send commands to the other daemon while simultaneously responding to
60 commands.
61 \item Any response that is of zero length, depending on the context, either
62 terminates the data stream being sent or terminates command mode prior to
63 closing the connection.
64 \item Any response that is of negative length is a special sign that normally
65 requires a response. For example, during data transfer from the File daemon
66 to the Storage daemon, normally the File daemon sends continuously without
67 intervening reads. However, periodically, the File daemon will send a packet
68 of length -1 indicating that the current data stream is complete and that the
69 Storage daemon should respond to the packet with an OK, ABORT JOB, PAUSE,
70 etc. This permits the File daemon to efficiently send data while at the same
71 time occasionally ``polling'' the Storage daemon for his status or any
72 special requests.
73
74 Currently, these negative lengths are specific to the daemon, but shortly,
75 the range 0 to -999 will be standard daemon wide signals, while -1000 to
76 -1999 will be for Director user, -2000 to -2999 for the File daemon, and
77 -3000 to -3999 for the Storage daemon.
78 \end{bitemize}
79
80 \section{The Protocol Used Between the Director and the Storage Daemon}
81 \index{Daemon!Protocol Used Between the Director and the Storage }
82 \index{Protocol Used Between the Director and the Storage Daemon }
83
84 Before sending commands to the File daemon, the Director opens a Message
85 channel with the Storage daemon, identifies itself and presents its password.
86 If the password check is OK, the Storage daemon accepts the Director. The
87 Director then passes the Storage daemon, the JobId to be run as well as the
88 File daemon authorization (append, read all, or read for a specific session).
89 The Storage daemon will then pass back to the Director a enabling key for this
90 JobId that must be presented by the File daemon when opening the job. Until
91 this process is complete, the Storage daemon is not available for use by File
92 daemons.
93
94 \begin{bVerbatim}
95 SD: listens
96 DR: makes connection
97 DR: Hello <Director-name> calling <password>
98 SD: 3000 OK Hello
99 DR: JobId=nnn Allow=(append, read) Session=(*, SessionId)
100 (Session not implemented yet)
101 SD: 3000 OK Job Authorization=<password>
102 DR: use device=<device-name> media_type=<media-type>
103 pool_name=<pool-name> pool_type=<pool_type>
104 SD: 3000 OK use device
105 \end{bVerbatim}
106
107 For the Director to be authorized, the \bbracket{Director-name} and the
108 \bbracket{password} must match the values in one of the Storage daemon's
109 Director resources (there may be several Directors that can access a single
110 Storage daemon).
111
112 \section{The Protocol Used Between the Director and the File Daemon}
113 \index{Daemon!Protocol Used Between the Director and the File }
114 \index{Protocol Used Between the Director and the File Daemon }
115
116 A typical conversation might look like the following:
117
118 \begin{bVerbatim}
119 FD: listens
120 DR: makes connection
121 DR: Hello <Director-name> calling <password>
122 FD: 2000 OK Hello
123 DR: JobId=nnn Authorization=<password>
124 FD: 2000 OK Job
125 DR: storage address = <Storage daemon address> port = <port-number>
126 name = <DeviceName> mediatype = <MediaType>
127 FD: 2000 OK storage
128 DR: include
129 DR: <directory1>
130 DR: <directory2>
131 ...
132 DR: Null packet
133 FD: 2000 OK include
134 DR: exclude
135 DR: <directory1>
136 DR: <directory2>
137 ...
138 DR: Null packet
139 FD: 2000 OK exclude
140 DR: full
141 FD: 2000 OK full
142 DR: save
143 FD: 2000 OK save
144 FD: Attribute record for each file as sent to the
145 Storage daemon (described above).
146 FD: Null packet
147 FD: <append close responses from Storage daemon>
148 e.g.
149 3000 OK Volumes = <number of volumes>
150 3001 Volume = <volume-id> <start file> <start block>
151 <end file> <end block> <volume session-id>
152 3002 Volume data = <date/time of last write> <Number bytes written>
153 <number errors>
154 ... additional Volume / Volume data pairs for volumes 2 .. n
155 FD: Null packet
156 FD: close socket
157 \end{bVerbatim}
158
159 \section{The Save Protocol Between the File Daemon and the Storage Daemon}
160 \index{Save Protocol Between the File Daemon and the Storage Daemon }
161 \index{Daemon!Save Protocol Between the File Daemon and the Storage }
162
163 Once the Director has send a \bcommandname{save} command to the File daemon, the File
164 daemon will contact the Storage daemon to begin the save.
165
166 In what follows: FD: refers to information set via the network from the File
167 daemon to the Storage daemon, and SD: refers to information set from the
168 Storage daemon to the File daemon.
169
170 \subsection{Command and Control Information}
171 \index{Information!Command and Control }
172 \index{Command and Control Information }
173
174 Command and control information is exchanged in human readable ASCII commands.
175
176
177 \begin{bVerbatim}
178 FD: listens
179 SD: makes connection
180 FD: append open session = <JobId> [<password>]
181 SD: 3000 OK ticket = <number>
182 FD: append data <ticket-number>
183 SD: 3000 OK data address = <IPaddress> port = <port>
184 \end{bVerbatim}
185
186 \subsection{Data Information}
187 \index{Information!Data }
188 \index{Data Information }
189
190 The Data information consists of the file attributes and data to the Storage
191 daemon. For the most part, the data information is sent one way: from the File
192 daemon to the Storage daemon. This allows the File daemon to transfer
193 information as fast as possible without a lot of handshaking and network
194 overhead.
195
196 However, from time to time, the File daemon needs to do a sort of checkpoint
197 of the situation to ensure that everything is going well with the Storage
198 daemon. To do so, the File daemon sends a packet with a negative length
199 indicating that he wishes the Storage daemon to respond by sending a packet of
200 information to the File daemon. The File daemon then waits to receive a packet
201 from the Storage daemon before continuing.
202
203 All data sent are in binary format except for the header packet, which is in
204 ASCII. There are two packet types used data transfer mode: a header packet,
205 the contents of which are known to the Storage daemon, and a data packet, the
206 contents of which are never examined by the Storage daemon.
207
208 The first data packet to the Storage daemon will be an ASCII header packet
209 consisting of the following data.
210
211 \bbracket{File-Index} \bbracket{Stream-Id} \bbracket{Info} where
212 \textbf{\bbracket{File-Index}} is a sequential number beginning from one that
213 increments with each file (or directory) sent.
214
215 where \textbf{\bbracket{Stream-Id}} will be 1 for the Attributes record and 2 for
216 uncompressed File data. 3 is reserved for the MD5 signature for the file.
217
218 where \textbf{\bbracket{Info}} transmit information about the Stream to the
219 Storage Daemon. It is a character string field where each character has a
220 meaning. The only character currently defined is 0 (zero), which is simply a
221 place holder (a no op). In the future, there may be codes indicating
222 compressed data, encrypted data, etc.
223
224 Immediately following the header packet, the Storage daemon will expect any
225 number of data packets. The series of data packets is terminated by a zero
226 length packet, which indicates to the Storage daemon that the next packet will
227 be another header packet. As previously mentioned, a negative length packet is
228 a request for the Storage daemon to temporarily enter command mode and send a
229 reply to the File daemon. Thus an actual conversation might contain the
230 following exchanges:
231
232 \begin{bVerbatim}
233 FD: <1 1 0> (header packet)
234 FD: <data packet containing file-attributes>
235 FD: Null packet
236 FD: <1 2 0>
237 FD: <multiple data packets containing the file data>
238 FD: Packet length = -1
239 SD: 3000 OK
240 FD: <2 1 0>
241 FD: <data packet containing file-attributes>
242 FD: Null packet
243 FD: <2 2 0>
244 FD: <multiple data packets containing the file data>
245 FD: Null packet
246 FD: Null packet
247 FD: append end session <ticket-number>
248 SD: 3000 OK end
249 FD: append close session <ticket-number>
250 SD: 3000 OK Volumes = <number of volumes>
251 SD: 3001 Volume = <volumeid> <start file> <start block>
252 <end file> <end block> <volume session-id>
253 SD: 3002 Volume data = <date/time of last write> <Number bytes written>
254 <number errors>
255 SD: ... additional Volume / Volume data pairs for
256 volumes 2 .. n
257 FD: close socket
258 \end{bVerbatim}
259
260 The information returned to the File daemon by the Storage daemon in response
261 to the \textbf{append close session} is transmit in turn to the Director.