| 1 |
\chapter{Daemon Protocol} |
| 2 |
\label{blb:TheChapterStart2} |
| 3 |
\index{Protocol!Daemon } |
| 4 |
\index{Daemon Protocol } |
| 5 |
|
| 6 |
\section{General} |
| 7 |
\index{General } |
| 8 |
|
| 9 |
This document describes the protocols used between the various daemons. As |
| 10 |
\bacula{} has developed, it has become quite out of date. The general idea still |
| 11 |
holds true, but the details of the fields for each command, and indeed the |
| 12 |
commands themselves have changed considerably. |
| 13 |
|
| 14 |
It is intended to be a technical discussion of the general daemon protocols |
| 15 |
and as such is not targeted at end users but rather at developers and system |
| 16 |
administrators that want or need to know more of the working details of \textbf{\bacula{}}. |
| 17 |
|
| 18 |
\section{Low Level Network Protocol} |
| 19 |
\index{Protocol!Low Level Network } |
| 20 |
\index{Low Level Network Protocol } |
| 21 |
|
| 22 |
At the lowest level, the network protocol is handled by \textbf{BSOCK} packets |
| 23 |
which contain a lot of information about the status of the network connection: |
| 24 |
who is at the other end, etc. Each basic \textbf{\bacula{}} network read or write |
| 25 |
actually consists of two low level network read/writes. The first write always |
| 26 |
sends four bytes of data in machine independent byte order. If data is to |
| 27 |
follow, the first four bytes are a positive non-zero integer indicating the |
| 28 |
length of the data that follow in the subsequent write. If the four byte |
| 29 |
integer is zero or negative, it indicates a special request, a sort of network |
| 30 |
signaling capability. In this case, no data packet will follow. The low level |
| 31 |
BSOCK routines expect that only a single thread is accessing the socket at a |
| 32 |
time. It is advised that multiple threads do not read/write the same socket. |
| 33 |
If you must do this, you must provide some sort of locking mechanism. It would |
| 34 |
not be appropriate for efficiency reasons to make every call to the BSOCK |
| 35 |
routines lock and unlock the packet. |
| 36 |
|
| 37 |
\section{General Daemon Protocol} |
| 38 |
\index{General Daemon Protocol } |
| 39 |
\index{Protocol!General Daemon } |
| 40 |
|
| 41 |
In general, all the daemons follow the following global rules. There may be |
| 42 |
exceptions depending on the specific case. Normally, one daemon will be |
| 43 |
sending commands to another daemon (specifically, the Director to the Storage |
| 44 |
daemon and the Director to the File daemon). |
| 45 |
|
| 46 |
\begin{bitemize} |
| 47 |
\item Commands are always ASCII commands that are upper/lower case dependent |
| 48 |
as well as space sensitive. |
| 49 |
\item All binary data is converted into ASCII (either with printf statements |
| 50 |
or using base64 encoding). |
| 51 |
\item All responses to commands sent are always prefixed with a return |
| 52 |
numeric code where codes in the 1000's are reserved for the Director, the |
| 53 |
2000's are reserved for the File daemon, and the 3000's are reserved for the |
| 54 |
Storage daemon. |
| 55 |
\item Any response that is not prefixed with a numeric code is a command (or |
| 56 |
subcommand if you like) coming from the other end. For example, while the |
| 57 |
Director is corresponding with the Storage daemon, the Storage daemon can |
| 58 |
request Catalog services from the Director. This convention permits each side |
| 59 |
to send commands to the other daemon while simultaneously responding to |
| 60 |
commands. |
| 61 |
\item Any response that is of zero length, depending on the context, either |
| 62 |
terminates the data stream being sent or terminates command mode prior to |
| 63 |
closing the connection. |
| 64 |
\item Any response that is of negative length is a special sign that normally |
| 65 |
requires a response. For example, during data transfer from the File daemon |
| 66 |
to the Storage daemon, normally the File daemon sends continuously without |
| 67 |
intervening reads. However, periodically, the File daemon will send a packet |
| 68 |
of length -1 indicating that the current data stream is complete and that the |
| 69 |
Storage daemon should respond to the packet with an OK, ABORT JOB, PAUSE, |
| 70 |
etc. This permits the File daemon to efficiently send data while at the same |
| 71 |
time occasionally ``polling'' the Storage daemon for his status or any |
| 72 |
special requests. |
| 73 |
|
| 74 |
Currently, these negative lengths are specific to the daemon, but shortly, |
| 75 |
the range 0 to -999 will be standard daemon wide signals, while -1000 to |
| 76 |
-1999 will be for Director user, -2000 to -2999 for the File daemon, and |
| 77 |
-3000 to -3999 for the Storage daemon. |
| 78 |
\end{bitemize} |
| 79 |
|
| 80 |
\section{The Protocol Used Between the Director and the Storage Daemon} |
| 81 |
\index{Daemon!Protocol Used Between the Director and the Storage } |
| 82 |
\index{Protocol Used Between the Director and the Storage Daemon } |
| 83 |
|
| 84 |
Before sending commands to the File daemon, the Director opens a Message |
| 85 |
channel with the Storage daemon, identifies itself and presents its password. |
| 86 |
If the password check is OK, the Storage daemon accepts the Director. The |
| 87 |
Director then passes the Storage daemon, the JobId to be run as well as the |
| 88 |
File daemon authorization (append, read all, or read for a specific session). |
| 89 |
The Storage daemon will then pass back to the Director a enabling key for this |
| 90 |
JobId that must be presented by the File daemon when opening the job. Until |
| 91 |
this process is complete, the Storage daemon is not available for use by File |
| 92 |
daemons. |
| 93 |
|
| 94 |
\begin{bVerbatim} |
| 95 |
SD: listens |
| 96 |
DR: makes connection |
| 97 |
DR: Hello <Director-name> calling <password> |
| 98 |
SD: 3000 OK Hello |
| 99 |
DR: JobId=nnn Allow=(append, read) Session=(*, SessionId) |
| 100 |
(Session not implemented yet) |
| 101 |
SD: 3000 OK Job Authorization=<password> |
| 102 |
DR: use device=<device-name> media_type=<media-type> |
| 103 |
pool_name=<pool-name> pool_type=<pool_type> |
| 104 |
SD: 3000 OK use device |
| 105 |
\end{bVerbatim} |
| 106 |
|
| 107 |
For the Director to be authorized, the \bbracket{Director-name} and the |
| 108 |
\bbracket{password} must match the values in one of the Storage daemon's |
| 109 |
Director resources (there may be several Directors that can access a single |
| 110 |
Storage daemon). |
| 111 |
|
| 112 |
\section{The Protocol Used Between the Director and the File Daemon} |
| 113 |
\index{Daemon!Protocol Used Between the Director and the File } |
| 114 |
\index{Protocol Used Between the Director and the File Daemon } |
| 115 |
|
| 116 |
A typical conversation might look like the following: |
| 117 |
|
| 118 |
\begin{bVerbatim} |
| 119 |
FD: listens |
| 120 |
DR: makes connection |
| 121 |
DR: Hello <Director-name> calling <password> |
| 122 |
FD: 2000 OK Hello |
| 123 |
DR: JobId=nnn Authorization=<password> |
| 124 |
FD: 2000 OK Job |
| 125 |
DR: storage address = <Storage daemon address> port = <port-number> |
| 126 |
name = <DeviceName> mediatype = <MediaType> |
| 127 |
FD: 2000 OK storage |
| 128 |
DR: include |
| 129 |
DR: <directory1> |
| 130 |
DR: <directory2> |
| 131 |
... |
| 132 |
DR: Null packet |
| 133 |
FD: 2000 OK include |
| 134 |
DR: exclude |
| 135 |
DR: <directory1> |
| 136 |
DR: <directory2> |
| 137 |
... |
| 138 |
DR: Null packet |
| 139 |
FD: 2000 OK exclude |
| 140 |
DR: full |
| 141 |
FD: 2000 OK full |
| 142 |
DR: save |
| 143 |
FD: 2000 OK save |
| 144 |
FD: Attribute record for each file as sent to the |
| 145 |
Storage daemon (described above). |
| 146 |
FD: Null packet |
| 147 |
FD: <append close responses from Storage daemon> |
| 148 |
e.g. |
| 149 |
3000 OK Volumes = <number of volumes> |
| 150 |
3001 Volume = <volume-id> <start file> <start block> |
| 151 |
<end file> <end block> <volume session-id> |
| 152 |
3002 Volume data = <date/time of last write> <Number bytes written> |
| 153 |
<number errors> |
| 154 |
... additional Volume / Volume data pairs for volumes 2 .. n |
| 155 |
FD: Null packet |
| 156 |
FD: close socket |
| 157 |
\end{bVerbatim} |
| 158 |
|
| 159 |
\section{The Save Protocol Between the File Daemon and the Storage Daemon} |
| 160 |
\index{Save Protocol Between the File Daemon and the Storage Daemon } |
| 161 |
\index{Daemon!Save Protocol Between the File Daemon and the Storage } |
| 162 |
|
| 163 |
Once the Director has send a \bcommandname{save} command to the File daemon, the File |
| 164 |
daemon will contact the Storage daemon to begin the save. |
| 165 |
|
| 166 |
In what follows: FD: refers to information set via the network from the File |
| 167 |
daemon to the Storage daemon, and SD: refers to information set from the |
| 168 |
Storage daemon to the File daemon. |
| 169 |
|
| 170 |
\subsection{Command and Control Information} |
| 171 |
\index{Information!Command and Control } |
| 172 |
\index{Command and Control Information } |
| 173 |
|
| 174 |
Command and control information is exchanged in human readable ASCII commands. |
| 175 |
|
| 176 |
|
| 177 |
\begin{bVerbatim} |
| 178 |
FD: listens |
| 179 |
SD: makes connection |
| 180 |
FD: append open session = <JobId> [<password>] |
| 181 |
SD: 3000 OK ticket = <number> |
| 182 |
FD: append data <ticket-number> |
| 183 |
SD: 3000 OK data address = <IPaddress> port = <port> |
| 184 |
\end{bVerbatim} |
| 185 |
|
| 186 |
\subsection{Data Information} |
| 187 |
\index{Information!Data } |
| 188 |
\index{Data Information } |
| 189 |
|
| 190 |
The Data information consists of the file attributes and data to the Storage |
| 191 |
daemon. For the most part, the data information is sent one way: from the File |
| 192 |
daemon to the Storage daemon. This allows the File daemon to transfer |
| 193 |
information as fast as possible without a lot of handshaking and network |
| 194 |
overhead. |
| 195 |
|
| 196 |
However, from time to time, the File daemon needs to do a sort of checkpoint |
| 197 |
of the situation to ensure that everything is going well with the Storage |
| 198 |
daemon. To do so, the File daemon sends a packet with a negative length |
| 199 |
indicating that he wishes the Storage daemon to respond by sending a packet of |
| 200 |
information to the File daemon. The File daemon then waits to receive a packet |
| 201 |
from the Storage daemon before continuing. |
| 202 |
|
| 203 |
All data sent are in binary format except for the header packet, which is in |
| 204 |
ASCII. There are two packet types used data transfer mode: a header packet, |
| 205 |
the contents of which are known to the Storage daemon, and a data packet, the |
| 206 |
contents of which are never examined by the Storage daemon. |
| 207 |
|
| 208 |
The first data packet to the Storage daemon will be an ASCII header packet |
| 209 |
consisting of the following data. |
| 210 |
|
| 211 |
\bbracket{File-Index} \bbracket{Stream-Id} \bbracket{Info} where |
| 212 |
\textbf{\bbracket{File-Index}} is a sequential number beginning from one that |
| 213 |
increments with each file (or directory) sent. |
| 214 |
|
| 215 |
where \textbf{\bbracket{Stream-Id}} will be 1 for the Attributes record and 2 for |
| 216 |
uncompressed File data. 3 is reserved for the MD5 signature for the file. |
| 217 |
|
| 218 |
where \textbf{\bbracket{Info}} transmit information about the Stream to the |
| 219 |
Storage Daemon. It is a character string field where each character has a |
| 220 |
meaning. The only character currently defined is 0 (zero), which is simply a |
| 221 |
place holder (a no op). In the future, there may be codes indicating |
| 222 |
compressed data, encrypted data, etc. |
| 223 |
|
| 224 |
Immediately following the header packet, the Storage daemon will expect any |
| 225 |
number of data packets. The series of data packets is terminated by a zero |
| 226 |
length packet, which indicates to the Storage daemon that the next packet will |
| 227 |
be another header packet. As previously mentioned, a negative length packet is |
| 228 |
a request for the Storage daemon to temporarily enter command mode and send a |
| 229 |
reply to the File daemon. Thus an actual conversation might contain the |
| 230 |
following exchanges: |
| 231 |
|
| 232 |
\begin{bVerbatim} |
| 233 |
FD: <1 1 0> (header packet) |
| 234 |
FD: <data packet containing file-attributes> |
| 235 |
FD: Null packet |
| 236 |
FD: <1 2 0> |
| 237 |
FD: <multiple data packets containing the file data> |
| 238 |
FD: Packet length = -1 |
| 239 |
SD: 3000 OK |
| 240 |
FD: <2 1 0> |
| 241 |
FD: <data packet containing file-attributes> |
| 242 |
FD: Null packet |
| 243 |
FD: <2 2 0> |
| 244 |
FD: <multiple data packets containing the file data> |
| 245 |
FD: Null packet |
| 246 |
FD: Null packet |
| 247 |
FD: append end session <ticket-number> |
| 248 |
SD: 3000 OK end |
| 249 |
FD: append close session <ticket-number> |
| 250 |
SD: 3000 OK Volumes = <number of volumes> |
| 251 |
SD: 3001 Volume = <volumeid> <start file> <start block> |
| 252 |
<end file> <end block> <volume session-id> |
| 253 |
SD: 3002 Volume data = <date/time of last write> <Number bytes written> |
| 254 |
<number errors> |
| 255 |
SD: ... additional Volume / Volume data pairs for |
| 256 |
volumes 2 .. n |
| 257 |
FD: close socket |
| 258 |
\end{bVerbatim} |
| 259 |
|
| 260 |
The information returned to the File daemon by the Storage daemon in response |
| 261 |
to the \textbf{append close session} is transmit in turn to the Director. |