| 1 |
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> |
| 2 |
<html><head><title>RELP - The Reliable Event Logging Protocol (Specification)</title></head> |
| 3 |
<body> |
| 4 |
<h1>RELP - The Reliable Event Logging Protocol</h1> |
| 5 |
<p>This is the specification for the reliable event logging |
| 6 |
protocol, called "RELP".</p> |
| 7 |
<p>Version: 0.0.1<br> |
| 8 |
Date: 2008-03-19<br> |
| 9 |
Author: <a href="http://www.gerhards.net/rainer">Rainer |
| 10 |
Gerhards</a></p> |
| 11 |
<p>Copyright (C) 2008 by Rainer Gerhards and Adiscon GmbH. |
| 12 |
Released under the terms of the GNU FDL.</p> |
| 13 |
<h2>DESCRIPTION OF THE RELP PROTOCOL</h2> |
| 14 |
<p>Relp uses a client-server model with (mostly fixed roles. The |
| 15 |
initiating part of the |
| 16 |
connection is called the client, the listening part the server. In the |
| 17 |
state |
| 18 |
diagrams below, C stand for client and S for server. |
| 19 |
</p> |
| 20 |
<p>Relp employs a command-response model, that is the client |
| 21 |
issues commands to |
| 22 |
which the server responds. To facilitate full-duplex communication, |
| 23 |
multiple |
| 24 |
commands can be issued at the same time, thus multiple responses may be |
| 25 |
outstanding at a given time. The server may reply in any order. To |
| 26 |
conserve |
| 27 |
ressources, the number of outstanding commands is limited by a window. |
| 28 |
Each |
| 29 |
command is assigned a (relative) unique, monotonically increasing ID |
| 30 |
(called the |
| 31 |
"transaction number" or txnr for short. Each |
| 32 |
response must include that ID. Responses must be sent by the server in |
| 33 |
the exact same order as commands where received.</p> |
| 34 |
<p>There is one potential issue with this communications model: |
| 35 |
In it, the server is able to respond to the client only in form or a |
| 36 |
response package. It is not capable of issuing a command itself. During |
| 37 |
normal operations, this does not pose any problem. However, if the |
| 38 |
server is shut down, there is no way for it to tell the client of its |
| 39 |
shutdown intension, especially if the client does not send any data at |
| 40 |
that moment. The same situation arises when the server timeouts a |
| 41 |
sessions where the client has not sent any data for an extended period |
| 42 |
of time (this probably is a good thing to conserve server ressources). |
| 43 |
Of course, an obvious answer to that need is that the server |
| 44 |
simply closes the connection after having sent all responses (or all |
| 45 |
that it is still capable of). The client will notice the closed |
| 46 |
connection at latest when it tries to send new commands. However, it |
| 47 |
may be worth considering an alternate mode where the server can issue |
| 48 |
commands to the client. One potential way to do this would be to |
| 49 |
reserve a specific txnr (0, for example) for this use. That txnr is |
| 50 |
only to be used by the server and only to send "out-of-band" commands. |
| 51 |
These commands must not be acknowledged by the client, because that |
| 52 |
would complicate the protocol more than useful for this case. So |
| 53 |
out-of-band-commands have a "hint" status - they hint the client about |
| 54 |
something, but if they do not arrive at the client, nothing fatal |
| 55 |
should happen. The ultimate session closure indication would still be |
| 56 |
the tcp session close. [Please see the "High Latency Environment" use |
| 57 |
case below for some implications this hinting process may have].</p> |
| 58 |
<p>For the time being, it has been decided that such out of band |
| 59 |
hints be not supported.</p> |
| 60 |
<p>A command and its response is called a relp transaction. A |
| 61 |
response must always carry the same txnr as the original command. |
| 62 |
</p> |
| 63 |
<p>If something goes really wrong, both the client and the server |
| 64 |
may terminate |
| 65 |
the TCP connection at any time. Any outstanding commands are considered |
| 66 |
to |
| 67 |
have been unsuccessful in this case.</p> |
| 68 |
<h3>ERROR HANDLING</h3> |
| 69 |
<p> |
| 70 |
If something goes wrong (e.g. invalid framing), the peer that detects |
| 71 |
the problem |
| 72 |
closes the underlying (TCP) connection. It may send an error/abort |
| 73 |
hint, but |
| 74 |
MUST NOT wait for its response. The reasoning for this is that today's |
| 75 |
Internet is |
| 76 |
not a friendly place. Trying to gracefully resolve communications |
| 77 |
problems may |
| 78 |
lead to an attack vector. On the other hand, if the problem was of |
| 79 |
temporary nature, |
| 80 |
it may be cleared up by the connection reset and a new connect. So |
| 81 |
aborting errored |
| 82 |
connections is considered both secure as well as reasonable efficient.</p> |
| 83 |
<p>Note: the ability to send abort/error messages depends on the |
| 84 |
decision of out-of-band-hint-commands (see above). |
| 85 |
</p> |
| 86 |
<h3>SENDING MESSAGES</h3> |
| 87 |
Because it is so important, I'd like to point it specifically out: |
| 88 |
sending a |
| 89 |
message is "just" another RELP command. The reply to that command is |
| 90 |
the ACK/NAK |
| 91 |
for the message. So every message *is* acknowledged. RELP options |
| 92 |
indicate how |
| 93 |
"deep" this acknowledge is (once we have implemented that), in the most |
| 94 |
extreme |
| 95 |
case a RELP client may ask a RELP server to ack only after the message |
| 96 |
has been |
| 97 |
completely acted upon (e.g. successfully written to database) - but |
| 98 |
this is far |
| 99 |
away in the future. For now, keep in mind that message loss will always |
| 100 |
be detected |
| 101 |
because we have app-level acknowledgement. |
| 102 |
<h3>RELP FRAME</h3> |
| 103 |
All relp transaction are carried out over a consistent framing. |
| 104 |
<pre>RELP-FRAME = HEADER DATA TRAILER<br>DATA = [SP 1*OCTET] ; command-defined data, if DATALEN is 0, no data is present<br>HEADER = TXNR SP COMMAND SP DATALEN<br>TXNR = NUMBER ; relp transaction number, monotonically increases, starts at 1<br>DATALEN = NUMBER<br>#old:COMMAND = "open" / "syslog" / "close" / "rsp" / "abort" ; max length = 32<br>COMMAND = 1*32ALPHA<br>TRAILER = LF ; to detect framing errors and enhance human readibility<br>ALPHA = letter ; ('a'..'z', 'A'..'Z')<br>NUMBER = 1*9DIGIT<br>DIGIT = %d48-57<br>LF = %d10<br>SP = %d32<br><br>RSP DATA CONTENT:<br>RSP-HEADER = TXNR SP RSP-CODE [SP HUMANMSG] LF [CMDDATA]<br>RSP-CODE = 200 / 500 ; 200 is ok, all the rest currently erros<br>HUAMANMSG = *OCTET ; a human-readble message without LF in it<br>CMDDATA = *OCTET ; semantics depend on original command<br>TXNR is as in the relp frame, it is the TXNR of the frame being responded to.<br></pre> |
| 105 |
<p> |
| 106 |
DATALEN is the number of octets in DATA (so the frame length excluding |
| 107 |
the length |
| 108 |
of HEADER and TRAILER). Please note that the theoretical maximum |
| 109 |
(999,999,999 octets) |
| 110 |
is in (almost?) all cases unsuitable for actual message transfer. Thus, |
| 111 |
the actual |
| 112 |
maximum data length is negotiated during session setup. In relp version |
| 113 |
1, it is |
| 114 |
always 128K. In later versions, it shall be operator-configurable. |
| 115 |
</p> |
| 116 |
<p>TXNR 0 is reserved for hint commands. A RELP |
| 117 |
connection start with TXNR 1. Note that TXNR montonically increases but |
| 118 |
is latched at 999,999,999. After that TXNR, the next TXNR is |
| 119 |
1. </p> |
| 120 |
<h3>COMMAND SEMANTICS</h3> |
| 121 |
Command can be either regular commands or hints. Regular commands are |
| 122 |
responded to by a rsp package. They are transmitted on txnrs of 1 or |
| 123 |
above. Hints are commands that are not being responded. They are always |
| 124 |
transmitted with a txnr of zero. Regular commands can only be issued by |
| 125 |
the client. Hints can be issued both by the client and the server.<br> |
| 126 |
<h4>"Command" "rsp"</h4> |
| 127 |
This is not a real command but a response to a client-issued command. |
| 128 |
The TXNR MUST match the client's |
| 129 |
command |
| 130 |
TXNR. The data part contains RSP-HEADER as defined above. It is a |
| 131 |
response code, |
| 132 |
optionally followed by a space and additional data (depending on the |
| 133 |
client's command). Return state values are: 200 - OK, 500 - error.<br> |
| 134 |
<br> |
| 135 |
The ABNF for a "rsp"'s data portion is as follows:<br> |
| 136 |
<pre>RSPDATA = STATUS SP HUMAN-TEXT [LF DATA]<br>STATUS = 3DIGIT<br>HUMAN-TEXT = *80ALPHA<br>DATA = 1*OCTET<br></pre> |
| 137 |
<h4>Command "open"</h4> |
| 138 |
Opens a connection to the server. Must include offers inside the |
| 139 |
data, at a minimum the "relp_version" offer. Offers |
| 140 |
provide information about services supported by the client. |
| 141 |
<p>When the server receives an open, it parses the offers, checks |
| 142 |
what it itself supports |
| 143 |
and provides those offers that it accepts in the "rsp". The server may |
| 144 |
send a "rsp" with an empty offer part in case it doesn't like any of |
| 145 |
the offers. To establish a session, a "relp_version" offer MUST be |
| 146 |
included in the response. If it is not, the client MUST close the |
| 147 |
connection. </p> |
| 148 |
<p>When the client receives the "rsp", it checks the servers |
| 149 |
offers and finally selects |
| 150 |
those that should be used during the session. Please note that this |
| 151 |
doesn't imply the |
| 152 |
client selects e.g. security strength. To require a specific security |
| 153 |
strength, the |
| 154 |
server must be configured to offer only those options back to the |
| 155 |
client that it is |
| 156 |
happy to accept. So the client can only select from those. As such, |
| 157 |
even though the |
| 158 |
client makes the final feature selection, the server is dictating what |
| 159 |
needs to be used. |
| 160 |
</p> |
| 161 |
<p>Note that the connection is only ready AFTER the client has |
| 162 |
received the "open" response.</p> |
| 163 |
<h4>Command "close"</h4> |
| 164 |
Closes a connection to the server. Once the client has sent a "close" |
| 165 |
command, it must not transmit any other command over the session. When |
| 166 |
the server's rsp answer is received, the client must close the |
| 167 |
connection.<br><h4>Hint Command "serverclose"</h4>This |
| 168 |
hint SHOULD be sent by the server before it closes a RELP connection. |
| 169 |
It advised the client that the server has closed the connection. The |
| 170 |
client SHOULD use this knowledge to close the connection on the |
| 171 |
client-side, too. Please note that this is a hint, and as such |
| 172 |
transferred with a TXNR of 0 and without any response from the client. |
| 173 |
Note that the client can not ask the server to defer closing of the |
| 174 |
connection. A client MUST correctly process a server-closed connection |
| 175 |
even if the server does not send a "serverclose" hint. In thus case, it |
| 176 |
must rely on the transport layer connection status. The primary utility |
| 177 |
of "serverclose" is to enable the client to quickly detect a |
| 178 |
server-closed connection, clean up ressources and go into termination |
| 179 |
(or recovery code).<h4>Command "starttls"</h4> |
| 180 |
Does the same thing as ESMTP STARTTLS: switches the connection |
| 181 |
to TLS mode. |
| 182 |
<h4>Command "syslog"</h4> |
| 183 |
This command is used to transmit a syslog message, which (in syslog |
| 184 |
message format) is contained within the commands data portion. |
| 185 |
<h3>OFFERS</h3> |
| 186 |
During session setup, "offers" are exchange between client and server. |
| 187 |
An "offer" describes |
| 188 |
a specific feature or operation mode. Always present must be the |
| 189 |
"relp_version" offer which |
| 190 |
tells the other side which version of relp is in use. |
| 191 |
<p>ABNF for offer strings |
| 192 |
</p> |
| 193 |
<pre>OFFER = LF FEATURENAME ["=" VALUE *("," VALUE)]<br>FEATURENAME = *32OCTET<br>VALUE = *255OCTET<br><br>Currently defined values:<br>FEATURENAME VALUE<br>relp_version 1 (this specification)<br></pre> |
| 194 |
<h4>relp_version</h4> |
| 195 |
<p>This offer describes the protocol version. It MUST be provided |
| 196 |
in the |
| 197 |
open command. It has one parameter, the numerical version id. The |
| 198 |
intial version is 1. There also exists an experimental version 0, which |
| 199 |
shall not be used in practice. |
| 200 |
</p> |
| 201 |
<h4>commands</h4> |
| 202 |
<p>This offer describes which commands are supported by the |
| 203 |
remote peer. Command that must be specified to drive the RELP protocol |
| 204 |
itself (e.g. "open", "close", "rsp") MUST NOT be specified. Optional |
| 205 |
commands must be specified. There may be multiple commands specified.</p> |
| 206 |
<p>Sample: commands=syslog,eventlog (eventlog is a hypothetical, |
| 207 |
not yet known command).</p> |
| 208 |
<p>Please note that new commands may be specified outside of the |
| 209 |
relp specification. Also note that if the server response does not |
| 210 |
include the command the client is interested in, this command MUST NOT |
| 211 |
be used. The client may then either resort to another, less featured |
| 212 |
supported command or may terminate the relp connection. |
| 213 |
</p> |
| 214 |
<h4>relp_software</h4> |
| 215 |
<p>This offer describes the software (library) that is processing |
| 216 |
the relp |
| 217 |
protocol. It is an optional offer. The idea is that when each peer |
| 218 |
knows which software on the other peer is handling the protocol, it may |
| 219 |
be able to work-around some known software issues. The "relp_software" |
| 220 |
has two parameters, the first one being the name of the software as it |
| 221 |
calls itself, the second one the version as it calls itself and the |
| 222 |
third one being an URL where information about this software may be |
| 223 |
found. It is suggested that all three parameters are given, but only |
| 224 |
the first one MUST actually be provided. |
| 225 |
</p> |
| 226 |
<h3>STATE DIAGRAMS</h3> |
| 227 |
<p>... detailling some communications scenarios: |
| 228 |
</p> |
| 229 |
<pre>Session Startup:<br>C S<br>cmd: "open", data: offer -----> (selects supported offers)<br>(selects offers to use) <----- cmd: "rsp", data "accepted offers"<br><br> ... transmission channel is ready to use ....<br><br>Message Transmission<br>C S<br>cmd: "syslog", data: syslogmsg -----> (processes message)<br>(indicates syslog as processed) <----- cmd: "rsp", data OK/Error<br><br>Session Termination<br>C S<br>cmd: "close", data: none? -----> (processes termination request)<br>(terminates session) <----- cmd: "rsp", data OK/Error<br> (terminates session)<br></pre> |
| 230 |
<h3>SECURITY CONSIDERATIONS</h3> |
| 231 |
<h4>Window Size</h4> |
| 232 |
A very large window can be abused for denial of service attacks. |
| 233 |
<h4>Maximum DATALEN</h4> |
| 234 |
A too-large DATALEN can be absued for denial of service attacks. |
| 235 |
<h2>Use Cases</h2> |
| 236 |
These uses cases are not really a part of the |
| 237 |
specification. I have added them to provide some insight into |
| 238 |
implementation specifics. These use cases may be removed (or moved to |
| 239 |
another part of the doc set) some time in the future. In the mean time, |
| 240 |
the serve a valuable purpose when thinking about protocol features. |
| 241 |
<h3>Server Session Closure with a Hibernated Client</h3> |
| 242 |
I define this use case in support of <a href="http://www.rsyslog.com">rsyslog</a> and other |
| 243 |
possible clients which try to reduce ressource consumption.<br> |
| 244 |
<br> |
| 245 |
Rsyslog utilizes highly threaded design, among others to take advantage |
| 246 |
of multicore-machines. On |
| 247 |
the other hand, it conserves resources by only running as many worker |
| 248 |
threads as actually needed. So if there is no traffic, it may even be |
| 249 |
running no workers at all. The RELP client will be implemented as an |
| 250 |
output |
| 251 |
plugin and will be executed on a worker thread. So in low-traffic |
| 252 |
cases, the RELP client kind of hibernates, maybe even for hours (if no |
| 253 |
new message is scheduled for transmission). |
| 254 |
<p>In this use case, I examine a typical scenario in a |
| 255 |
low-message volume case. Here, the client is asked to transmit a |
| 256 |
message every now and than, at a pace of at most one message every few |
| 257 |
seconds. It may also happen that no message is to be sent for several |
| 258 |
minutes or even hours. Such a scenario is not as extreme as it sounds - |
| 259 |
it may, for example, to be used for error notifications, which |
| 260 |
hopefully happen very infrequently.</p> |
| 261 |
<p>In this scenario, the client sends a command, but will usually |
| 262 |
not receive an immediate response, because client processing is already |
| 263 |
terminated when the server sends the response packet. Thus, the tcp |
| 264 |
stacks receive buffer buffers the server's response until the client |
| 265 |
has another message to send. Then, the client first checks for any |
| 266 |
outstanding responses, processes them and then send the new message. It |
| 267 |
is expected that in this scenario, each client invokation will process |
| 268 |
the past response and send a new packet. The response to that packet |
| 269 |
will not be processed by the invocation that sent the command but by |
| 270 |
the next one. So we always have one outstanding response inside the OS |
| 271 |
buffers. Now consider the server is shutting down the connection due to |
| 272 |
timeout or due to the fact that it needs to shutdown itself (maybe even |
| 273 |
on an urgent case, like power failure - the point is it can not wait). |
| 274 |
On next client invokation, the client finds an outstanding response and |
| 275 |
processes it. Then, it tries to send a new command. That will fail, |
| 276 |
because the connection has been terminated. The usual session recovery |
| 277 |
logic is used to restablish the connection when the server is back |
| 278 |
online. The command in question is transferred after session |
| 279 |
re-establishment.</p> |
| 280 |
<h3>High Latency Environment</h3> |
| 281 |
<p>We have a high latency environment that is otherwise quite |
| 282 |
reliable |
| 283 |
(satellite link). We have a high traffic load. To cover this, a large<br> |
| 284 |
transmission |
| 285 |
window has been selected (let's say 1000 messages). Now the client |
| 286 |
sends a burst of messages but then has nothing to do and |
| 287 |
falls |
| 288 |
asleep. Then, the server starts sending acks. These acks are not taken |
| 289 |
off the wire by the client (as it is inactive - rsyslog case). |
| 290 |
Eventually, the tcp window fills. Thus the server can no longer send |
| 291 |
acks. This does not immediately pose any problems, as the server adds |
| 292 |
the ack frames to its internal send queue.</p> |
| 293 |
<p>If the client comes |
| 294 |
out of hibernation, it receives all previous sent frames, freeing the |
| 295 |
server's tcp send window. Thus, the server can continue to send data |
| 296 |
from its send queue (and drain it). The client, having received the |
| 297 |
server's acks, will not stall because its own RELP window has been |
| 298 |
cleared by the acks.</p> |
| 299 |
<p>The situation is more complicated if |
| 300 |
the server intends to shut down while the client is in hibernation. To |
| 301 |
do so, the server usually sends a "adviseclose" command followed by a |
| 302 |
"close" command. However, neither of these commands can be sent because |
| 303 |
the client does not take them off the wire. The close operation is now |
| 304 |
stalled. The only option left to the server is an unconditional |
| 305 |
termination of the session. While everything that has already been sent |
| 306 |
to the client will be picked up by it when it comes out of hibernation, |
| 307 |
anything left in the server's send queue will be lost in this case. As |
| 308 |
such, acks are potentially lost. This will lead to message duplication |
| 309 |
as the client assumes that the frames unacked at time of force-close |
| 310 |
where not processed (there is no other safe assumption). Consequently, |
| 311 |
the client will re-send these frames in the next session.</p> |
| 312 |
<p>A cure |
| 313 |
to this situation is to have the client concurrently listen to server |
| 314 |
requests, e.g. by running a receiver on a separate thread. The RELP |
| 315 |
protocol does not demand this behaviour. However, it can be used to |
| 316 |
solve the above server stall. Let's assume this is being done. Do we |
| 317 |
now have a sufficient guarantee that the server does not stall? Under |
| 318 |
normal conditions, it is a safe assumption that the client will be able |
| 319 |
to receive all frames sent by the server. So a buildup of server queues |
| 320 |
should, if at all, happen only for a short instant. However, if |
| 321 |
something goes wrong on the transmission line (especially in |
| 322 |
high-latency cases), there may be a somewhat extended period of time in |
| 323 |
which the server can not send acks (but only in a magnitude of a few |
| 324 |
seconds at most). If the server intends to shutdown during such a |
| 325 |
period, a short timeout may enable it to avoid a fore-shutdown. |
| 326 |
However, there are still cases thinkable where a force shutdown may be |
| 327 |
required. These are deemed to be highly unlikely.</p> |
| 328 |
<p>It is the |
| 329 |
protocol implementer's choice if the slight less chance of a server |
| 330 |
force-shutdown justifies the addition of a background thread to a |
| 331 |
program that otherwise doesn't need one. From a protocol point of view, |
| 332 |
a force-shutdown is a valid operation. Also, while it causes potential |
| 333 |
message duplication, it can not cause message loss.</p> |
| 334 |
<h2>Feedback and Discussion</h2> |
| 335 |
<p>Feedback on this document and the RELP protocol is |
| 336 |
appreciated. The <a href="http://lists.adiscon.net/mailman/listinfo/relp">RELP |
| 337 |
mailing list</a> is probably a good place to provide it. |
| 338 |
Questions are also happily accepted.</p> |
| 339 |
<h2>Copyright</h2> |
| 340 |
<p>Copyright (C) 2008, 2009 Rainer Gerhards and Adiscon. |
| 341 |
<p> |
| 342 |
Permission is granted to copy, distribute and/or modify this document |
| 343 |
under the terms of the GNU Free Documentation License, Version 1.3 |
| 344 |
or any later version published by the Free Software Foundation; |
| 345 |
with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. |
| 346 |
A copy of the license can be obtained from <a href="http://www.gnu.org/copyleft/fdl.html">http://www.gnu.org/copyleft/fdl.html</a> and is also included in the <a href="gfdl.html">license file</a> (if there is any difference in those two, the one contained in the license file is the one that this work is licensed under). |
| 347 |
</body></html> |