Debian Codegraph @

doc/relp.html

raw
1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
2 <html><head><title>RELP - The Reliable Event Logging Protocol (Specification)</title></head>
3 <body>
4 <h1>RELP - The Reliable Event Logging Protocol</h1>
5 <p>This is the specification for the reliable event logging
6 protocol, called "RELP".</p>
7 <p>Version: 0.0.1<br>
8 Date: &nbsp;2008-03-19<br>
9 Author: <a href="http://www.gerhards.net/rainer">Rainer
10 Gerhards</a></p>
11 <p>Copyright (C) 2008 by Rainer Gerhards and Adiscon GmbH.
12 Released under the terms of the GNU FDL.</p>
13 <h2>DESCRIPTION OF THE RELP PROTOCOL</h2>
14 <p>Relp uses a client-server model with (mostly fixed roles. The
15 initiating part of the
16 connection is called the client, the listening part the server. In the
17 state
18 diagrams below, C stand for client and S for server.
19 </p>
20 <p>Relp employs a command-response model, that is the client
21 issues commands to
22 which the server responds. To facilitate full-duplex communication,
23 multiple
24 commands can be issued at the same time, thus multiple responses may be
25 outstanding at a given time. The server may reply in any order. To
26 conserve
27 ressources, the number of outstanding commands is limited by a window.
28 Each
29 command is assigned a (relative) unique, monotonically increasing ID
30 (called the
31 "transaction number" or txnr for short. Each
32 response must include that ID. Responses must be sent by the server in
33 the exact same order as commands where received.</p>
34 <p>There is one potential issue with this communications model:
35 In it, the server is able to respond to the client only in form or a
36 response package. It is not capable of issuing a command itself. During
37 normal operations, this does not pose any problem. However, if the
38 server is shut down, there is no way for it to tell the client of its
39 shutdown intension, especially if the client does not send any data at
40 that moment. The same situation arises when the server timeouts a
41 sessions where the client has not sent any data for an extended period
42 of time (this probably is a good thing to conserve server ressources).
43 &nbsp;Of course, an obvious answer to that need is that the server
44 simply closes the connection after having sent all responses (or all
45 that it is still capable of). The client will notice the closed
46 connection at latest when it tries to send new commands. However, it
47 may be worth considering an alternate mode where the server can issue
48 commands to the client. One potential way to do this would be to
49 reserve a specific txnr (0, for example) for this use. That txnr is
50 only to be used by the server and only to send "out-of-band" commands.
51 These commands must not be acknowledged by the client, because that
52 would complicate the protocol more than useful for this case. So
53 out-of-band-commands have a "hint" status - they hint the client about
54 something, but if they do not arrive at the client, nothing fatal
55 should happen. The ultimate session closure indication would still be
56 the tcp session close. [Please see the "High Latency Environment" use
57 case below for some implications this hinting process may have].</p>
58 <p>For the time being, it has been decided that such out of band
59 hints be not supported.</p>
60 <p>A command and its response is called a relp transaction. A
61 response must always carry the same txnr as the original command.
62 </p>
63 <p>If something goes really wrong, both the client and the server
64 may terminate
65 the TCP connection at any time. Any outstanding commands are considered
66 to
67 have been unsuccessful in this case.</p>
68 <h3>ERROR HANDLING</h3>
69 <p>
70 If something goes wrong (e.g. invalid framing), the peer that detects
71 the problem
72 closes the underlying (TCP) connection. It may send an error/abort
73 hint, but
74 MUST NOT wait for its response. The reasoning for this is that today's
75 Internet is
76 not a friendly place. Trying to gracefully resolve communications
77 problems may
78 lead to an attack vector. On the other hand, if the problem was of
79 temporary nature,
80 it may be cleared up by the connection reset and a new connect. So
81 aborting errored
82 connections is considered both secure as well as reasonable efficient.</p>
83 <p>Note: the ability to send abort/error messages depends on the
84 decision of out-of-band-hint-commands (see above).
85 </p>
86 <h3>SENDING MESSAGES</h3>
87 Because it is so important, I'd like to point it specifically out:
88 sending a
89 message is "just" another RELP command. The reply to that command is
90 the ACK/NAK
91 for the message. So every message *is* acknowledged. RELP options
92 indicate how
93 "deep" this acknowledge is (once we have implemented that), in the most
94 extreme
95 case a RELP client may ask a RELP server to ack only after the message
96 has been
97 completely acted upon (e.g. successfully written to database) - but
98 this is far
99 away in the future. For now, keep in mind that message loss will always
100 be detected
101 because we have app-level acknowledgement.
102 <h3>RELP FRAME</h3>
103 All relp transaction are carried out over a consistent framing.
104 <pre>RELP-FRAME = HEADER DATA TRAILER<br>DATA = [SP 1*OCTET] ; command-defined data, if DATALEN is 0, no data is present<br>HEADER = TXNR SP COMMAND SP DATALEN<br>TXNR = NUMBER ; relp transaction number, monotonically increases, starts at 1<br>DATALEN = NUMBER<br>#old:COMMAND = "open" / "syslog" / "close" / "rsp" / "abort" ; max length = 32<br>COMMAND = 1*32ALPHA<br>TRAILER = LF ; to detect framing errors and enhance human readibility<br>ALPHA = letter ; ('a'..'z', 'A'..'Z')<br>NUMBER = 1*9DIGIT<br>DIGIT = %d48-57<br>LF = %d10<br>SP = %d32<br><br>RSP DATA CONTENT:<br>RSP-HEADER = TXNR SP RSP-CODE [SP HUMANMSG] LF [CMDDATA]<br>RSP-CODE = 200 / 500 ; 200 is ok, all the rest currently erros<br>HUAMANMSG = *OCTET ; a human-readble message without LF in it<br>CMDDATA = *OCTET ; semantics depend on original command<br>TXNR is as in the relp frame, it is the TXNR of the frame being responded to.<br></pre>
105 <p>
106 DATALEN is the number of octets in DATA (so the frame length excluding
107 the length
108 of HEADER and TRAILER). Please note that the theoretical maximum
109 (999,999,999 octets)
110 is in (almost?) all cases unsuitable for actual message transfer. Thus,
111 the actual
112 maximum data length is negotiated during session setup. In relp version
113 1, it is
114 always 128K. In later versions, it shall be operator-configurable.
115 </p>
116 <p>TXNR 0 is reserved for hint commands. A RELP
117 connection start with TXNR 1. Note that TXNR montonically increases but
118 is latched at 999,999,999. After that TXNR, the next TXNR is
119 1.&nbsp;</p>
120 <h3>COMMAND SEMANTICS</h3>
121 Command can be either regular commands or hints. Regular commands are
122 responded to by a rsp package. They are transmitted on txnrs of 1 or
123 above. Hints are commands that are not being responded. They are always
124 transmitted with a txnr of zero. Regular commands can only be issued by
125 the client. Hints can be issued both by the client and the server.<br>
126 <h4>"Command" "rsp"</h4>
127 This is not a real command but a response to a client-issued command.
128 The TXNR MUST match the client's
129 command
130 TXNR. The data part contains RSP-HEADER as defined above. It is a
131 response code,
132 optionally followed by a space and additional data (depending on the
133 client's command). Return state values are: 200 - OK, 500 - error.<br>
134 <br>
135 The ABNF for a "rsp"'s &nbsp;data portion is as follows:<br>
136 <pre>RSPDATA = STATUS SP HUMAN-TEXT [LF DATA]<br>STATUS = 3DIGIT<br>HUMAN-TEXT = *80ALPHA<br>DATA = 1*OCTET<br></pre>
137 <h4>Command "open"</h4>
138 Opens a connection to the server. Must include offers inside the
139 data, at a minimum the "relp_version" offer. Offers
140 provide information about services supported by the client.
141 <p>When the server receives an open, it parses the offers, checks
142 what it itself supports
143 and provides those offers that it accepts in the "rsp". The server may
144 send a "rsp" with an empty offer part in case it doesn't like any of
145 the offers. To establish a session, a "relp_version" offer MUST be
146 included in the response. If it is not, the client MUST close the
147 connection. </p>
148 <p>When the client receives the "rsp", it checks the servers
149 offers and finally selects
150 those that should be used during the session. Please note that this
151 doesn't imply the
152 client selects e.g. security strength. To require a specific security
153 strength, the
154 server must be configured to offer only those options back to the
155 client that it is
156 happy to accept. So the client can only select from those. As such,
157 even though the
158 client makes the final feature selection, the server is dictating what
159 needs to be used.
160 </p>
161 <p>Note that the connection is only ready AFTER the client has
162 received&nbsp;the "open" response.</p>
163 <h4>Command "close"</h4>
164 Closes a connection to the server. Once the client has sent a "close"
165 command, it must not transmit any other command over the session. When
166 the server's rsp answer is received, the client must close the
167 connection.<br><h4>Hint Command "serverclose"</h4>This
168 hint SHOULD be sent by the server before it closes a RELP connection.
169 It advised the client that the server has closed the connection. The
170 client SHOULD use this knowledge to close the connection on the
171 client-side, too. Please note that this is a hint, and as such
172 transferred with a TXNR of 0 and without any response from the client.
173 Note that the client can not ask the server to defer closing of the
174 connection. A client MUST correctly process a server-closed connection
175 even if the server does not send a "serverclose" hint. In thus case, it
176 must rely on the transport layer connection status. The primary utility
177 of "serverclose" is to enable the client to quickly detect a
178 server-closed connection, clean up ressources and go into termination
179 (or recovery code).<h4>Command "starttls"</h4>
180 Does the same thing as ESMTP STARTTLS: switches the connection
181 to TLS mode.
182 <h4>Command "syslog"</h4>
183 This command is used to transmit a syslog message, which (in syslog
184 message format) is contained within the commands data portion.
185 <h3>OFFERS</h3>
186 During session setup, "offers" are exchange between client and server.
187 An "offer" describes
188 a specific feature or operation mode. Always present must be the
189 "relp_version" offer which
190 tells the other side which version of relp is in use.
191 <p>ABNF for offer strings
192 </p>
193 <pre>OFFER = LF FEATURENAME ["=" VALUE *("," VALUE)]<br>FEATURENAME = *32OCTET<br>VALUE = *255OCTET<br><br>Currently defined values:<br>FEATURENAME VALUE<br>relp_version 1 (this specification)<br></pre>
194 <h4>relp_version</h4>
195 <p>This offer describes the protocol version. It MUST be provided
196 in the
197 open command. It has one parameter, the numerical version id. The
198 intial version is 1. There also exists an experimental version 0, which
199 shall not be used in practice.
200 </p>
201 <h4>commands</h4>
202 <p>This offer describes which commands are supported by the
203 remote peer. Command that must be specified to drive the RELP protocol
204 itself (e.g. "open", "close", "rsp") MUST NOT be specified. Optional
205 commands must be specified. There may be multiple commands specified.</p>
206 <p>Sample: commands=syslog,eventlog (eventlog is a hypothetical,
207 not yet known command).</p>
208 <p>Please note that new commands may be specified outside of the
209 relp specification. Also note that if the server response does not
210 include the command the client is interested in, this command MUST NOT
211 be used. The client may then either resort to another, less featured
212 supported command or may terminate the relp connection.
213 </p>
214 <h4>relp_software</h4>
215 <p>This offer describes the software (library) that is processing
216 the relp
217 protocol. It is an optional offer. The idea is that when each peer
218 knows which software on the other peer is handling the protocol, it may
219 be able to work-around some known software issues. The "relp_software"
220 has two parameters, the first one being the name of the software as it
221 calls itself, the second one the version as it calls itself and the
222 third one being an URL where information about this software may be
223 found. It is suggested that all three parameters are given, but only
224 the first one MUST actually be provided.
225 </p>
226 <h3>STATE DIAGRAMS</h3>
227 <p>... detailling some communications scenarios:
228 </p>
229 <pre>Session Startup:<br>C S<br>cmd: "open", data: offer -----&gt; (selects supported offers)<br>(selects offers to use) &lt;----- cmd: "rsp", data "accepted offers"<br><br> ... transmission channel is ready to use ....<br><br>Message Transmission<br>C S<br>cmd: "syslog", data: syslogmsg -----&gt; (processes message)<br>(indicates syslog as processed) &lt;----- cmd: "rsp", data OK/Error<br><br>Session Termination<br>C S<br>cmd: "close", data: none? -----&gt; (processes termination request)<br>(terminates session) &lt;----- cmd: "rsp", data OK/Error<br> (terminates session)<br></pre>
230 <h3>SECURITY CONSIDERATIONS</h3>
231 <h4>Window Size</h4>
232 A very large window can be abused for denial of service attacks.
233 <h4>Maximum DATALEN</h4>
234 A too-large DATALEN can be absued for denial of service attacks.
235 <h2>Use Cases</h2>
236 These uses cases are not really a part of the
237 specification. I have added them to provide some insight into
238 implementation specifics. These use cases may be removed (or moved to
239 another part of the doc set) some time in the future. In the mean time,
240 the serve a valuable purpose when thinking about protocol features.
241 <h3>Server Session Closure with a Hibernated Client</h3>
242 I define this use case in support of <a href="http://www.rsyslog.com">rsyslog</a> and other
243 possible clients which try to reduce ressource consumption.<br>
244 <br>
245 Rsyslog utilizes highly threaded design, among others to take advantage
246 of multicore-machines. On
247 the other hand, it conserves resources by only running as many worker
248 threads as actually needed. So if there is no traffic, it may even be
249 running no workers at all. The RELP client will be implemented as an
250 output
251 plugin and will be executed on a worker thread. So in low-traffic
252 cases, the RELP client kind of hibernates, maybe even for hours (if no
253 new message is scheduled for transmission).
254 <p>In this use case, I examine a typical scenario in a
255 low-message volume case. Here, the client is asked to transmit a
256 message every now and than, at a pace of at most one message every few
257 seconds. It may also happen that no message is to be sent for several
258 minutes or even hours. Such a scenario is not as extreme as it sounds -
259 it may, for example, to be used for error notifications, which
260 hopefully happen very infrequently.</p>
261 <p>In this scenario, the client sends a command, but will usually
262 not receive an immediate response, because client processing is already
263 terminated when the server sends the response packet. Thus, the tcp
264 stacks receive buffer buffers the server's response until the client
265 has another message to send. Then, the client first checks for any
266 outstanding responses, processes them and then send the new message. It
267 is expected that in this scenario, each client invokation will process
268 the past response and send a new packet. The response to that packet
269 will not be processed by the invocation that sent the command but by
270 the next one. So we always have one outstanding response inside the OS
271 buffers. Now consider the server is shutting down the connection due to
272 timeout or due to the fact that it needs to shutdown itself (maybe even
273 on an urgent case, like power failure - the point is it can not wait).
274 On next client invokation, the client finds an outstanding response and
275 processes it. Then, it tries to send a new command. That will fail,
276 because the connection has been terminated. The usual session recovery
277 logic is used to restablish the connection when the server is back
278 online. The command in question is transferred after session
279 re-establishment.</p>
280 <h3>High Latency Environment</h3>
281 <p>We have a high latency environment that is otherwise quite
282 reliable
283 (satellite link). We have a high traffic load. To cover this, a large<br>
284 transmission
285 window has been selected (let's say 1000 messages). Now the client
286 sends a burst of messages but then&nbsp; has nothing to do and
287 falls
288 asleep. Then, the server starts sending acks. These acks are not taken
289 off the wire by the client (as it is inactive - rsyslog case).
290 Eventually, the tcp window fills. Thus the server can no longer send
291 acks. This does not immediately pose any problems, as the server adds
292 the ack frames to its internal send queue.</p>
293 <p>If the client comes
294 out of hibernation, it receives all previous sent frames, freeing the
295 server's tcp send window. Thus, the server can continue to send data
296 from its send queue (and drain it). The client, having received the
297 server's acks, will not stall because its own RELP window has been
298 cleared by the acks.</p>
299 <p>The situation is&nbsp; more complicated if
300 the server intends to shut down while the client is in hibernation. To
301 do so, the server usually sends a "adviseclose" command followed by a
302 "close" command. However, neither of these commands can be sent because
303 the client does not take them off the wire. The close operation is now
304 stalled. The only option left to the server is an unconditional
305 termination of the session. While everything that has already been sent
306 to the client will be picked up by it when it comes out of hibernation,
307 anything left in the server's send queue will be lost in this case. As
308 such, acks are potentially lost. This will lead to message duplication
309 as the client assumes that the frames unacked at time of force-close
310 where not processed (there is no other safe assumption). Consequently,
311 the client will re-send these frames in the next session.</p>
312 <p>A cure
313 to this situation is to have the client concurrently listen to server
314 requests, e.g. by running a receiver on a separate thread. The RELP
315 protocol does not demand this behaviour. However, it can be used to
316 solve the above server stall. Let's assume this is being done. Do we
317 now have a sufficient guarantee that the server does not stall? Under
318 normal conditions, it is a safe assumption that the client will be able
319 to receive all frames sent by the server. So a buildup of server queues
320 should, if at all, happen only for a short instant. However, if
321 something goes wrong on the transmission line (especially in
322 high-latency cases), there may be a somewhat extended period of time in
323 which the server can not send acks (but only in a magnitude of a few
324 seconds at most). If the server intends to shutdown during such a
325 period, a short timeout may enable it to avoid a fore-shutdown.
326 However, there are still cases thinkable where a force shutdown may be
327 required. These are deemed to be highly unlikely.</p>
328 <p>It is the
329 protocol implementer's choice if the slight less chance of a server
330 force-shutdown justifies the addition of a background thread to a
331 program that otherwise doesn't need one. From a protocol point of view,
332 a force-shutdown is a valid operation. Also, while it causes potential
333 message duplication, it can not cause message loss.</p>
334 <h2>Feedback and Discussion</h2>
335 <p>Feedback on this document and the RELP protocol is
336 appreciated. The <a href="http://lists.adiscon.net/mailman/listinfo/relp">RELP
337 mailing list</a> is probably a good place to provide it.
338 Questions are also happily accepted.</p>
339 <h2>Copyright</h2>
340 <p>Copyright (C) 2008, 2009 Rainer Gerhards and Adiscon.
341 <p>
342 Permission is granted to copy, distribute and/or modify this document
343 under the terms of the GNU Free Documentation License, Version 1.3
344 or any later version published by the Free Software Foundation;
345 with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
346 A copy of the license can be obtained from <a href="http://www.gnu.org/copyleft/fdl.html">http://www.gnu.org/copyleft/fdl.html</a> and is also included in the <a href="gfdl.html">license file</a> (if there is any difference in those two, the one contained in the license file is the one that this work is licensed under).
347 </body></html>