Debian Codegraph @

doc/dapl_end_point_design.txt

raw
1 #######################################################################
2 # #
3 # DAPL End Point Management Design #
4 # #
5 # Steve Sears #
6 # sjs2 at users.sourceforge.net #
7 # #
8 # 10/04/2002 #
9 # Updates #
10 # 02/06/04 #
11 # 10/07/04 #
12 # #
13 #######################################################################
14
15
16 ======================================================================
17 Referenced Documents
18 ======================================================================
19
20 uDAPL: User Direct Access Programming Library, Version 1.1. Published
21 05/08/2003. http://www.datcollaborative.org/uDAPL_050803.pdf.
22 Referred to in this document as the "DAT Specification".
23
24 InfiniBand Access Application Programming Interface Specification,
25 Version 1.2, 4/15/2002. In DAPL SourceForge repository at
26 doc/api/access_api.pdf. Referred to in this document as the "IBM
27 Access API Specification".
28
29 InfiniBand Architecture Specification Volume 1, Release 1.0.a Referred
30 to in this document at the "InfiniBand Spec".
31
32 ======================================================================
33 Introduction to EndPoints
34 ======================================================================
35
36 An EndPoint is the fundamental channel abstraction for the DAT API. An
37 application communicates and exchanges data using an EndPoint. Most of
38 the time EndPoints are explicitly allocated, but there is an exception
39 whereby a connection event can yield an EndPoint as a side effect; this
40 is not supported by all transports or implementations, but it is
41 supported in the InfiniBand reference implementation.
42
43 Each DAT API function is implemented in a file named
44
45 dapl_<function name>.c
46
47 There is a simple mapping provided by the dat library that maps dat_* to
48 dapl_*. For example, dat_pz_create is implemented in dapl_pz_create.c.
49 Other examples:
50
51 DAT DAPL Found in
52 ------------ --------------- ------------------
53 dat_ep_create dapl_ep_create dapl_ep_create.c
54 dat_ep_query dapl_ep_query dapl_ep_query.c
55
56 There are very few exceptions to this naming convention, the Reference
57 Implementation tried to be consistent.
58
59 There are also dapl_<object name>_util.{h,c} files for each object. For
60 example, there are dapl_pz_util.h and dapl_pz_util.c files which contain
61 common helper functions specific to the 'pz' subsystem. The use of util
62 files follows the convention used elsewhere in the DAPL reference
63 implementation. These files contain common object creation and
64 destruction code, link list manipulation, other helper functions.
65
66 This implementation has a simple naming convention designed to alert
67 someone reading the source code to the nature and scope of a
68 function. The convention is in the function name, such that:
69
70 dapl_ Primary entry from a dat_ function, e.g.
71 dapl_ep_create(), which mirrors dat_ep_create().
72 dapls_ The 's' restricts it to the subsystem, e.g. the
73 'ep' subsystem. dapls_ functions are not exposed
74 externally, but are internal to dapl.
75 dapli_ The 'i' restricts the function to the file where it
76 is declared. These functions are always 'static' C
77 functions.
78
79 This convention is not followed as consistently as we would like, but is
80 common in the reference implementation.
81
82 1. End Points (EPs)
83 -------------------------
84
85 DAPL End Points provide a channel abstraction necessary to transmit and
86 receive data. EPs interact with Service Points, either Public Service
87 Points or Reserved Service Points, to establish a connection from one
88 provider to another.
89
90 The primary EP entry points in the DAT API as they relate to DAPL are
91 listed in the following table:
92
93 dat_ep_create
94 dat_ep_query
95 dat_ep_modify
96 dat_ep_connect
97 dat_ep_dup_connect
98 dat_ep_disconnect
99 dat_ep_post_send
100 dat_ep_post_recv
101 dat_ep_post_rdma_read
102 dat_ep_post_rdma_write
103 dat_ep_get_status
104 dat_ep_reset
105 dat_ep_free
106
107 Additionally, the following connection functions interact with
108 EndPoints:
109 dat_psp_create
110 dat_psp_query
111 dat_psp_free
112 dat_rsp_create
113 dat_rsp_query
114 dat_rsp_free
115 dat_cr_accept
116 dat_cr_reject
117 dat_cr_query
118 dat_cr_handoff
119
120 The reference implementation maps the EndPoint abstraction onto an
121 InfiniBand Queue Pair (QP).
122
123 The DAPL_EP structure is used to maintain the state and components of
124 the EP object and the underlying QP. As will be explained below, keeping
125 track of the QP state is critical for successful operation. Access to
126 the DAPL_EP fields are done atomically.
127
128
129 ======================================================================
130 Goals
131 ======================================================================
132
133 Initial goals
134 -------------
135 -- Implement all of the dat_ep_* calls described in the DAT
136 Specification.
137
138 -- Implement connection calls described in the DAT Specification with
139 the following exception:
140 - dat_cr_handoff. This is best done with kernel mediation, and is
141 therefore out of scope for the reference implementation.
142
143 -- The implementation should be as portable as possible, to facilitate
144 HCA Vendors efforts to implement vendor-specific versions of DAPL.
145
146 -- The implementation must be able to work during ongoing development
147 of provider software agents, drivers, etc.
148
149 Later goals
150 -----------
151 -- Examine various possible performance optimizations. This document
152 lists potential performance improvements, but the specific
153 performance improvements implemented should be guided by customer
154 requirements.
155
156 ============================================
157 Requirements, constraints, and design inputs
158 ============================================
159
160 The EndPoint is the base channel abstraction. An Endpoint must be
161 established before data can be exchanged with a remote node. The
162 EndPoint is mapped to the underlying InfiniBand QP channel abstraction.
163 When a connection is initiated, the InfiniBand Connection Manager will
164 be solicited. The implementation is constrained by the capabilities and
165 behavior of the underlying InfiniBand facilities.
166
167 Note that transports other than InfiniBand may not need to rely on
168 Connection Managers or other infrastructure, this is an artifact of
169 this transport.
170
171 An EP is not an exact match to an InfiniBand QP, the differences
172 introduce constraints that are not obvious. There are three primary
173 areas of conflict between the DAPL and InfiniBand models:
174
175 1) EP and QP creation differences
176 2) Provider provided EPs on passive side of connections
177 3) Connection timeouts
178
179 -- EP and QP creation
180
181 The most obvious difference between an EP and a QP is the presence of a
182 memory handle when the object is created. InfiniBand requires a
183 Protection Domain (PD) be specified when a QP is created; in the DAPL
184 world, a Protection Zone (PZ) maps to an InfiniBand Protection Domain.
185 DAPL does not require a PZ to be present when an EP is created, and that
186 introduces two problems:
187
188 1) If a PZ is NULL when an EP is created, a QP will not be bound to
189 the EP until dat_ep_modify() is used to assign it later. A PZ is
190 required before RECV requests can be posted and before a connection
191 can be established.
192
193 2) If a DAPL user changes the PZ on an EP before it is connected,
194 DAPL must release the current QP and create a new one with a
195 new Protection Domain.
196
197 -- Provider provided EPs on connection
198
199 The second area where the DAPL and IB models conflict is a direct result
200 of the requirement to specify a Protection Domain when a QP is created.
201
202 DAPL allows a PSP to be created in such a way that an EP will
203 automatically be provided to the user when a connection occurs. This is
204 not critical to the DAPL model but in fact does provide some convenience
205 to the user. InfiniBand provides a similar mechanism, but with an
206 important difference: InfiniBand requires the user to supply the
207 Protection Domain for the passive connection endpoint that will be
208 supplied to all QPs created as a result of connection requests; DAPL
209 mandates a NULL PZ and requires the user to change the PZ before using
210 the EP.
211
212 The reference implementation creates an 'empty' EP when the user
213 specifies the DAT_PSP_PROVIDER flag; it is empty in the sense that a QP
214 is not attached to the EP. Before the user can dat_cr_accept the
215 connection, the EP must be modified to have a PZ bound to it, which in
216 turn will cause a QP to be bound to the EP.
217
218 To keep track of the current state of the EP, the DAPL_EP structure
219 has a qp_state field. The type of this field is specific to the
220 provider and the states are provider-specified states for a particular
221 transport, with the addition of a single state from dapl:
222 DAPL_QP_STATE_UNATTACHED, indicating that no QP has been bound to the
223 EP. The qp_state field is an open enumerator, containing a single DAPL
224 state in addition states specified by the provider.
225 DAPL_QP_STATE_UNATTACHED is randomly defined to be 0xFFF0, a
226 value selected strictly because it has the property that it will not
227 collide with provider states; if this is not true, this value must be
228 changed such that it is unique.
229
230 The common layer of DAPL only looks at this single value for qp_state,
231 it cannot be aware of states that are unique to the provider. However,
232 the provider layer is free to update this field and may use it as a
233 cache for current QP state. The field must be updated when a QP (or
234 other endpoint resource) is bound to the EP.
235
236 DAPL 1.2 provides DAT level states that will make this obsolete, but it
237 exists in pre DAPL 1.2 code.
238
239
240 -- Connection Timeouts
241
242 The third difference in the DAPL and InfiniBand models has to do with
243 timeouts on connections. InfiniBand does not provide a way to specify a
244 connection timeout, so it will wait indefinitely for a connection to
245 occur. dat_ep_connect supports a timeout value providing the user with
246 control over how long they are willing to wait for a connection to
247 occur.
248
249 DAPL maintains a timer thread to watch over pending connections. A
250 shared timer queue has a sorted list of timeout values. If a timeout
251 is requested, dapl_ep_connect() will invoke dapls_timer_set(), which
252 will add a timer record to the sorted list of timeouts. The timeout
253 thread is started lazily: that is, it isn't started until a timeout is
254 requested. Once a timeout has been requested, the thread will continue
255 to exist until the application terminates.
256
257 The timer record is actually a part of the DAPL_EP structure, so there
258 are no extra memory allocations required for timeouts. dapls_timer_set()
259 will initialize the timer record and insert it into the sorted queue at
260 the appropriate place. If this is the first record, or is inserted
261 before the first record (which will be the 'next' timeout to expire),
262 the timer thread will be awakened so it can recalculate how long it must
263 sleep until the timeout occurs.
264
265 When a timeout does occur, the timeout code will cancel the connection
266 request by invoking the provider routine dapls_ib_disconnect_clean().
267 This allows the software module with explicit knowledge of the provider
268 to take appropriate action and cancel the connection attempt. As a side
269 effect, the EP will be placed into the UNCONNECTED state, and the QP
270 will be in the ERROR state. A side effect of this state change is that
271 all DTOs will be flushed. The provider must support a mechanism to
272 completely cancel a connection request.
273
274
275 ======================================================================
276 DAPL EP Subsystem Design
277 ======================================================================
278
279 In section 6.5.1 of the DAT Specification there is a UML state
280 transition diagram for an EndPoint which goes over the transitions and
281 states during the lifetime of an EP. It is nearly impossible to read.
282 The reference implementation is faithful to the DAT Spec and is
283 believed to be correct.
284
285 This description of the EP will follow from creation to connection to
286 termination. It will also discuss the source code organization as this
287 is part of the design expression.
288
289 -- EP and QP creation
290
291 The preamble to creating an EP requires us to verify the attributes
292 specified by the user. If a user were to specify max_recv_dtos as 0, for
293 example, the EP would not be useful in any regard. If the user does not
294 provide EP attrs, the DAPL layer will supply a set of common defaults
295 resulting in a reasonable EP. The defaults are set up in
296 dapli_ep_default_attrs(), and the default values are given at the top of
297 dapl_ep_util.c. Non-InfiniBand transports will want to examine these
298 values to make sure they are 'reasonable'. This simplistic mechanism may
299 change in the future.
300
301 A number of handles are bound to the EP, so a reference count is taken
302 on each of them. All reference counts in the DAPL system are incremented
303 or decremented using atomic operations; it is important to always use
304 the OS dependent atomic routines and not substitute a lock, as it will
305 not be observed elsewhere in the system and will have unpredictable
306 results.
307
308 Reference counts are taken if there are non NULL values on any of:
309 pz_handle
310 connect_evd_handle
311 recv_evd_handle
312 request_evd_handle
313
314 The purpose of reference counts should be obvious: to prevent premature
315 release of resources that are still being used.
316
317 As has been discussed above, each EP is bound to a QP before it can be
318 connected. If a valid PZ is provided at creation time then a QP is bound
319 to the EP immediately. If the user later uses ep_modify() to change the
320 PZ, the QP will be destroyed and a new one created with the appropriate
321 Protection Domain.
322
323 Finally, an EP is an IA resource and is linked onto the EP chain of the
324 superior IA. EPs linked onto an IA are assumed to be complete, so this
325 is the final step of EP creation.
326
327 After an EP is created, the ep_state will be DAT_EP_STATE_UNCONNECTED
328 and the qp_state will either be DAPL_QP_STATE_UNATTACHED or assigned by
329 the provider layer (e.g.IB_QP_STATE_INIT). The qp_state indicates the QP
330 binding and the current state of the QP.
331
332 A qp_state of DAPL_QP_STATE_UNATTACHED indicates there is no QP bound
333 to this EP. This is a result of a NULL PZ when dat_ep_create() was
334 invoked, and which has been explained in detail above. The user must
335 call dat_ep_modify() and install a valid PZ before the EP can be used.
336
337 When an InfiniBand QP is created it is in the RESET state, which is
338 specified in the InfiniBand Spec, section 10.3. However, DAPL creates
339 the EP in the UNCONNECTED state and requires an unconnected EP to be
340 able to queue RECV requests before a connection occurs. The InfiniBand
341 spec allows RECV requests to be queued on an QP if the QP is in the INIT
342 state, so after creating a QP the DAPL provider code must transition it
343 to the INIT state.
344
345 There is a mapping between the DAPL EP state and the InfiniBand QP
346 state. DAPL_QP_STATE_UNATTACHED indicates the underlying QP is in the
347 INIT state. This is critical: RECV DTOs can be posted on an EP in the
348 UNATTACHED state, so the underlying QP must be in the appropriate state
349 to allow this to happen.
350
351 There is an obvious design tradeoff in transitioning the QP
352 state. Immediately moving the state to INIT takes extra time at creation
353 but allows immediate posting of RECV operations; however, it will
354 involve a more complex tear down procedure if the QP must be replaced as
355 a side effect of a dat_ep_modify operation. The alternative would be to
356 delay transitioning the QP to INIT until a post operation is invoked,
357 but that requires a run time check for every post operation. This design
358 assumes users will infrequently cause a QP to be replaced after it is
359 created and prefer to pay the state transition penalty at creation time.
360
361 -- EP Query and Modify operations
362
363 Because all of the ep_param data are kept up to date in the dapl_ep
364 structure, and because they use the complete DAT specified structure, a
365 query operation is trivial; a simple assignment from the internal
366 structure to the user parameter. uDAPL allows the implementation to
367 either return the fields specified by the user, or to return more than
368 the user requested; the reference implementation does the latter. It is
369 simpler and faster to copy the entire structure rather than to determine
370 which of all of the possible fields the user requested.
371
372 dat_ep_query() requires the implementation to report the address of the
373 remote node, if the EP is connected. This is different from standard
374 InfiniBand, if only because of the difference in name space. InfiniBand
375 has the information on the remote LID, but it does not have the remote
376 IP address, which is what DAT specifies. The reference implementation
377 makes use of a lookup/name-service called ATS ( Address Translation
378 Service), which is built using the InfiniBand Subnet Administrator. ATS
379 is InfiniBand only, other transports will use a different mechanism.
380
381 A driver will register itself and one or more IP addresses with ATS
382 at some point before a connection can be made. How the addresses are
383 provided to the driver, or how this is managed by the driver is not
384 specified. The ATS proposal is available from the DAT Collaborative.
385
386 When dat_ep_query() is invoked on a connected EP, it will request the
387 remote address from the provider layer. The provider layer will use
388 whatever means are necessary to obtain the IP address of the other end of
389 the connection. The results are placed into a buffer that is part of the
390 EP structure. Finally, the address of the EP structure is placed into
391 the ep_param.remote_ia_address_ptr field.
392
393 The ep_modify operation will modify the fields in the DAT_PARAM
394 structure. There are some fields that cannot be updated, and there are
395 others that can only be updated if the EP is in the correct state. The
396 uDAPL spec outlines the EP states permitting ep modifications, but
397 generally they are DAT_EP_STATE_UNCONNECTED and
398 DAT_EP_STATE_PASSIVE_CONNECTION_PENDING.
399
400 When replacing EVD handles it is a simple matter of releasing a
401 reference on the previous handle and taking a new reference on the new
402 handle. The Reference Implementation manages resource tracking using
403 reference counts, which guarantees a particular handle will not be
404 released prematurely. Reference counts are checked in the free routines
405 of various objects.
406
407 As has been mentioned previously, if the PZ handle is changed then the
408 QP must be released, if already assigned, and a new QP must be created
409 to bind to this EP.
410
411 There are some fields in the DAT_PARAM structure that are related to the
412 underlying hardware implementation. For these values DAPL will do a
413 fresh query of the QP, rather than depend on stale values. Even so, the
414 values returned are 'best effort' as a competing thread may change
415 certain values before the requesting thread has the opportunity to read
416 them. Applications should protect against this.
417
418 Finally, the underlying provider is invoked to update the QP with new
419 values, but only if some of the attributes have been changed. As is
420 true of most of the implementation, we only invoke the provider code
421 when necessary.
422
423 ======================================================================
424 Connections
425 ======================================================================
426
427 There are of course two sides to a connection, and in the DAPL model
428 there is an Active and a Passive side. For clarity, the Passive side
429 is a server waiting for a connection, and the Active side is a client
430 requesting a connection from the Passive server. We will discuss each
431 of these in turn.
432
433 Connections happen in the InfiniBand world by using a Connection Manager
434 (CM) interface. Those unfamiliar with the IB model of addressing and
435 management agents may want to familiarize themselves with these aspects
436 of the IB spec before proceeding in this document. Be warned that the
437 connection section of the IB spec is the most ambiguous portion of the
438 spec.
439
440 First, let's walk through a primitive diagram of a connection:
441
442
443 SERVER (passive) CLIENT (active)
444 --------------- ---------------
445 1. dapl_psp_create
446 or dapl_rsp_create
447 [ now listening ]
448
449 2. dapl_ep_connect
450 <-------------
451 3. dapls_cr_callback
452 DAT_CONNECTION_REQUEST_EVENT
453 [ Create and post a DAT_CONNECTION_REQUEST_EVENT event ]
454
455 4. Event code processing
456
457 5. Create an EP if necessary
458 (according to the flags
459 when the PSP was created)
460
461 6. dapl_cr_accept or dapl_cr_reject
462 ------------->
463 7. dapl_evd_connection_callback
464 DAT_CONNECTION_EVENT_ESTABLISHED
465 [ Create and post a
466 DAT_CONNECTION_EVENT_ESTABLISHED
467 event ]
468
469 8. <------------- RTU
470
471 9. dapls_cr_callback
472 DAT_CONNECTION_EVENT_ESTABLISHED
473 [ Create and post a DAT_CONNECTION_EVENT_ESTABLISHED
474 event ]
475
476 10. ...processing...
477
478 11. Either side issues a dat_ep_disconnect
479
480 12. dapls_cr_callback
481 DAT_CONNECTION_EVENT_DISCONNECTED
482
483 [ Create and post a
484 DAT_CONNECTION_EVENT_DISCONNECTED
485 event ]
486
487 13. dapl_evd_connection_callback
488 DAT_CONNECTION_EVENT_DISCONNECTED
489 [ Create and post a
490 DAT_CONNECTION_EVENT_DISCONNECTED
491 event ]
492
493
494 In the above diagram, time is numbered in the left hand column and is
495 represented vertically.
496
497 We will continue our discussion of connections using the above diagram,
498 following a sequential order for connection establishment.
499
500 There are in fact two types of service points detailed in the uDAPL
501 specification. We will limit our discussion to PSPs for convenience, but
502 there are only minor differences between PSPs and RSPs.
503
504 The reader should observe that all passive-side connection events will
505 be received by dapls_cr_callback(), and all active side connection
506 events occur through dapl_evd_connection_callback(). At one point during
507 the implementation these routines were combined as they are very
508 similar, but there are subtle differences causing them to remain
509 separate.
510
511 Progressing through the series of events as outlined in the diagram
512 above:
513
514 1. dapl_psp_create
515
516 When a PSP is created, the final act will be to set it listening for
517 connections from remote nodes. It is important to realize that a
518 connection may in fact arrive from a remote node before the routine
519 setting up a listener has returned to dapl_psp_create; as soon as
520 dapls_ib_setup_conn_listener() is invoked connection callbacks may
521 arrive. To reduce race conditions this routine must be called as the
522 last practical operation when creating a PSP.
523
524 dapls_ib_setup_conn_listener() is provider specific. The key insight
525 is that the DAPL connection qualifier (conn_qual) will become the
526 InfiniBand Service ID. The passive side of the connection is now
527 listening for connection requests. It should be obvious that the
528 conn_qual must be unique.
529
530 InfiniBand allows a 64 bit connection qualifier, which is supported
531 by the DAT spec. IP based networks may be limited to 16 bits, so
532 provider implementations may want to return an error if it exceeds
533 the maximum allowable by the transport.
534
535 2. dapl_ep_connect
536
537 The active side initiates a connection with dapl_ep_connect, which
538 will transition the EP into DAT_EP_STATE_ACTIVE_CONNECTION_PENDING.
539 Again, connections are in the domain of the providers' Connection
540 Manager and the mechanics are very much provider specific. The key
541 points are that a DAT_IA_ADDRESS_PTR must be translated to a GID
542 before a connection initiation can occur. This is discussed below.
543
544 InfiniBand supports different amounts of private data on various
545 connection functions. Other transports allow variable sizes of
546 private data with no practical limit.The DAPL connection code does
547 not enforce a fixed amount of private data, but rather makes
548 available to the user all it has available, as specified by
549 DAPL_MAX_PRIVATE_DATA_SIZE.
550
551 Private data will be stored in a fixed buffer as part of the
552 connection record, which is the primary reason to limit the size.
553
554 To assist development on new transports that do not have a full
555 connection infrastructure in place, there are a couple of compile time
556 flags that will include certain code: CM_BUSTED and
557 IBOSTS_NAMING. These are discussed below in more detail, but
558 essentially:
559
560 CM_BUSTED: fakes a connection on both sides of the wire, does not
561 transmit any private data.
562
563 IBHOSTS_NAMING: provides a simple IP_ADDRESS to LID translation
564 mechanism in a text file, which is read when the dapl library
565 loads. Private data is exchanged in this case, but it includes a
566 header that contains the remote IP address. Technically, this defines
567 a protocol and is in violation of the DAT spec, but it has proved
568 useful in development.
569
570 3. dapls_cr_callback
571
572 The connection sequence is entirely event driven. An operation is
573 posted, then an asynchronous event will occur some time later. The
574 event may cause other actions to occur which may result in still
575 more events.
576
577 dapls_ib_setup_conn_listener() registered for a callback for
578 connection events, and we now receive a DAT event for a connection
579 request. The provider layer will translate the native event type to
580 a DAT event.
581
582 An upcall is invoked on the server side of the connection with an
583 event of type DAT_CONNECTION_REQUEST_EVENT. This is a unique event
584 in the callback code as it is the only case when an EP is not
585 already in play; in all other cases, it is possible to look up the
586 relevant EP for an operation.
587
588 Code exists to make sure the relevant connection object, the PSP or
589 RSP, is actually in a useful state and ready to be connected
590 to. One of the critical differences between a PSP and an RSP is
591 that an RSP is a one-shot connection object; once a connection
592 occurs, no other connections can be made to it.
593
594 There is a small difference in the InfiniBand and DAPL connection
595 models here as well. DAPL may disable a PSP at any time without
596 affecting current connections. When you tear down an InfiniBand
597 service endpoint, all of the connections are torn down too. Because
598 of this difference, when a DAPL app frees a PSP, only a state
599 change is made. The underlying service point is still available and
600 technically capable of receiving connections. If a connection
601 request arrives when the PSP is in this state, a rejection message
602 is sent such that the requesting node believes no service point is
603 listening.
604
605 Once the connection has been examined, it will continue with the
606 connection protocol. The EP will move to a CONNECTION_PENDING
607 state.
608
609 The connection request will cause a CR record to be allocated,
610 which holds all of the important connection request
611 information. The CR record will be linked onto the PSP structure
612 for retrieval in the future when other requests arrive.
613
614 The astute reader of the spec will observe that there is not a
615 dapl_cr_create call: CR records are created as part of a connection
616 attempt on the passive side of the connection. A CR is created now
617 and set up. A point that will become important later, caps for
618 emphasis:
619
620 A CR WILL EXIST FOR THE LIFE OF A CONNECTION; THEY ARE DESTROYED AT
621 DISCONNECT TIME.
622
623 In the connection request processing a CR and an EVENT are created,
624 the event will be posted along with the connection information just
625 received.
626
627 Private data is also copied into the CR record. Private data
628 arrived with the connection request and is not a permanent
629 resource, so it is copied into the dapl space to be used at a later
630 time. Different transports have varying capabilities on the size of
631 private data, so a call to the provider is invoked to determine how
632 big it actually is. There is an upper bound on the amount of
633 private data the implementation will deal with, set at
634 DAPL_MAX_PRIVATE_DATA_SIZE (256 bytes at this writing).
635
636 4. Event code processing
637
638 The final stage in a connection request is to generate an event on
639 a connection EVD using dapls_evd_post_cr_arrival_event().
640
641 5. Create an EP if necessary
642
643 When the app processes a connection event, it needs to respond. If
644 the PSP is configured to create an EP automatically, the callback
645 code has already done it; creating an EP with no attached QP. Else,
646 the user must provide an EP to make the connection.
647
648 (4) and (5) are all done in user mode. The only interesting thing is
649 that when the user calls dat_cr_accept(), a ready EP must be
650 provided. If the EP was supplied by the PSP in the callback, it
651 must have a PZ associated with it and whatever other attributes
652 need to be set.
653
654 6. dapl_cr_accept or dapl_cr_reject
655
656 For discussion purposes, we will follow the accept
657 path. dapl_cr_reject says you are done and there will be no further
658 events to deal with.
659
660 Assuming it accepts the connection for our example, the dapl code
661 will verify that an EP is in place and will deal with private data
662 that can be transmitted in a cr_accept call. The underlying
663 provider is invoked to complete this leg of the protocol.
664
665 7. dapl_evd_connection_callback
666
667 An EVD callback is always a response to a connection oriented
668 request. As such, an EP is always present, and in fact is passed
669 into the upcall as the 'context' argument.
670
671 Connection requests may take an arbitrary amount of time, so the EP
672 is always checked for a running timer when the upcall is made. As
673 has been discussed above, if a timer expires before an upcall
674 occurs, the connection must be completely canceled such that there
675 is no upcall.
676
677 The event signifying completion of the connection is
678 DAT_CONNECTION_EVENT_ESTABLISHED, and it will move the EP to the
679 CONNECTED state and post this event on the connection EVD. Private
680 data will be copied to an area in the EP structure, which is
681 persistent.
682
683 At this point, the EP is connected and the application is free to
684 post DTOs.
685
686 8i. RTU
687
688 This item is labeled "8i" as it is internal to the InfiniBand
689 implementation, it is not initiated by dapl. The final leg of a
690 connection is an RTU sent from the initiating node to the server
691 node, indicating the connection has been made successfully.
692
693 Other transports may have a different connection protocol.
694
695 9. dapls_cr_callback
696
697 When the RTU arrives, an upcall is invoked with a
698 DAT_CONNECTION_EVENT_ESTABLISHED event, which will be posted to the
699 connection EVD event queue. The EP is moved to the CONNECTED
700 state.
701
702 There is no private data for dapl to deal with, even though some
703 transports may provide private data at each step of a connection.
704
705 The connection activity is occurring on a separate channel than the
706 EP, so this is inherently a racy operation. The correct
707 application will always post RECV buffers on an EP before
708 initiating a connection sequence, as it is entirely possible for
709 DTOs to arrive *before* the final connection event arrives.
710
711 The architecturally interesting feature of this exchange occurs
712 because of differences in the InfiniBand and the DAT connection
713 models, which are briefly outlined here.
714
715 InfiniBand maintains the original connecting objects throughout the
716 life of the connection. That is, we originally get a callback event
717 associated with the Service (DAT PSP) that is listening for
718 connection events. A QP will be connected but the callback event
719 will still be received on the Service. Later, a callback event will
720 occur for a DISCONNECT, and again the Service will be the object of
721 the connection. In the DAPL implementation, the Service will
722 provide the PSP that is registered as listening on that connection
723 qualifier.
724
725 The difference is that DAT has a PSP receive a connection event,
726 but subsequently hands all connection events off to an EP. After a
727 dat_cr_accept is issued, all connection/disconnection events occur
728 on the EP. DAT more closely follows the IP connection model.
729
730 To support the DAT model, a CR is maintained through the life of
731 the connection. There is exactly one CR per connection, but any
732 number of CRs may exist for any given PSP. CRs are maintained on a
733 linked list pointed to by the PSP structure. A lookup routine will
734 match the cm_handle, unique for each connection, with the
735 appropriate CR. This allows us to find the appropriate EP which
736 will be used to create an event to be posted to the user.
737
738 * dat_psp_destroy
739
740 It should be understood that the PSP will maintain all of the CR
741 records, and hence the PSP must persist until the final disconnect.
742 In the DAT model there is no association between a PSP and a
743 connected QP, so there is no reason not to destroy a PSP before the
744 final disconnect.
745
746 Because of the model mismatch we must preserve the PSP until the
747 final disconnect. If the user invokes dat_psp_destroy(), all of the
748 associations maintained by the PSP will be severed; but the PSP
749 structure itself remains as a container for the CR records. The PSP
750 structure maintains a simple count of CR records so we can easily
751 determine the final disconnect and release memory. Once a
752 disconnect event is received for a specific cm_handle, no further
753 events will be received and it is safe to discard the CR record.
754
755 10. ...processing...
756
757 This is just a place holder to show that applications actually do
758 something after making a connection. They might not too...
759
760 11. Either side issues a dat_ep_disconnect
761
762 dat_ep_disconnect() can be initiated by either side of a
763 connection. There are two kinds of disconnect flags that can be
764 passed in, but the final result is largely the same.
765
766 DAT_CLOSE_ABRUPT_FLAG will cause the connection to be immediately
767 terminated. In InfiniBand terms, the QP is immediately moved to the
768 ERROR state, and after some time it will be moved to the RESET
769 state.
770
771 DAT_CLOSE_GRACEFUL_FLAG will allow in-progress DTOs to complete.
772 The underlying implementation will first transition the QP to the
773 SQE state, before going to RESET.
774
775 Both cases are handled by the underlying CM, there is no extra work
776 for DAPL.
777
778 12. dapls_cr_callback
779
780 A disconnect will arrive on the passive side of the connection
781 through dapls_cr_callback() with connection event
782 DAT_CONNECTION_EVENT_DISCONNECTED. With this event the EP lookup
783 code will free the CR associated with the connection, and may free
784 the PSP if it is no longer listening, indicating it has been freed
785 by the application.
786
787 The callback will create and post a
788 DAT_CONNECTION_EVENT_DISCONNECTED event for the application.
789
790 13. dapl_evd_connection_callback
791
792 The active side of the connection will receive
793 DAT_CONNECTION_EVENT_DISCONNECTED as the connection event for
794 dapl_evd_connection_callback(), and will create and post a
795 DAT_CONNECTION_EVENT_DISCONNECTED event. Other than transitioning
796 the EP to the DISCONNECTED state, there is no further processing.
797
798
799 Observe that there are a number of exception conditions resulting in a
800 disconnect of the EP, most of which will generate unique DAT events
801 for the application to deal with.
802
803
804 * Addressing and Naming
805
806 The DAT Spec calls for a DAT_IA_ADDRESS_PTR to be an IP address,
807 either IPv4 or IPv6. It is in fact a struct sockaddr in most
808 systems. The dapl structures typically use IPv6 data types to
809 accommodate the largest possible addresses, but most implementations
810 use IPv4 formatted addresses.
811
812 InfiniBand uses a transport specific address known as a LID, which
813 typically is dynamically assigned by a Subnet Manager. Each HCA
814 also has a global address, similar to an Ethernet MAC address,
815 known as a GUID. ATS, mentioned above, is a mechanism using
816 InfiniBand infrastructure to map from GUID/LID to IP addresses. It
817 is not necessary for transports that use IP addresses natively,
818 such as Ethernet devices.
819
820 If a new implementation does not yet have a name service
821 infrastructure, the DAPL implementation provides a simple name
822 service facility under the #ifdef NO_NAME_SERVICE. This depends on
823 two things: valid IP addresses registered and available to standard
824 DNS system calls such as gethostbyname(); and a name/GID mapping
825 file.
826
827 IP addresses may be set up by system administrators or by a local
828 power-user simply by editing the values into the /etc/hosts file.
829 Setting IP addresses up in this manner is beyond the scope of this
830 document.
831
832 A simple mapping of names to GIDs is maintained in the ibhosts
833 file, currently located at /etc/dapl/ibhosts. The format of
834 the file is:
835
836 <IP name> 0x<GID Prefix> 0x<GUID>
837
838 For example:
839
840 dat-linux3-ib0p0 0xfe80000000000000 0x0001730000003d11
841 dat-linux3-ib0p1 0xfe80000000000000 0x0001730000003d11
842 dat-linux3-ib1 0xfe80000000000000 0x0001730000003d52
843 dat-linux5-ib0 0xfe80000000000000 0x0001730000003d91
844
845 And for each hostname, there must be an entry in the /etc/hosts file
846 similar to:
847
848 dat_linux3-ib0a 198.165.10.11
849 dat_linux3-ib0b 198.165.10.12
850 dat_linux3-ib1a 198.165.10.21
851 dat_linux5-ib0a 198.165.10.31
852
853
854 In this example we have adopted the convention of naming each
855 InfiniBand interface by using the form
856
857 <node_name>-ib<device_number>[port_number]
858
859 In the above example we can see that the machine dat_linux3 has three
860 InfiniBand interfaces, which in this case we have named two ports on
861 the first HCA and another port on a second. Utilizing standard DNS
862 naming, the conventions used for identifying individual ports is
863 completely up to the administrator.
864
865 The GID Prefix and GUID are obtained from the HCA and map a port on
866 the HCA: together they form the GID that is required by a CM to
867 connect with the remote node.
868
869 The simple name service builds an internal table after processing
870 the ibhosts file which contains IP addresses and GIDs. It will use
871 the standard getaddrinfo() function to obtain IP address
872 information.
873
874 When an application invoked dat_ep_connect(), the
875 DAT_IA_ADDRESS_PTR will be compared in the table for a match and
876 the destination GID established if found. If the address is not
877 found then the user must first add the name to the ibhosts file.
878
879 With a valid GID for the destination node, the underlying CM is
880 invoked to make a connection.
881
882 * Connection Management
883
884 Getting a working CM has taken some time, in fact the DAPL project
885 was nearly complete by the time a CM was available. In order to
886 make progress, a connection hack was introduced that allows
887 specific connections to take place. This is noted in the code by
888 the CM_BUSTED #def.
889
890 CM_BUSTED takes the place of a CM and will manually transition a QP
891 through the various states to connect: INIT->RTR->RTS. It will also
892 disconnect the connection, although the Torrent implementation
893 simply destroys the QP and recreates a new one rather than
894 transitioning through the typical disconnect states (which didn't
895 work on early IB implementations).
896
897 CM_BUSTED makes some assumptions about the remote end of the
898 connection as no real information is exchanged. The ibapi
899 implementation assumes both HCAs have the same LID, which implies
900 there is no SM running. The vapi implementation assumes the LIDs
901 are 0 and 1. Depending on the hardware, the LID value may in fact
902 not make any difference. This code does not set the Global Route
903 Header (GRH), which would cause the InfiniBand chip to be carefully
904 checking LID information.
905
906 The QP number is assumed to be identical on both ends of the
907 connection, or differing by 1 if this is a loopback. There is an
908 environment variable that will be read at initialization time if
909 you are configured with a loopback, this value is checked when
910 setting up a QP. The obvious downside to this scheme is that
911 applications must stay synchronized in their QP usage or the
912 initial exchange will fail as they are not truly connected.
913
914 Add to this the limitation that HCAs must be connected in
915 Point-to-Point topology or in a loopback. Without a GRH it will not
916 work in a fabric. Again, using an SM will not work when CM_BUSTED
917 is enabled.
918
919 Despite these shortcomings, CM_BUSTED has proven very useful and
920 will remain in the code for a while in order to aid development
921 groups with new hardware and software. It is a hack to be sure, but
922 it is relatively well isolated.
923
924
925 -- Notes on Disconnecting
926
927 An EP can only be disconnected if it is connected or unconnected; you
928 cannot disconnect 'in progress' connections. An 'in progress
929 connection may in fact time out, but the DAT Spec does not allow you
930 to 'kill' it. DAPL will use the CM interface to disconnect from the
931 remote node; this of course results in an asynchronous callback
932 notifying the application the disconnect is complete.
933
934 Disconnecting an unconnected EP is currently the only way to remove
935 pending RECV operations from the EP. The DAPL spec notes that all
936 DTOs must be removed from an EP before it can be deallocated, yet
937 there is no explicit interface to remove pending RECV DTOs. The user
938 will disconnect an unconnected EP to force the pending operations off
939 of the queue, resulting in DTO callbacks indicating an error. The
940 underlying InfiniBand implementation will cause the correct behavior
941 to result. When doing this operation the DAT_CLOSE flag is ignored,
942 DAPL will instruct the provider layer to abruptly disconnect the QP.
943
944 As has been noted previously, specifying DAT_CLOSE_ABRUPT_FLAG as the
945 disconnect completion flag will cause the CM implementation to
946 transition the QP to the ERROR state to abort all operations, and then
947 transition to the RESET state; if the flag is DAT_CLOSE_GRACEFUL_FLAG,
948 the CM will first move to the SQE state and allow all pending I/O's to
949 drain before moving to the RESET state. In either case, DAPL only
950 needs to know that the QP is now in the RESET state, as it will need
951 to be transitioned to the INIT state before it can be used again.
952
953 ======================================================================
954 Data Transfer Operations (DTOs)
955 ======================================================================
956
957 The DTO code is a straightforward translation of the DAT_LMR_TRIPLET
958 to an InfiniBand work request. Unfortunately, IB does not specify what
959 a work request looks like so this tends to be very vendor specific
960 code. Each provider will supply a routine for this operation.
961
962 InfiniBand allows the DTO to attach a unique 64 bit work_req_id to
963 each work request. The DAPL implementation will install a pointer to a
964 DAPL_DTO_COOKIE in this field. Observe that a DAPL_DTO_COOKIE is not
965 the same as the user DAT_DTO_COOKIE; indeed, the former has a pointer
966 field pointing to the latter. Different values will be placed in the
967 cookie, according to the type of operation it is and the type of data
968 required by its completion event. This is a simple scheme to bind DAPL
969 data to the DTO and associated completion callback. Each DTO has a
970 unique cookie associated with it.
971
972 Observe that an InfiniBand work_request remains under control of the
973 user, and when a post operation occurs the underlying implementation
974 will copy data out of the work_request into a hardware based
975 structure. Further, no application can perform a DTO operation on the
976 same EP at the same time according to the thread guarantees mandated
977 by the specification. This allows us to provide a recv_iov and a
978 send_iov in the EP structure for all DTO operations, eliminating any
979 malloc operations from this critical path.
980
981 The underlying provider implementation will invoke
982 dapl_evd_dto_callback() upon completion of DTO operations.
983 dapl_evd_dto_callback() is the asynchronous completion for a DTO and
984 will create and post an event for the user. Much of this callback is
985 concerned with managing error completions.
986
987
988 ======================================================================
989 Data Structure
990 ======================================================================
991
992 The main data structure for an EndPoint is the dapl_ep structure,
993 defined in include/dapl.h. The reference implementation uses the
994 InfiniBand QP to maintain hardware state, providing a relatively
995 simple mapping.
996
997 /* DAPL_EP maps to DAT_EP_HANDLE */
998 struct dapl_ep
999 {
1000 DAPL_HEADER header;
1001 /* What the DAT Consumer asked for */
1002 DAT_EP_PARAM param;
1003
1004 /* The RC Queue Pair (IBM OS API) */
1005 ib_qp_handle_t qp_handle;
1006 unsigned int qpn; /* qp number */
1007 ib_qp_state_t qp_state;
1008
1009 /* communications manager handle (IBM OS API) */
1010 ib_cm_handle_t cm_handle;
1011 /* store the remote IA address here, reference from the param
1012 * struct which only has a pointer, no storage
1013 */
1014 DAT_SOCK_ADDR6 remote_ia_address;
1015
1016 /* For passive connections we maintain a back pointer to the CR */
1017 void * cr_ptr;
1018
1019 /* pointer to connection timer, if set */
1020 struct dapl_timer_entry *cxn_timer;
1021
1022 /* private data container */
1023 DAPL_PRIVATE private;
1024
1025 /* DTO data */
1026 DAPL_ATOMIC req_count;
1027 DAPL_ATOMIC recv_count;
1028
1029 DAPL_COOKIE_BUFFER req_buffer;
1030 DAPL_COOKIE_BUFFER recv_buffer;
1031
1032 ib_data_segment_t *recv_iov;
1033 DAT_COUNT recv_iov_num;
1034
1035 ib_data_segment_t *send_iov;
1036 DAT_COUNT send_iov_num;
1037 #ifdef DAPL_DBG_IO_TRC
1038 int ibt_dumped;
1039 struct io_buf_track *ibt_base;
1040 DAPL_RING_BUFFER ibt_queue;
1041 #endif /* DAPL_DBG_IO_TRC */
1042 };
1043
1044 The simple explanation of the fields in the dapl_ep structure follows:
1045
1046 header: The dapl object header, common to all dapl objects.
1047 It contains a lock field, links to appropriate lists, and
1048 handles specifying the IA domain it is a part of.
1049
1050 param: The bulk of the EP attributes called out in the DAT
1051 specification and are maintained in the DAT_EP_PARAM
1052 structure. All internal references to these fields
1053 use this structure.
1054
1055 qp_handle: Handle to the underlying InfiniBand provider implementation
1056 for a QP. All EPs are mapped to an InfiniBand QP.
1057
1058 qpn: Number of the QP as returned by the underlying provider
1059 implementation. Primarily useful for debugging.
1060
1061 qp_state: Current state of the QP. The values of this field indicate
1062 if a QP is bound to the EP, and the current state of a
1063 QP.
1064
1065 cm_handle: Handle to the IB provider's CMA (Connection Manager Agent).
1066 Used for CM operations used to connect and disconnect.
1067
1068 remote_ia_address:
1069 Remote IP address of the connection. Only valid after the user
1070 has asked for it.
1071
1072 cr_ptr: Attaches the EP to the appropriate CR. Assigned on the passive
1073 side of a connection in cr_accept. It is used when an abrupt
1074 disconnect is invoked by the app, and we need to 'fake' a
1075 callback. It is also used in clean up of an EP and removing
1076 connection elements from the associated PSP.
1077
1078 cxn_timer: Pointer to a timer entry, used as a token to set and remove
1079 timers.
1080
1081 private: Local Private data area on the active side of a connection.
1082
1083 req_count: Count of outstanding request DTO operations, including memory
1084 ops. Atomically incremented/decremented.
1085
1086 recv_count:Count of outstanding receive DTO operations. Atomically
1087 incremented/decremented.
1088
1089 req_buffer:Ring buffer of request cookies.
1090
1091 recv_buffer:
1092 Ring buffer of receive cookies.
1093
1094 recv_iov: Storage for provider receive work request.
1095
1096 recv_iov_num:
1097 Maximum number of receive IOVs. Number is obtained from
1098 the provider in a query.
1099
1100 send_iov: Storage for provider send work request.
1101
1102 send_iov_num:
1103 Maximum number of send IOVs. Number is obtained from the
1104 provider in a query.
1105
1106 ibt_dumped:DTO debugging aid. Boolean value to control how often DTO
1107 tracing data is printed.
1108
1109 ibt_base: DTO debugging aid. Base address of DTO ring buffer containing
1110 information on DTO processing.
1111
1112 ibt_queue: Ring buffer containing information n DTO processing.
1113
1114
1115 ** Debug
1116
1117 The Reference Implementation includes a trace facility that allows a
1118 developer to see all DTO operations, specifically to catch those that
1119 are not completing as expected. The DAPL_DBG_IO_TRC conditional will
1120 enable this code.
1121
1122 A simple ring buffer is used to account for all outstanding DTO
1123 traffic. The buffer may be dumped when DTOs are not getting
1124 completions, with enough data to aid the developer to determine where
1125 things went wrong.
1126
1127 It is implemented as a ring buffer as there are often bugs in this
1128 part of a provider's implementation which do not manifest until
1129 intensive data exchange has occurred for many hours.