| 1 |
####################################################################### |
| 2 |
# # |
| 3 |
# DAPL End Point Management Design # |
| 4 |
# # |
| 5 |
# Steve Sears # |
| 6 |
# sjs2 at users.sourceforge.net # |
| 7 |
# # |
| 8 |
# 10/04/2002 # |
| 9 |
# Updates # |
| 10 |
# 02/06/04 # |
| 11 |
# 10/07/04 # |
| 12 |
# # |
| 13 |
####################################################################### |
| 14 |
|
| 15 |
|
| 16 |
====================================================================== |
| 17 |
Referenced Documents |
| 18 |
====================================================================== |
| 19 |
|
| 20 |
uDAPL: User Direct Access Programming Library, Version 1.1. Published |
| 21 |
05/08/2003. http://www.datcollaborative.org/uDAPL_050803.pdf. |
| 22 |
Referred to in this document as the "DAT Specification". |
| 23 |
|
| 24 |
InfiniBand Access Application Programming Interface Specification, |
| 25 |
Version 1.2, 4/15/2002. In DAPL SourceForge repository at |
| 26 |
doc/api/access_api.pdf. Referred to in this document as the "IBM |
| 27 |
Access API Specification". |
| 28 |
|
| 29 |
InfiniBand Architecture Specification Volume 1, Release 1.0.a Referred |
| 30 |
to in this document at the "InfiniBand Spec". |
| 31 |
|
| 32 |
====================================================================== |
| 33 |
Introduction to EndPoints |
| 34 |
====================================================================== |
| 35 |
|
| 36 |
An EndPoint is the fundamental channel abstraction for the DAT API. An |
| 37 |
application communicates and exchanges data using an EndPoint. Most of |
| 38 |
the time EndPoints are explicitly allocated, but there is an exception |
| 39 |
whereby a connection event can yield an EndPoint as a side effect; this |
| 40 |
is not supported by all transports or implementations, but it is |
| 41 |
supported in the InfiniBand reference implementation. |
| 42 |
|
| 43 |
Each DAT API function is implemented in a file named |
| 44 |
|
| 45 |
dapl_<function name>.c |
| 46 |
|
| 47 |
There is a simple mapping provided by the dat library that maps dat_* to |
| 48 |
dapl_*. For example, dat_pz_create is implemented in dapl_pz_create.c. |
| 49 |
Other examples: |
| 50 |
|
| 51 |
DAT DAPL Found in |
| 52 |
------------ --------------- ------------------ |
| 53 |
dat_ep_create dapl_ep_create dapl_ep_create.c |
| 54 |
dat_ep_query dapl_ep_query dapl_ep_query.c |
| 55 |
|
| 56 |
There are very few exceptions to this naming convention, the Reference |
| 57 |
Implementation tried to be consistent. |
| 58 |
|
| 59 |
There are also dapl_<object name>_util.{h,c} files for each object. For |
| 60 |
example, there are dapl_pz_util.h and dapl_pz_util.c files which contain |
| 61 |
common helper functions specific to the 'pz' subsystem. The use of util |
| 62 |
files follows the convention used elsewhere in the DAPL reference |
| 63 |
implementation. These files contain common object creation and |
| 64 |
destruction code, link list manipulation, other helper functions. |
| 65 |
|
| 66 |
This implementation has a simple naming convention designed to alert |
| 67 |
someone reading the source code to the nature and scope of a |
| 68 |
function. The convention is in the function name, such that: |
| 69 |
|
| 70 |
dapl_ Primary entry from a dat_ function, e.g. |
| 71 |
dapl_ep_create(), which mirrors dat_ep_create(). |
| 72 |
dapls_ The 's' restricts it to the subsystem, e.g. the |
| 73 |
'ep' subsystem. dapls_ functions are not exposed |
| 74 |
externally, but are internal to dapl. |
| 75 |
dapli_ The 'i' restricts the function to the file where it |
| 76 |
is declared. These functions are always 'static' C |
| 77 |
functions. |
| 78 |
|
| 79 |
This convention is not followed as consistently as we would like, but is |
| 80 |
common in the reference implementation. |
| 81 |
|
| 82 |
1. End Points (EPs) |
| 83 |
------------------------- |
| 84 |
|
| 85 |
DAPL End Points provide a channel abstraction necessary to transmit and |
| 86 |
receive data. EPs interact with Service Points, either Public Service |
| 87 |
Points or Reserved Service Points, to establish a connection from one |
| 88 |
provider to another. |
| 89 |
|
| 90 |
The primary EP entry points in the DAT API as they relate to DAPL are |
| 91 |
listed in the following table: |
| 92 |
|
| 93 |
dat_ep_create |
| 94 |
dat_ep_query |
| 95 |
dat_ep_modify |
| 96 |
dat_ep_connect |
| 97 |
dat_ep_dup_connect |
| 98 |
dat_ep_disconnect |
| 99 |
dat_ep_post_send |
| 100 |
dat_ep_post_recv |
| 101 |
dat_ep_post_rdma_read |
| 102 |
dat_ep_post_rdma_write |
| 103 |
dat_ep_get_status |
| 104 |
dat_ep_reset |
| 105 |
dat_ep_free |
| 106 |
|
| 107 |
Additionally, the following connection functions interact with |
| 108 |
EndPoints: |
| 109 |
dat_psp_create |
| 110 |
dat_psp_query |
| 111 |
dat_psp_free |
| 112 |
dat_rsp_create |
| 113 |
dat_rsp_query |
| 114 |
dat_rsp_free |
| 115 |
dat_cr_accept |
| 116 |
dat_cr_reject |
| 117 |
dat_cr_query |
| 118 |
dat_cr_handoff |
| 119 |
|
| 120 |
The reference implementation maps the EndPoint abstraction onto an |
| 121 |
InfiniBand Queue Pair (QP). |
| 122 |
|
| 123 |
The DAPL_EP structure is used to maintain the state and components of |
| 124 |
the EP object and the underlying QP. As will be explained below, keeping |
| 125 |
track of the QP state is critical for successful operation. Access to |
| 126 |
the DAPL_EP fields are done atomically. |
| 127 |
|
| 128 |
|
| 129 |
====================================================================== |
| 130 |
Goals |
| 131 |
====================================================================== |
| 132 |
|
| 133 |
Initial goals |
| 134 |
------------- |
| 135 |
-- Implement all of the dat_ep_* calls described in the DAT |
| 136 |
Specification. |
| 137 |
|
| 138 |
-- Implement connection calls described in the DAT Specification with |
| 139 |
the following exception: |
| 140 |
- dat_cr_handoff. This is best done with kernel mediation, and is |
| 141 |
therefore out of scope for the reference implementation. |
| 142 |
|
| 143 |
-- The implementation should be as portable as possible, to facilitate |
| 144 |
HCA Vendors efforts to implement vendor-specific versions of DAPL. |
| 145 |
|
| 146 |
-- The implementation must be able to work during ongoing development |
| 147 |
of provider software agents, drivers, etc. |
| 148 |
|
| 149 |
Later goals |
| 150 |
----------- |
| 151 |
-- Examine various possible performance optimizations. This document |
| 152 |
lists potential performance improvements, but the specific |
| 153 |
performance improvements implemented should be guided by customer |
| 154 |
requirements. |
| 155 |
|
| 156 |
============================================ |
| 157 |
Requirements, constraints, and design inputs |
| 158 |
============================================ |
| 159 |
|
| 160 |
The EndPoint is the base channel abstraction. An Endpoint must be |
| 161 |
established before data can be exchanged with a remote node. The |
| 162 |
EndPoint is mapped to the underlying InfiniBand QP channel abstraction. |
| 163 |
When a connection is initiated, the InfiniBand Connection Manager will |
| 164 |
be solicited. The implementation is constrained by the capabilities and |
| 165 |
behavior of the underlying InfiniBand facilities. |
| 166 |
|
| 167 |
Note that transports other than InfiniBand may not need to rely on |
| 168 |
Connection Managers or other infrastructure, this is an artifact of |
| 169 |
this transport. |
| 170 |
|
| 171 |
An EP is not an exact match to an InfiniBand QP, the differences |
| 172 |
introduce constraints that are not obvious. There are three primary |
| 173 |
areas of conflict between the DAPL and InfiniBand models: |
| 174 |
|
| 175 |
1) EP and QP creation differences |
| 176 |
2) Provider provided EPs on passive side of connections |
| 177 |
3) Connection timeouts |
| 178 |
|
| 179 |
-- EP and QP creation |
| 180 |
|
| 181 |
The most obvious difference between an EP and a QP is the presence of a |
| 182 |
memory handle when the object is created. InfiniBand requires a |
| 183 |
Protection Domain (PD) be specified when a QP is created; in the DAPL |
| 184 |
world, a Protection Zone (PZ) maps to an InfiniBand Protection Domain. |
| 185 |
DAPL does not require a PZ to be present when an EP is created, and that |
| 186 |
introduces two problems: |
| 187 |
|
| 188 |
1) If a PZ is NULL when an EP is created, a QP will not be bound to |
| 189 |
the EP until dat_ep_modify() is used to assign it later. A PZ is |
| 190 |
required before RECV requests can be posted and before a connection |
| 191 |
can be established. |
| 192 |
|
| 193 |
2) If a DAPL user changes the PZ on an EP before it is connected, |
| 194 |
DAPL must release the current QP and create a new one with a |
| 195 |
new Protection Domain. |
| 196 |
|
| 197 |
-- Provider provided EPs on connection |
| 198 |
|
| 199 |
The second area where the DAPL and IB models conflict is a direct result |
| 200 |
of the requirement to specify a Protection Domain when a QP is created. |
| 201 |
|
| 202 |
DAPL allows a PSP to be created in such a way that an EP will |
| 203 |
automatically be provided to the user when a connection occurs. This is |
| 204 |
not critical to the DAPL model but in fact does provide some convenience |
| 205 |
to the user. InfiniBand provides a similar mechanism, but with an |
| 206 |
important difference: InfiniBand requires the user to supply the |
| 207 |
Protection Domain for the passive connection endpoint that will be |
| 208 |
supplied to all QPs created as a result of connection requests; DAPL |
| 209 |
mandates a NULL PZ and requires the user to change the PZ before using |
| 210 |
the EP. |
| 211 |
|
| 212 |
The reference implementation creates an 'empty' EP when the user |
| 213 |
specifies the DAT_PSP_PROVIDER flag; it is empty in the sense that a QP |
| 214 |
is not attached to the EP. Before the user can dat_cr_accept the |
| 215 |
connection, the EP must be modified to have a PZ bound to it, which in |
| 216 |
turn will cause a QP to be bound to the EP. |
| 217 |
|
| 218 |
To keep track of the current state of the EP, the DAPL_EP structure |
| 219 |
has a qp_state field. The type of this field is specific to the |
| 220 |
provider and the states are provider-specified states for a particular |
| 221 |
transport, with the addition of a single state from dapl: |
| 222 |
DAPL_QP_STATE_UNATTACHED, indicating that no QP has been bound to the |
| 223 |
EP. The qp_state field is an open enumerator, containing a single DAPL |
| 224 |
state in addition states specified by the provider. |
| 225 |
DAPL_QP_STATE_UNATTACHED is randomly defined to be 0xFFF0, a |
| 226 |
value selected strictly because it has the property that it will not |
| 227 |
collide with provider states; if this is not true, this value must be |
| 228 |
changed such that it is unique. |
| 229 |
|
| 230 |
The common layer of DAPL only looks at this single value for qp_state, |
| 231 |
it cannot be aware of states that are unique to the provider. However, |
| 232 |
the provider layer is free to update this field and may use it as a |
| 233 |
cache for current QP state. The field must be updated when a QP (or |
| 234 |
other endpoint resource) is bound to the EP. |
| 235 |
|
| 236 |
DAPL 1.2 provides DAT level states that will make this obsolete, but it |
| 237 |
exists in pre DAPL 1.2 code. |
| 238 |
|
| 239 |
|
| 240 |
-- Connection Timeouts |
| 241 |
|
| 242 |
The third difference in the DAPL and InfiniBand models has to do with |
| 243 |
timeouts on connections. InfiniBand does not provide a way to specify a |
| 244 |
connection timeout, so it will wait indefinitely for a connection to |
| 245 |
occur. dat_ep_connect supports a timeout value providing the user with |
| 246 |
control over how long they are willing to wait for a connection to |
| 247 |
occur. |
| 248 |
|
| 249 |
DAPL maintains a timer thread to watch over pending connections. A |
| 250 |
shared timer queue has a sorted list of timeout values. If a timeout |
| 251 |
is requested, dapl_ep_connect() will invoke dapls_timer_set(), which |
| 252 |
will add a timer record to the sorted list of timeouts. The timeout |
| 253 |
thread is started lazily: that is, it isn't started until a timeout is |
| 254 |
requested. Once a timeout has been requested, the thread will continue |
| 255 |
to exist until the application terminates. |
| 256 |
|
| 257 |
The timer record is actually a part of the DAPL_EP structure, so there |
| 258 |
are no extra memory allocations required for timeouts. dapls_timer_set() |
| 259 |
will initialize the timer record and insert it into the sorted queue at |
| 260 |
the appropriate place. If this is the first record, or is inserted |
| 261 |
before the first record (which will be the 'next' timeout to expire), |
| 262 |
the timer thread will be awakened so it can recalculate how long it must |
| 263 |
sleep until the timeout occurs. |
| 264 |
|
| 265 |
When a timeout does occur, the timeout code will cancel the connection |
| 266 |
request by invoking the provider routine dapls_ib_disconnect_clean(). |
| 267 |
This allows the software module with explicit knowledge of the provider |
| 268 |
to take appropriate action and cancel the connection attempt. As a side |
| 269 |
effect, the EP will be placed into the UNCONNECTED state, and the QP |
| 270 |
will be in the ERROR state. A side effect of this state change is that |
| 271 |
all DTOs will be flushed. The provider must support a mechanism to |
| 272 |
completely cancel a connection request. |
| 273 |
|
| 274 |
|
| 275 |
====================================================================== |
| 276 |
DAPL EP Subsystem Design |
| 277 |
====================================================================== |
| 278 |
|
| 279 |
In section 6.5.1 of the DAT Specification there is a UML state |
| 280 |
transition diagram for an EndPoint which goes over the transitions and |
| 281 |
states during the lifetime of an EP. It is nearly impossible to read. |
| 282 |
The reference implementation is faithful to the DAT Spec and is |
| 283 |
believed to be correct. |
| 284 |
|
| 285 |
This description of the EP will follow from creation to connection to |
| 286 |
termination. It will also discuss the source code organization as this |
| 287 |
is part of the design expression. |
| 288 |
|
| 289 |
-- EP and QP creation |
| 290 |
|
| 291 |
The preamble to creating an EP requires us to verify the attributes |
| 292 |
specified by the user. If a user were to specify max_recv_dtos as 0, for |
| 293 |
example, the EP would not be useful in any regard. If the user does not |
| 294 |
provide EP attrs, the DAPL layer will supply a set of common defaults |
| 295 |
resulting in a reasonable EP. The defaults are set up in |
| 296 |
dapli_ep_default_attrs(), and the default values are given at the top of |
| 297 |
dapl_ep_util.c. Non-InfiniBand transports will want to examine these |
| 298 |
values to make sure they are 'reasonable'. This simplistic mechanism may |
| 299 |
change in the future. |
| 300 |
|
| 301 |
A number of handles are bound to the EP, so a reference count is taken |
| 302 |
on each of them. All reference counts in the DAPL system are incremented |
| 303 |
or decremented using atomic operations; it is important to always use |
| 304 |
the OS dependent atomic routines and not substitute a lock, as it will |
| 305 |
not be observed elsewhere in the system and will have unpredictable |
| 306 |
results. |
| 307 |
|
| 308 |
Reference counts are taken if there are non NULL values on any of: |
| 309 |
pz_handle |
| 310 |
connect_evd_handle |
| 311 |
recv_evd_handle |
| 312 |
request_evd_handle |
| 313 |
|
| 314 |
The purpose of reference counts should be obvious: to prevent premature |
| 315 |
release of resources that are still being used. |
| 316 |
|
| 317 |
As has been discussed above, each EP is bound to a QP before it can be |
| 318 |
connected. If a valid PZ is provided at creation time then a QP is bound |
| 319 |
to the EP immediately. If the user later uses ep_modify() to change the |
| 320 |
PZ, the QP will be destroyed and a new one created with the appropriate |
| 321 |
Protection Domain. |
| 322 |
|
| 323 |
Finally, an EP is an IA resource and is linked onto the EP chain of the |
| 324 |
superior IA. EPs linked onto an IA are assumed to be complete, so this |
| 325 |
is the final step of EP creation. |
| 326 |
|
| 327 |
After an EP is created, the ep_state will be DAT_EP_STATE_UNCONNECTED |
| 328 |
and the qp_state will either be DAPL_QP_STATE_UNATTACHED or assigned by |
| 329 |
the provider layer (e.g.IB_QP_STATE_INIT). The qp_state indicates the QP |
| 330 |
binding and the current state of the QP. |
| 331 |
|
| 332 |
A qp_state of DAPL_QP_STATE_UNATTACHED indicates there is no QP bound |
| 333 |
to this EP. This is a result of a NULL PZ when dat_ep_create() was |
| 334 |
invoked, and which has been explained in detail above. The user must |
| 335 |
call dat_ep_modify() and install a valid PZ before the EP can be used. |
| 336 |
|
| 337 |
When an InfiniBand QP is created it is in the RESET state, which is |
| 338 |
specified in the InfiniBand Spec, section 10.3. However, DAPL creates |
| 339 |
the EP in the UNCONNECTED state and requires an unconnected EP to be |
| 340 |
able to queue RECV requests before a connection occurs. The InfiniBand |
| 341 |
spec allows RECV requests to be queued on an QP if the QP is in the INIT |
| 342 |
state, so after creating a QP the DAPL provider code must transition it |
| 343 |
to the INIT state. |
| 344 |
|
| 345 |
There is a mapping between the DAPL EP state and the InfiniBand QP |
| 346 |
state. DAPL_QP_STATE_UNATTACHED indicates the underlying QP is in the |
| 347 |
INIT state. This is critical: RECV DTOs can be posted on an EP in the |
| 348 |
UNATTACHED state, so the underlying QP must be in the appropriate state |
| 349 |
to allow this to happen. |
| 350 |
|
| 351 |
There is an obvious design tradeoff in transitioning the QP |
| 352 |
state. Immediately moving the state to INIT takes extra time at creation |
| 353 |
but allows immediate posting of RECV operations; however, it will |
| 354 |
involve a more complex tear down procedure if the QP must be replaced as |
| 355 |
a side effect of a dat_ep_modify operation. The alternative would be to |
| 356 |
delay transitioning the QP to INIT until a post operation is invoked, |
| 357 |
but that requires a run time check for every post operation. This design |
| 358 |
assumes users will infrequently cause a QP to be replaced after it is |
| 359 |
created and prefer to pay the state transition penalty at creation time. |
| 360 |
|
| 361 |
-- EP Query and Modify operations |
| 362 |
|
| 363 |
Because all of the ep_param data are kept up to date in the dapl_ep |
| 364 |
structure, and because they use the complete DAT specified structure, a |
| 365 |
query operation is trivial; a simple assignment from the internal |
| 366 |
structure to the user parameter. uDAPL allows the implementation to |
| 367 |
either return the fields specified by the user, or to return more than |
| 368 |
the user requested; the reference implementation does the latter. It is |
| 369 |
simpler and faster to copy the entire structure rather than to determine |
| 370 |
which of all of the possible fields the user requested. |
| 371 |
|
| 372 |
dat_ep_query() requires the implementation to report the address of the |
| 373 |
remote node, if the EP is connected. This is different from standard |
| 374 |
InfiniBand, if only because of the difference in name space. InfiniBand |
| 375 |
has the information on the remote LID, but it does not have the remote |
| 376 |
IP address, which is what DAT specifies. The reference implementation |
| 377 |
makes use of a lookup/name-service called ATS ( Address Translation |
| 378 |
Service), which is built using the InfiniBand Subnet Administrator. ATS |
| 379 |
is InfiniBand only, other transports will use a different mechanism. |
| 380 |
|
| 381 |
A driver will register itself and one or more IP addresses with ATS |
| 382 |
at some point before a connection can be made. How the addresses are |
| 383 |
provided to the driver, or how this is managed by the driver is not |
| 384 |
specified. The ATS proposal is available from the DAT Collaborative. |
| 385 |
|
| 386 |
When dat_ep_query() is invoked on a connected EP, it will request the |
| 387 |
remote address from the provider layer. The provider layer will use |
| 388 |
whatever means are necessary to obtain the IP address of the other end of |
| 389 |
the connection. The results are placed into a buffer that is part of the |
| 390 |
EP structure. Finally, the address of the EP structure is placed into |
| 391 |
the ep_param.remote_ia_address_ptr field. |
| 392 |
|
| 393 |
The ep_modify operation will modify the fields in the DAT_PARAM |
| 394 |
structure. There are some fields that cannot be updated, and there are |
| 395 |
others that can only be updated if the EP is in the correct state. The |
| 396 |
uDAPL spec outlines the EP states permitting ep modifications, but |
| 397 |
generally they are DAT_EP_STATE_UNCONNECTED and |
| 398 |
DAT_EP_STATE_PASSIVE_CONNECTION_PENDING. |
| 399 |
|
| 400 |
When replacing EVD handles it is a simple matter of releasing a |
| 401 |
reference on the previous handle and taking a new reference on the new |
| 402 |
handle. The Reference Implementation manages resource tracking using |
| 403 |
reference counts, which guarantees a particular handle will not be |
| 404 |
released prematurely. Reference counts are checked in the free routines |
| 405 |
of various objects. |
| 406 |
|
| 407 |
As has been mentioned previously, if the PZ handle is changed then the |
| 408 |
QP must be released, if already assigned, and a new QP must be created |
| 409 |
to bind to this EP. |
| 410 |
|
| 411 |
There are some fields in the DAT_PARAM structure that are related to the |
| 412 |
underlying hardware implementation. For these values DAPL will do a |
| 413 |
fresh query of the QP, rather than depend on stale values. Even so, the |
| 414 |
values returned are 'best effort' as a competing thread may change |
| 415 |
certain values before the requesting thread has the opportunity to read |
| 416 |
them. Applications should protect against this. |
| 417 |
|
| 418 |
Finally, the underlying provider is invoked to update the QP with new |
| 419 |
values, but only if some of the attributes have been changed. As is |
| 420 |
true of most of the implementation, we only invoke the provider code |
| 421 |
when necessary. |
| 422 |
|
| 423 |
====================================================================== |
| 424 |
Connections |
| 425 |
====================================================================== |
| 426 |
|
| 427 |
There are of course two sides to a connection, and in the DAPL model |
| 428 |
there is an Active and a Passive side. For clarity, the Passive side |
| 429 |
is a server waiting for a connection, and the Active side is a client |
| 430 |
requesting a connection from the Passive server. We will discuss each |
| 431 |
of these in turn. |
| 432 |
|
| 433 |
Connections happen in the InfiniBand world by using a Connection Manager |
| 434 |
(CM) interface. Those unfamiliar with the IB model of addressing and |
| 435 |
management agents may want to familiarize themselves with these aspects |
| 436 |
of the IB spec before proceeding in this document. Be warned that the |
| 437 |
connection section of the IB spec is the most ambiguous portion of the |
| 438 |
spec. |
| 439 |
|
| 440 |
First, let's walk through a primitive diagram of a connection: |
| 441 |
|
| 442 |
|
| 443 |
SERVER (passive) CLIENT (active) |
| 444 |
--------------- --------------- |
| 445 |
1. dapl_psp_create |
| 446 |
or dapl_rsp_create |
| 447 |
[ now listening ] |
| 448 |
|
| 449 |
2. dapl_ep_connect |
| 450 |
<------------- |
| 451 |
3. dapls_cr_callback |
| 452 |
DAT_CONNECTION_REQUEST_EVENT |
| 453 |
[ Create and post a DAT_CONNECTION_REQUEST_EVENT event ] |
| 454 |
|
| 455 |
4. Event code processing |
| 456 |
|
| 457 |
5. Create an EP if necessary |
| 458 |
(according to the flags |
| 459 |
when the PSP was created) |
| 460 |
|
| 461 |
6. dapl_cr_accept or dapl_cr_reject |
| 462 |
-------------> |
| 463 |
7. dapl_evd_connection_callback |
| 464 |
DAT_CONNECTION_EVENT_ESTABLISHED |
| 465 |
[ Create and post a |
| 466 |
DAT_CONNECTION_EVENT_ESTABLISHED |
| 467 |
event ] |
| 468 |
|
| 469 |
8. <------------- RTU |
| 470 |
|
| 471 |
9. dapls_cr_callback |
| 472 |
DAT_CONNECTION_EVENT_ESTABLISHED |
| 473 |
[ Create and post a DAT_CONNECTION_EVENT_ESTABLISHED |
| 474 |
event ] |
| 475 |
|
| 476 |
10. ...processing... |
| 477 |
|
| 478 |
11. Either side issues a dat_ep_disconnect |
| 479 |
|
| 480 |
12. dapls_cr_callback |
| 481 |
DAT_CONNECTION_EVENT_DISCONNECTED |
| 482 |
|
| 483 |
[ Create and post a |
| 484 |
DAT_CONNECTION_EVENT_DISCONNECTED |
| 485 |
event ] |
| 486 |
|
| 487 |
13. dapl_evd_connection_callback |
| 488 |
DAT_CONNECTION_EVENT_DISCONNECTED |
| 489 |
[ Create and post a |
| 490 |
DAT_CONNECTION_EVENT_DISCONNECTED |
| 491 |
event ] |
| 492 |
|
| 493 |
|
| 494 |
In the above diagram, time is numbered in the left hand column and is |
| 495 |
represented vertically. |
| 496 |
|
| 497 |
We will continue our discussion of connections using the above diagram, |
| 498 |
following a sequential order for connection establishment. |
| 499 |
|
| 500 |
There are in fact two types of service points detailed in the uDAPL |
| 501 |
specification. We will limit our discussion to PSPs for convenience, but |
| 502 |
there are only minor differences between PSPs and RSPs. |
| 503 |
|
| 504 |
The reader should observe that all passive-side connection events will |
| 505 |
be received by dapls_cr_callback(), and all active side connection |
| 506 |
events occur through dapl_evd_connection_callback(). At one point during |
| 507 |
the implementation these routines were combined as they are very |
| 508 |
similar, but there are subtle differences causing them to remain |
| 509 |
separate. |
| 510 |
|
| 511 |
Progressing through the series of events as outlined in the diagram |
| 512 |
above: |
| 513 |
|
| 514 |
1. dapl_psp_create |
| 515 |
|
| 516 |
When a PSP is created, the final act will be to set it listening for |
| 517 |
connections from remote nodes. It is important to realize that a |
| 518 |
connection may in fact arrive from a remote node before the routine |
| 519 |
setting up a listener has returned to dapl_psp_create; as soon as |
| 520 |
dapls_ib_setup_conn_listener() is invoked connection callbacks may |
| 521 |
arrive. To reduce race conditions this routine must be called as the |
| 522 |
last practical operation when creating a PSP. |
| 523 |
|
| 524 |
dapls_ib_setup_conn_listener() is provider specific. The key insight |
| 525 |
is that the DAPL connection qualifier (conn_qual) will become the |
| 526 |
InfiniBand Service ID. The passive side of the connection is now |
| 527 |
listening for connection requests. It should be obvious that the |
| 528 |
conn_qual must be unique. |
| 529 |
|
| 530 |
InfiniBand allows a 64 bit connection qualifier, which is supported |
| 531 |
by the DAT spec. IP based networks may be limited to 16 bits, so |
| 532 |
provider implementations may want to return an error if it exceeds |
| 533 |
the maximum allowable by the transport. |
| 534 |
|
| 535 |
2. dapl_ep_connect |
| 536 |
|
| 537 |
The active side initiates a connection with dapl_ep_connect, which |
| 538 |
will transition the EP into DAT_EP_STATE_ACTIVE_CONNECTION_PENDING. |
| 539 |
Again, connections are in the domain of the providers' Connection |
| 540 |
Manager and the mechanics are very much provider specific. The key |
| 541 |
points are that a DAT_IA_ADDRESS_PTR must be translated to a GID |
| 542 |
before a connection initiation can occur. This is discussed below. |
| 543 |
|
| 544 |
InfiniBand supports different amounts of private data on various |
| 545 |
connection functions. Other transports allow variable sizes of |
| 546 |
private data with no practical limit.The DAPL connection code does |
| 547 |
not enforce a fixed amount of private data, but rather makes |
| 548 |
available to the user all it has available, as specified by |
| 549 |
DAPL_MAX_PRIVATE_DATA_SIZE. |
| 550 |
|
| 551 |
Private data will be stored in a fixed buffer as part of the |
| 552 |
connection record, which is the primary reason to limit the size. |
| 553 |
|
| 554 |
To assist development on new transports that do not have a full |
| 555 |
connection infrastructure in place, there are a couple of compile time |
| 556 |
flags that will include certain code: CM_BUSTED and |
| 557 |
IBOSTS_NAMING. These are discussed below in more detail, but |
| 558 |
essentially: |
| 559 |
|
| 560 |
CM_BUSTED: fakes a connection on both sides of the wire, does not |
| 561 |
transmit any private data. |
| 562 |
|
| 563 |
IBHOSTS_NAMING: provides a simple IP_ADDRESS to LID translation |
| 564 |
mechanism in a text file, which is read when the dapl library |
| 565 |
loads. Private data is exchanged in this case, but it includes a |
| 566 |
header that contains the remote IP address. Technically, this defines |
| 567 |
a protocol and is in violation of the DAT spec, but it has proved |
| 568 |
useful in development. |
| 569 |
|
| 570 |
3. dapls_cr_callback |
| 571 |
|
| 572 |
The connection sequence is entirely event driven. An operation is |
| 573 |
posted, then an asynchronous event will occur some time later. The |
| 574 |
event may cause other actions to occur which may result in still |
| 575 |
more events. |
| 576 |
|
| 577 |
dapls_ib_setup_conn_listener() registered for a callback for |
| 578 |
connection events, and we now receive a DAT event for a connection |
| 579 |
request. The provider layer will translate the native event type to |
| 580 |
a DAT event. |
| 581 |
|
| 582 |
An upcall is invoked on the server side of the connection with an |
| 583 |
event of type DAT_CONNECTION_REQUEST_EVENT. This is a unique event |
| 584 |
in the callback code as it is the only case when an EP is not |
| 585 |
already in play; in all other cases, it is possible to look up the |
| 586 |
relevant EP for an operation. |
| 587 |
|
| 588 |
Code exists to make sure the relevant connection object, the PSP or |
| 589 |
RSP, is actually in a useful state and ready to be connected |
| 590 |
to. One of the critical differences between a PSP and an RSP is |
| 591 |
that an RSP is a one-shot connection object; once a connection |
| 592 |
occurs, no other connections can be made to it. |
| 593 |
|
| 594 |
There is a small difference in the InfiniBand and DAPL connection |
| 595 |
models here as well. DAPL may disable a PSP at any time without |
| 596 |
affecting current connections. When you tear down an InfiniBand |
| 597 |
service endpoint, all of the connections are torn down too. Because |
| 598 |
of this difference, when a DAPL app frees a PSP, only a state |
| 599 |
change is made. The underlying service point is still available and |
| 600 |
technically capable of receiving connections. If a connection |
| 601 |
request arrives when the PSP is in this state, a rejection message |
| 602 |
is sent such that the requesting node believes no service point is |
| 603 |
listening. |
| 604 |
|
| 605 |
Once the connection has been examined, it will continue with the |
| 606 |
connection protocol. The EP will move to a CONNECTION_PENDING |
| 607 |
state. |
| 608 |
|
| 609 |
The connection request will cause a CR record to be allocated, |
| 610 |
which holds all of the important connection request |
| 611 |
information. The CR record will be linked onto the PSP structure |
| 612 |
for retrieval in the future when other requests arrive. |
| 613 |
|
| 614 |
The astute reader of the spec will observe that there is not a |
| 615 |
dapl_cr_create call: CR records are created as part of a connection |
| 616 |
attempt on the passive side of the connection. A CR is created now |
| 617 |
and set up. A point that will become important later, caps for |
| 618 |
emphasis: |
| 619 |
|
| 620 |
A CR WILL EXIST FOR THE LIFE OF A CONNECTION; THEY ARE DESTROYED AT |
| 621 |
DISCONNECT TIME. |
| 622 |
|
| 623 |
In the connection request processing a CR and an EVENT are created, |
| 624 |
the event will be posted along with the connection information just |
| 625 |
received. |
| 626 |
|
| 627 |
Private data is also copied into the CR record. Private data |
| 628 |
arrived with the connection request and is not a permanent |
| 629 |
resource, so it is copied into the dapl space to be used at a later |
| 630 |
time. Different transports have varying capabilities on the size of |
| 631 |
private data, so a call to the provider is invoked to determine how |
| 632 |
big it actually is. There is an upper bound on the amount of |
| 633 |
private data the implementation will deal with, set at |
| 634 |
DAPL_MAX_PRIVATE_DATA_SIZE (256 bytes at this writing). |
| 635 |
|
| 636 |
4. Event code processing |
| 637 |
|
| 638 |
The final stage in a connection request is to generate an event on |
| 639 |
a connection EVD using dapls_evd_post_cr_arrival_event(). |
| 640 |
|
| 641 |
5. Create an EP if necessary |
| 642 |
|
| 643 |
When the app processes a connection event, it needs to respond. If |
| 644 |
the PSP is configured to create an EP automatically, the callback |
| 645 |
code has already done it; creating an EP with no attached QP. Else, |
| 646 |
the user must provide an EP to make the connection. |
| 647 |
|
| 648 |
(4) and (5) are all done in user mode. The only interesting thing is |
| 649 |
that when the user calls dat_cr_accept(), a ready EP must be |
| 650 |
provided. If the EP was supplied by the PSP in the callback, it |
| 651 |
must have a PZ associated with it and whatever other attributes |
| 652 |
need to be set. |
| 653 |
|
| 654 |
6. dapl_cr_accept or dapl_cr_reject |
| 655 |
|
| 656 |
For discussion purposes, we will follow the accept |
| 657 |
path. dapl_cr_reject says you are done and there will be no further |
| 658 |
events to deal with. |
| 659 |
|
| 660 |
Assuming it accepts the connection for our example, the dapl code |
| 661 |
will verify that an EP is in place and will deal with private data |
| 662 |
that can be transmitted in a cr_accept call. The underlying |
| 663 |
provider is invoked to complete this leg of the protocol. |
| 664 |
|
| 665 |
7. dapl_evd_connection_callback |
| 666 |
|
| 667 |
An EVD callback is always a response to a connection oriented |
| 668 |
request. As such, an EP is always present, and in fact is passed |
| 669 |
into the upcall as the 'context' argument. |
| 670 |
|
| 671 |
Connection requests may take an arbitrary amount of time, so the EP |
| 672 |
is always checked for a running timer when the upcall is made. As |
| 673 |
has been discussed above, if a timer expires before an upcall |
| 674 |
occurs, the connection must be completely canceled such that there |
| 675 |
is no upcall. |
| 676 |
|
| 677 |
The event signifying completion of the connection is |
| 678 |
DAT_CONNECTION_EVENT_ESTABLISHED, and it will move the EP to the |
| 679 |
CONNECTED state and post this event on the connection EVD. Private |
| 680 |
data will be copied to an area in the EP structure, which is |
| 681 |
persistent. |
| 682 |
|
| 683 |
At this point, the EP is connected and the application is free to |
| 684 |
post DTOs. |
| 685 |
|
| 686 |
8i. RTU |
| 687 |
|
| 688 |
This item is labeled "8i" as it is internal to the InfiniBand |
| 689 |
implementation, it is not initiated by dapl. The final leg of a |
| 690 |
connection is an RTU sent from the initiating node to the server |
| 691 |
node, indicating the connection has been made successfully. |
| 692 |
|
| 693 |
Other transports may have a different connection protocol. |
| 694 |
|
| 695 |
9. dapls_cr_callback |
| 696 |
|
| 697 |
When the RTU arrives, an upcall is invoked with a |
| 698 |
DAT_CONNECTION_EVENT_ESTABLISHED event, which will be posted to the |
| 699 |
connection EVD event queue. The EP is moved to the CONNECTED |
| 700 |
state. |
| 701 |
|
| 702 |
There is no private data for dapl to deal with, even though some |
| 703 |
transports may provide private data at each step of a connection. |
| 704 |
|
| 705 |
The connection activity is occurring on a separate channel than the |
| 706 |
EP, so this is inherently a racy operation. The correct |
| 707 |
application will always post RECV buffers on an EP before |
| 708 |
initiating a connection sequence, as it is entirely possible for |
| 709 |
DTOs to arrive *before* the final connection event arrives. |
| 710 |
|
| 711 |
The architecturally interesting feature of this exchange occurs |
| 712 |
because of differences in the InfiniBand and the DAT connection |
| 713 |
models, which are briefly outlined here. |
| 714 |
|
| 715 |
InfiniBand maintains the original connecting objects throughout the |
| 716 |
life of the connection. That is, we originally get a callback event |
| 717 |
associated with the Service (DAT PSP) that is listening for |
| 718 |
connection events. A QP will be connected but the callback event |
| 719 |
will still be received on the Service. Later, a callback event will |
| 720 |
occur for a DISCONNECT, and again the Service will be the object of |
| 721 |
the connection. In the DAPL implementation, the Service will |
| 722 |
provide the PSP that is registered as listening on that connection |
| 723 |
qualifier. |
| 724 |
|
| 725 |
The difference is that DAT has a PSP receive a connection event, |
| 726 |
but subsequently hands all connection events off to an EP. After a |
| 727 |
dat_cr_accept is issued, all connection/disconnection events occur |
| 728 |
on the EP. DAT more closely follows the IP connection model. |
| 729 |
|
| 730 |
To support the DAT model, a CR is maintained through the life of |
| 731 |
the connection. There is exactly one CR per connection, but any |
| 732 |
number of CRs may exist for any given PSP. CRs are maintained on a |
| 733 |
linked list pointed to by the PSP structure. A lookup routine will |
| 734 |
match the cm_handle, unique for each connection, with the |
| 735 |
appropriate CR. This allows us to find the appropriate EP which |
| 736 |
will be used to create an event to be posted to the user. |
| 737 |
|
| 738 |
* dat_psp_destroy |
| 739 |
|
| 740 |
It should be understood that the PSP will maintain all of the CR |
| 741 |
records, and hence the PSP must persist until the final disconnect. |
| 742 |
In the DAT model there is no association between a PSP and a |
| 743 |
connected QP, so there is no reason not to destroy a PSP before the |
| 744 |
final disconnect. |
| 745 |
|
| 746 |
Because of the model mismatch we must preserve the PSP until the |
| 747 |
final disconnect. If the user invokes dat_psp_destroy(), all of the |
| 748 |
associations maintained by the PSP will be severed; but the PSP |
| 749 |
structure itself remains as a container for the CR records. The PSP |
| 750 |
structure maintains a simple count of CR records so we can easily |
| 751 |
determine the final disconnect and release memory. Once a |
| 752 |
disconnect event is received for a specific cm_handle, no further |
| 753 |
events will be received and it is safe to discard the CR record. |
| 754 |
|
| 755 |
10. ...processing... |
| 756 |
|
| 757 |
This is just a place holder to show that applications actually do |
| 758 |
something after making a connection. They might not too... |
| 759 |
|
| 760 |
11. Either side issues a dat_ep_disconnect |
| 761 |
|
| 762 |
dat_ep_disconnect() can be initiated by either side of a |
| 763 |
connection. There are two kinds of disconnect flags that can be |
| 764 |
passed in, but the final result is largely the same. |
| 765 |
|
| 766 |
DAT_CLOSE_ABRUPT_FLAG will cause the connection to be immediately |
| 767 |
terminated. In InfiniBand terms, the QP is immediately moved to the |
| 768 |
ERROR state, and after some time it will be moved to the RESET |
| 769 |
state. |
| 770 |
|
| 771 |
DAT_CLOSE_GRACEFUL_FLAG will allow in-progress DTOs to complete. |
| 772 |
The underlying implementation will first transition the QP to the |
| 773 |
SQE state, before going to RESET. |
| 774 |
|
| 775 |
Both cases are handled by the underlying CM, there is no extra work |
| 776 |
for DAPL. |
| 777 |
|
| 778 |
12. dapls_cr_callback |
| 779 |
|
| 780 |
A disconnect will arrive on the passive side of the connection |
| 781 |
through dapls_cr_callback() with connection event |
| 782 |
DAT_CONNECTION_EVENT_DISCONNECTED. With this event the EP lookup |
| 783 |
code will free the CR associated with the connection, and may free |
| 784 |
the PSP if it is no longer listening, indicating it has been freed |
| 785 |
by the application. |
| 786 |
|
| 787 |
The callback will create and post a |
| 788 |
DAT_CONNECTION_EVENT_DISCONNECTED event for the application. |
| 789 |
|
| 790 |
13. dapl_evd_connection_callback |
| 791 |
|
| 792 |
The active side of the connection will receive |
| 793 |
DAT_CONNECTION_EVENT_DISCONNECTED as the connection event for |
| 794 |
dapl_evd_connection_callback(), and will create and post a |
| 795 |
DAT_CONNECTION_EVENT_DISCONNECTED event. Other than transitioning |
| 796 |
the EP to the DISCONNECTED state, there is no further processing. |
| 797 |
|
| 798 |
|
| 799 |
Observe that there are a number of exception conditions resulting in a |
| 800 |
disconnect of the EP, most of which will generate unique DAT events |
| 801 |
for the application to deal with. |
| 802 |
|
| 803 |
|
| 804 |
* Addressing and Naming |
| 805 |
|
| 806 |
The DAT Spec calls for a DAT_IA_ADDRESS_PTR to be an IP address, |
| 807 |
either IPv4 or IPv6. It is in fact a struct sockaddr in most |
| 808 |
systems. The dapl structures typically use IPv6 data types to |
| 809 |
accommodate the largest possible addresses, but most implementations |
| 810 |
use IPv4 formatted addresses. |
| 811 |
|
| 812 |
InfiniBand uses a transport specific address known as a LID, which |
| 813 |
typically is dynamically assigned by a Subnet Manager. Each HCA |
| 814 |
also has a global address, similar to an Ethernet MAC address, |
| 815 |
known as a GUID. ATS, mentioned above, is a mechanism using |
| 816 |
InfiniBand infrastructure to map from GUID/LID to IP addresses. It |
| 817 |
is not necessary for transports that use IP addresses natively, |
| 818 |
such as Ethernet devices. |
| 819 |
|
| 820 |
If a new implementation does not yet have a name service |
| 821 |
infrastructure, the DAPL implementation provides a simple name |
| 822 |
service facility under the #ifdef NO_NAME_SERVICE. This depends on |
| 823 |
two things: valid IP addresses registered and available to standard |
| 824 |
DNS system calls such as gethostbyname(); and a name/GID mapping |
| 825 |
file. |
| 826 |
|
| 827 |
IP addresses may be set up by system administrators or by a local |
| 828 |
power-user simply by editing the values into the /etc/hosts file. |
| 829 |
Setting IP addresses up in this manner is beyond the scope of this |
| 830 |
document. |
| 831 |
|
| 832 |
A simple mapping of names to GIDs is maintained in the ibhosts |
| 833 |
file, currently located at /etc/dapl/ibhosts. The format of |
| 834 |
the file is: |
| 835 |
|
| 836 |
<IP name> 0x<GID Prefix> 0x<GUID> |
| 837 |
|
| 838 |
For example: |
| 839 |
|
| 840 |
dat-linux3-ib0p0 0xfe80000000000000 0x0001730000003d11 |
| 841 |
dat-linux3-ib0p1 0xfe80000000000000 0x0001730000003d11 |
| 842 |
dat-linux3-ib1 0xfe80000000000000 0x0001730000003d52 |
| 843 |
dat-linux5-ib0 0xfe80000000000000 0x0001730000003d91 |
| 844 |
|
| 845 |
And for each hostname, there must be an entry in the /etc/hosts file |
| 846 |
similar to: |
| 847 |
|
| 848 |
dat_linux3-ib0a 198.165.10.11 |
| 849 |
dat_linux3-ib0b 198.165.10.12 |
| 850 |
dat_linux3-ib1a 198.165.10.21 |
| 851 |
dat_linux5-ib0a 198.165.10.31 |
| 852 |
|
| 853 |
|
| 854 |
In this example we have adopted the convention of naming each |
| 855 |
InfiniBand interface by using the form |
| 856 |
|
| 857 |
<node_name>-ib<device_number>[port_number] |
| 858 |
|
| 859 |
In the above example we can see that the machine dat_linux3 has three |
| 860 |
InfiniBand interfaces, which in this case we have named two ports on |
| 861 |
the first HCA and another port on a second. Utilizing standard DNS |
| 862 |
naming, the conventions used for identifying individual ports is |
| 863 |
completely up to the administrator. |
| 864 |
|
| 865 |
The GID Prefix and GUID are obtained from the HCA and map a port on |
| 866 |
the HCA: together they form the GID that is required by a CM to |
| 867 |
connect with the remote node. |
| 868 |
|
| 869 |
The simple name service builds an internal table after processing |
| 870 |
the ibhosts file which contains IP addresses and GIDs. It will use |
| 871 |
the standard getaddrinfo() function to obtain IP address |
| 872 |
information. |
| 873 |
|
| 874 |
When an application invoked dat_ep_connect(), the |
| 875 |
DAT_IA_ADDRESS_PTR will be compared in the table for a match and |
| 876 |
the destination GID established if found. If the address is not |
| 877 |
found then the user must first add the name to the ibhosts file. |
| 878 |
|
| 879 |
With a valid GID for the destination node, the underlying CM is |
| 880 |
invoked to make a connection. |
| 881 |
|
| 882 |
* Connection Management |
| 883 |
|
| 884 |
Getting a working CM has taken some time, in fact the DAPL project |
| 885 |
was nearly complete by the time a CM was available. In order to |
| 886 |
make progress, a connection hack was introduced that allows |
| 887 |
specific connections to take place. This is noted in the code by |
| 888 |
the CM_BUSTED #def. |
| 889 |
|
| 890 |
CM_BUSTED takes the place of a CM and will manually transition a QP |
| 891 |
through the various states to connect: INIT->RTR->RTS. It will also |
| 892 |
disconnect the connection, although the Torrent implementation |
| 893 |
simply destroys the QP and recreates a new one rather than |
| 894 |
transitioning through the typical disconnect states (which didn't |
| 895 |
work on early IB implementations). |
| 896 |
|
| 897 |
CM_BUSTED makes some assumptions about the remote end of the |
| 898 |
connection as no real information is exchanged. The ibapi |
| 899 |
implementation assumes both HCAs have the same LID, which implies |
| 900 |
there is no SM running. The vapi implementation assumes the LIDs |
| 901 |
are 0 and 1. Depending on the hardware, the LID value may in fact |
| 902 |
not make any difference. This code does not set the Global Route |
| 903 |
Header (GRH), which would cause the InfiniBand chip to be carefully |
| 904 |
checking LID information. |
| 905 |
|
| 906 |
The QP number is assumed to be identical on both ends of the |
| 907 |
connection, or differing by 1 if this is a loopback. There is an |
| 908 |
environment variable that will be read at initialization time if |
| 909 |
you are configured with a loopback, this value is checked when |
| 910 |
setting up a QP. The obvious downside to this scheme is that |
| 911 |
applications must stay synchronized in their QP usage or the |
| 912 |
initial exchange will fail as they are not truly connected. |
| 913 |
|
| 914 |
Add to this the limitation that HCAs must be connected in |
| 915 |
Point-to-Point topology or in a loopback. Without a GRH it will not |
| 916 |
work in a fabric. Again, using an SM will not work when CM_BUSTED |
| 917 |
is enabled. |
| 918 |
|
| 919 |
Despite these shortcomings, CM_BUSTED has proven very useful and |
| 920 |
will remain in the code for a while in order to aid development |
| 921 |
groups with new hardware and software. It is a hack to be sure, but |
| 922 |
it is relatively well isolated. |
| 923 |
|
| 924 |
|
| 925 |
-- Notes on Disconnecting |
| 926 |
|
| 927 |
An EP can only be disconnected if it is connected or unconnected; you |
| 928 |
cannot disconnect 'in progress' connections. An 'in progress |
| 929 |
connection may in fact time out, but the DAT Spec does not allow you |
| 930 |
to 'kill' it. DAPL will use the CM interface to disconnect from the |
| 931 |
remote node; this of course results in an asynchronous callback |
| 932 |
notifying the application the disconnect is complete. |
| 933 |
|
| 934 |
Disconnecting an unconnected EP is currently the only way to remove |
| 935 |
pending RECV operations from the EP. The DAPL spec notes that all |
| 936 |
DTOs must be removed from an EP before it can be deallocated, yet |
| 937 |
there is no explicit interface to remove pending RECV DTOs. The user |
| 938 |
will disconnect an unconnected EP to force the pending operations off |
| 939 |
of the queue, resulting in DTO callbacks indicating an error. The |
| 940 |
underlying InfiniBand implementation will cause the correct behavior |
| 941 |
to result. When doing this operation the DAT_CLOSE flag is ignored, |
| 942 |
DAPL will instruct the provider layer to abruptly disconnect the QP. |
| 943 |
|
| 944 |
As has been noted previously, specifying DAT_CLOSE_ABRUPT_FLAG as the |
| 945 |
disconnect completion flag will cause the CM implementation to |
| 946 |
transition the QP to the ERROR state to abort all operations, and then |
| 947 |
transition to the RESET state; if the flag is DAT_CLOSE_GRACEFUL_FLAG, |
| 948 |
the CM will first move to the SQE state and allow all pending I/O's to |
| 949 |
drain before moving to the RESET state. In either case, DAPL only |
| 950 |
needs to know that the QP is now in the RESET state, as it will need |
| 951 |
to be transitioned to the INIT state before it can be used again. |
| 952 |
|
| 953 |
====================================================================== |
| 954 |
Data Transfer Operations (DTOs) |
| 955 |
====================================================================== |
| 956 |
|
| 957 |
The DTO code is a straightforward translation of the DAT_LMR_TRIPLET |
| 958 |
to an InfiniBand work request. Unfortunately, IB does not specify what |
| 959 |
a work request looks like so this tends to be very vendor specific |
| 960 |
code. Each provider will supply a routine for this operation. |
| 961 |
|
| 962 |
InfiniBand allows the DTO to attach a unique 64 bit work_req_id to |
| 963 |
each work request. The DAPL implementation will install a pointer to a |
| 964 |
DAPL_DTO_COOKIE in this field. Observe that a DAPL_DTO_COOKIE is not |
| 965 |
the same as the user DAT_DTO_COOKIE; indeed, the former has a pointer |
| 966 |
field pointing to the latter. Different values will be placed in the |
| 967 |
cookie, according to the type of operation it is and the type of data |
| 968 |
required by its completion event. This is a simple scheme to bind DAPL |
| 969 |
data to the DTO and associated completion callback. Each DTO has a |
| 970 |
unique cookie associated with it. |
| 971 |
|
| 972 |
Observe that an InfiniBand work_request remains under control of the |
| 973 |
user, and when a post operation occurs the underlying implementation |
| 974 |
will copy data out of the work_request into a hardware based |
| 975 |
structure. Further, no application can perform a DTO operation on the |
| 976 |
same EP at the same time according to the thread guarantees mandated |
| 977 |
by the specification. This allows us to provide a recv_iov and a |
| 978 |
send_iov in the EP structure for all DTO operations, eliminating any |
| 979 |
malloc operations from this critical path. |
| 980 |
|
| 981 |
The underlying provider implementation will invoke |
| 982 |
dapl_evd_dto_callback() upon completion of DTO operations. |
| 983 |
dapl_evd_dto_callback() is the asynchronous completion for a DTO and |
| 984 |
will create and post an event for the user. Much of this callback is |
| 985 |
concerned with managing error completions. |
| 986 |
|
| 987 |
|
| 988 |
====================================================================== |
| 989 |
Data Structure |
| 990 |
====================================================================== |
| 991 |
|
| 992 |
The main data structure for an EndPoint is the dapl_ep structure, |
| 993 |
defined in include/dapl.h. The reference implementation uses the |
| 994 |
InfiniBand QP to maintain hardware state, providing a relatively |
| 995 |
simple mapping. |
| 996 |
|
| 997 |
/* DAPL_EP maps to DAT_EP_HANDLE */ |
| 998 |
struct dapl_ep |
| 999 |
{ |
| 1000 |
DAPL_HEADER header; |
| 1001 |
/* What the DAT Consumer asked for */ |
| 1002 |
DAT_EP_PARAM param; |
| 1003 |
|
| 1004 |
/* The RC Queue Pair (IBM OS API) */ |
| 1005 |
ib_qp_handle_t qp_handle; |
| 1006 |
unsigned int qpn; /* qp number */ |
| 1007 |
ib_qp_state_t qp_state; |
| 1008 |
|
| 1009 |
/* communications manager handle (IBM OS API) */ |
| 1010 |
ib_cm_handle_t cm_handle; |
| 1011 |
/* store the remote IA address here, reference from the param |
| 1012 |
* struct which only has a pointer, no storage |
| 1013 |
*/ |
| 1014 |
DAT_SOCK_ADDR6 remote_ia_address; |
| 1015 |
|
| 1016 |
/* For passive connections we maintain a back pointer to the CR */ |
| 1017 |
void * cr_ptr; |
| 1018 |
|
| 1019 |
/* pointer to connection timer, if set */ |
| 1020 |
struct dapl_timer_entry *cxn_timer; |
| 1021 |
|
| 1022 |
/* private data container */ |
| 1023 |
DAPL_PRIVATE private; |
| 1024 |
|
| 1025 |
/* DTO data */ |
| 1026 |
DAPL_ATOMIC req_count; |
| 1027 |
DAPL_ATOMIC recv_count; |
| 1028 |
|
| 1029 |
DAPL_COOKIE_BUFFER req_buffer; |
| 1030 |
DAPL_COOKIE_BUFFER recv_buffer; |
| 1031 |
|
| 1032 |
ib_data_segment_t *recv_iov; |
| 1033 |
DAT_COUNT recv_iov_num; |
| 1034 |
|
| 1035 |
ib_data_segment_t *send_iov; |
| 1036 |
DAT_COUNT send_iov_num; |
| 1037 |
#ifdef DAPL_DBG_IO_TRC |
| 1038 |
int ibt_dumped; |
| 1039 |
struct io_buf_track *ibt_base; |
| 1040 |
DAPL_RING_BUFFER ibt_queue; |
| 1041 |
#endif /* DAPL_DBG_IO_TRC */ |
| 1042 |
}; |
| 1043 |
|
| 1044 |
The simple explanation of the fields in the dapl_ep structure follows: |
| 1045 |
|
| 1046 |
header: The dapl object header, common to all dapl objects. |
| 1047 |
It contains a lock field, links to appropriate lists, and |
| 1048 |
handles specifying the IA domain it is a part of. |
| 1049 |
|
| 1050 |
param: The bulk of the EP attributes called out in the DAT |
| 1051 |
specification and are maintained in the DAT_EP_PARAM |
| 1052 |
structure. All internal references to these fields |
| 1053 |
use this structure. |
| 1054 |
|
| 1055 |
qp_handle: Handle to the underlying InfiniBand provider implementation |
| 1056 |
for a QP. All EPs are mapped to an InfiniBand QP. |
| 1057 |
|
| 1058 |
qpn: Number of the QP as returned by the underlying provider |
| 1059 |
implementation. Primarily useful for debugging. |
| 1060 |
|
| 1061 |
qp_state: Current state of the QP. The values of this field indicate |
| 1062 |
if a QP is bound to the EP, and the current state of a |
| 1063 |
QP. |
| 1064 |
|
| 1065 |
cm_handle: Handle to the IB provider's CMA (Connection Manager Agent). |
| 1066 |
Used for CM operations used to connect and disconnect. |
| 1067 |
|
| 1068 |
remote_ia_address: |
| 1069 |
Remote IP address of the connection. Only valid after the user |
| 1070 |
has asked for it. |
| 1071 |
|
| 1072 |
cr_ptr: Attaches the EP to the appropriate CR. Assigned on the passive |
| 1073 |
side of a connection in cr_accept. It is used when an abrupt |
| 1074 |
disconnect is invoked by the app, and we need to 'fake' a |
| 1075 |
callback. It is also used in clean up of an EP and removing |
| 1076 |
connection elements from the associated PSP. |
| 1077 |
|
| 1078 |
cxn_timer: Pointer to a timer entry, used as a token to set and remove |
| 1079 |
timers. |
| 1080 |
|
| 1081 |
private: Local Private data area on the active side of a connection. |
| 1082 |
|
| 1083 |
req_count: Count of outstanding request DTO operations, including memory |
| 1084 |
ops. Atomically incremented/decremented. |
| 1085 |
|
| 1086 |
recv_count:Count of outstanding receive DTO operations. Atomically |
| 1087 |
incremented/decremented. |
| 1088 |
|
| 1089 |
req_buffer:Ring buffer of request cookies. |
| 1090 |
|
| 1091 |
recv_buffer: |
| 1092 |
Ring buffer of receive cookies. |
| 1093 |
|
| 1094 |
recv_iov: Storage for provider receive work request. |
| 1095 |
|
| 1096 |
recv_iov_num: |
| 1097 |
Maximum number of receive IOVs. Number is obtained from |
| 1098 |
the provider in a query. |
| 1099 |
|
| 1100 |
send_iov: Storage for provider send work request. |
| 1101 |
|
| 1102 |
send_iov_num: |
| 1103 |
Maximum number of send IOVs. Number is obtained from the |
| 1104 |
provider in a query. |
| 1105 |
|
| 1106 |
ibt_dumped:DTO debugging aid. Boolean value to control how often DTO |
| 1107 |
tracing data is printed. |
| 1108 |
|
| 1109 |
ibt_base: DTO debugging aid. Base address of DTO ring buffer containing |
| 1110 |
information on DTO processing. |
| 1111 |
|
| 1112 |
ibt_queue: Ring buffer containing information n DTO processing. |
| 1113 |
|
| 1114 |
|
| 1115 |
** Debug |
| 1116 |
|
| 1117 |
The Reference Implementation includes a trace facility that allows a |
| 1118 |
developer to see all DTO operations, specifically to catch those that |
| 1119 |
are not completing as expected. The DAPL_DBG_IO_TRC conditional will |
| 1120 |
enable this code. |
| 1121 |
|
| 1122 |
A simple ring buffer is used to account for all outstanding DTO |
| 1123 |
traffic. The buffer may be dumped when DTOs are not getting |
| 1124 |
completions, with enough data to aid the developer to determine where |
| 1125 |
things went wrong. |
| 1126 |
|
| 1127 |
It is implemented as a ring buffer as there are often bugs in this |
| 1128 |
part of a provider's implementation which do not manifest until |
| 1129 |
intensive data exchange has occurred for many hours. |