| 1 |
Suggested Vendor-Specific Changes v. 0.92 |
| 2 |
----------------------------------------- |
| 3 |
|
| 4 |
================= |
| 5 |
Table of Contents |
| 6 |
================= |
| 7 |
|
| 8 |
* Table of Contents |
| 9 |
* Introduction |
| 10 |
* Referenced documents |
| 11 |
* Functionality Changes |
| 12 |
+ Missing Functionality |
| 13 |
+ dat_evd_resize |
| 14 |
+ Ordering guarantees on connect/disconnect. |
| 15 |
+ Shared memory |
| 16 |
+ dat_cr_handoff |
| 17 |
* Performance optimizations |
| 18 |
+ Reduction of context switches |
| 19 |
[Many interrelated optimizations] |
| 20 |
+ Reducing copying of data |
| 21 |
+ Avoidance of s/g list copy on posting |
| 22 |
+ Avoidance of event data copy from CQ to EVD |
| 23 |
+ Elimination of locks |
| 24 |
+ Eliminating subroutine calls |
| 25 |
|
| 26 |
|
| 27 |
============ |
| 28 |
Introduction |
| 29 |
============ |
| 30 |
|
| 31 |
This document is a list of functionality enhancements and |
| 32 |
optimizations hardware vendors porting uDAPL may want to consider as |
| 33 |
part of their port. The functionality enhancements mentioned in this |
| 34 |
document are situations in which HCA Vendors, with their access to |
| 35 |
driver and verb-level source code, and their reduced portability |
| 36 |
concerns, are in a much better position than the reference |
| 37 |
implementation to implement portions of the uDAPL v. 1.0 |
| 38 |
specification. (Additional areas in which the reference |
| 39 |
implementation, because of a lack of time or resources, did not fully |
| 40 |
implement the uDAPL 1.0 specification are not addressed in this file; |
| 41 |
see the file doc/dapl_unimplemented_functionality.txt, forthcoming). |
| 42 |
Vendors should be guided in their implementation of these |
| 43 |
functionality enhancements by their customers need for the features |
| 44 |
involved. |
| 45 |
|
| 46 |
The optimizations suggested in this document have been identified by |
| 47 |
the uDAPL Reference Implementation team as areas in which performance |
| 48 |
may be improved by "breaching" the IB Verbs API boundary. They are |
| 49 |
inappropriate for the reference implementation (which has portability |
| 50 |
as one of its primary goals) but may be appropriate for a HCA-specific |
| 51 |
port of uDAPL. Note that no expected performance gain is attached to |
| 52 |
the suggested optimizations. This is intentional. Vendors should be |
| 53 |
guided in their performance improvements by performance evaluations |
| 54 |
done in the context of a representative workload, and the expected |
| 55 |
benefit from a particular optimization weighed against the cost in |
| 56 |
code complexity and scheduling, before the improvement is implemented. |
| 57 |
This document is intended to seed that process; it is not intended to |
| 58 |
be a roadmap for that process. |
| 59 |
|
| 60 |
We divide functionality changes into two categories |
| 61 |
* Areas in which functionality is lacking in the reference |
| 62 |
implementation. |
| 63 |
* Areas in which the functionality is present in the reference |
| 64 |
implementation, but needs improvement. |
| 65 |
|
| 66 |
We divide performance improvements into three types: |
| 67 |
* Reducing context switches |
| 68 |
* Reducing copying of data (*) |
| 69 |
* Eliminating subroutine calls |
| 70 |
|
| 71 |
(*) Note that the data referred to in "reducing copying of data" is |
| 72 |
the meta data describing an operation (e.g. scatter/gather list or |
| 73 |
event information), not the actual data to be transferred. No data |
| 74 |
transfer copies are required within the uDAPL reference |
| 75 |
implementation. |
| 76 |
|
| 77 |
==================== |
| 78 |
Referenced Documents |
| 79 |
==================== |
| 80 |
|
| 81 |
uDAPL: User Direct Access Programming Library, Version 1.0. Published |
| 82 |
6/21/2002. http://www.datcollaborative.org/uDAPL_062102.pdf. |
| 83 |
Referred to in this document as the "DAT Specification". |
| 84 |
|
| 85 |
InfiniBand Access Application Programming Interface Specification, |
| 86 |
Version 1.2, 4/15/2002. In DAPL SourceForge repository at |
| 87 |
doc/api/access_api.pdf. Referred to in this document as the "IBM |
| 88 |
Access API Specification". |
| 89 |
|
| 90 |
uDAPL Reference Implementation Event System Design. In DAPL |
| 91 |
SourceForge repository at doc/dapl_event_design.txt. |
| 92 |
|
| 93 |
uDAPL Reference Implementation Shared Memory Design. In DAPL |
| 94 |
SourceForge repository at doc/dapl_shared_memory_design.txt. |
| 95 |
|
| 96 |
uDAPL list of unimplmented functionality. In DAPL SourceForge |
| 97 |
repository at doc/dapl_unimplemented_funcitonality.txt (forthcoming). |
| 98 |
|
| 99 |
=========================================== |
| 100 |
Suggested Vendor Functionality Enhancements |
| 101 |
=========================================== |
| 102 |
|
| 103 |
Missing Functionality |
| 104 |
--------------------- |
| 105 |
-- dat_evd_resize |
| 106 |
|
| 107 |
The uDAPL event system does not currently implement dat_evd_resize. |
| 108 |
The primary reason for this is that it is not currently possible to |
| 109 |
identify EVDs with the CQs that back them. Hence uDAPL must keep a |
| 110 |
separate list of events, and any changes to the size of that event |
| 111 |
list would require careful synchronization with all users of that EVD |
| 112 |
(see the uDAPL Event System design for more details). If the various |
| 113 |
vendor specific optimizations in this document were implemented that |
| 114 |
eliminated the requirement for the EVD to keep its own event list, |
| 115 |
dat_evd_resize might be easily implemented by a call or calls to |
| 116 |
ib_cq_resize. |
| 117 |
|
| 118 |
-- Ordering guarantees on connect/disconnect. |
| 119 |
|
| 120 |
The DAPL 1.1 specification specifies that if an EVD combines event |
| 121 |
streams for connection events and DTO events for the same endpoint, |
| 122 |
there is an ordering guarantee: the connection event on the AP occurs |
| 123 |
before any DTO events, and the disconnection event occurs after all |
| 124 |
successful DTO events. Since DTO events are provided by the IBM OS |
| 125 |
Access API through ib_completion_poll (in response to consumer |
| 126 |
request) and connection events are provided through callbacks (which |
| 127 |
may race with consumer requests) there is no trivial way to implement |
| 128 |
this functionality. The functionality may be implemented through |
| 129 |
under the table synchronizations between EVD and EP; specifically: |
| 130 |
* The first time a DTO event is seen on an endpoint, if the |
| 131 |
connection event has not yet arrived it is created and |
| 132 |
delivered ahead of that DTO event. |
| 133 |
* When a connection event is seen on an endpoint, if a |
| 134 |
connection event has already been created for that endpoint |
| 135 |
it is silently discarded. |
| 136 |
* When a disconnection event is seen on an endpoint, it is |
| 137 |
"held" until either: a) all expected DTO events for that |
| 138 |
endpoint have completed, or b) a DTO marked as "flushed by |
| 139 |
disconnect" is received. At that point it is delivered. |
| 140 |
|
| 141 |
Because of the complexity and performance overhead of implementating |
| 142 |
this feature, the DAPL 1.1 reference implementation has chosen to take |
| 143 |
the second approach allowed by the 1.1 specification: disallowing |
| 144 |
integration of connection and data transfer events on the same EVD. |
| 145 |
This fineses the problem, is in accordance with the specification, and |
| 146 |
is more closely aligned with the ITWG IT-API currently in development, |
| 147 |
which only allows a single event stream type for each simple EVD. |
| 148 |
However, other vendors may choose to implement the functionality |
| 149 |
described above in order to support more integration of event streams. |
| 150 |
|
| 151 |
-- Shared memory implementation |
| 152 |
|
| 153 |
The difficulties involved in the dapl shared memory implementation are |
| 154 |
fully described in doc/dapl_shared_memory_design.txt. To briefly |
| 155 |
recap: |
| 156 |
|
| 157 |
The uDAPL spec describes a peer-to-peer shared memory model; all uDAPL |
| 158 |
instances indicate that they want to share registration resources for |
| 159 |
a section of memory do so by providing the same cookie. No uDAPL |
| 160 |
instance is unique; all register their memory in the same way, and no |
| 161 |
communication between the instances is required. |
| 162 |
|
| 163 |
In contrast, the IB shared memory interface requires the first process |
| 164 |
to register the memory to do so using the standard memory registration |
| 165 |
verbs. All other processes after that must use the shared memory |
| 166 |
registration verb, and provide to that verb the memory region handle |
| 167 |
returned from the initial call. This means that the first process to |
| 168 |
register the memory must communicate the memory region handle it |
| 169 |
receives to all the other processes who wish to share this memory. |
| 170 |
This is a master-slave model of shared memory registration; the |
| 171 |
initial process (the master), is unique in its role, and it must tell |
| 172 |
the slaves how to register the memory after it. |
| 173 |
|
| 174 |
To translate between these two models, the uDAPL implementation |
| 175 |
requires some mapping between the shared cookie and the memory region |
| 176 |
handle. This mapping must be exclusive and must have inserts occur |
| 177 |
atomically with lookups (so that only one process can set the memory |
| 178 |
region handle; the others retrieve it). It must also track the |
| 179 |
deregistration of the shared memory, and the exiting of the processes |
| 180 |
registering the shared memory; when all processes have deregistered |
| 181 |
(possibly by exitting) it must remove the mapping from cookie to |
| 182 |
memory region handle. |
| 183 |
|
| 184 |
This mapping must obviously be shared between all uDAPL |
| 185 |
implementations on a given host. Implementing such a shared mapping |
| 186 |
is problematic in a pure user-space implementation (as the reference |
| 187 |
implementation is) but is expected to be relatively easy in vendor |
| 188 |
supplied uDAFS implementations, which will presumably include a |
| 189 |
kernel/device driver component. For this reason, we have chosen to |
| 190 |
leave this functionality unimplemented in the reference implementation. |
| 191 |
|
| 192 |
-- Implementation of dat_cr_handoff |
| 193 |
|
| 194 |
Given that the change of service point involves a change in associated |
| 195 |
connection qualifier, which has been advertised at the underlying |
| 196 |
Verbs/driver level, it is not clear how to implement this function |
| 197 |
cleanly within the reference implementation. We thus choose to defer |
| 198 |
it for implementation by the hardware vendors. |
| 199 |
|
| 200 |
========================= |
| 201 |
Performance Optimizations |
| 202 |
========================= |
| 203 |
|
| 204 |
|
| 205 |
Reduction of context switches |
| 206 |
----------------------------- |
| 207 |
Currently, three context switches are required on the standard |
| 208 |
uDAPL notification path. These are: |
| 209 |
* Invocation of the hardware interrupt handler in the kernel. |
| 210 |
Through this method the hardware notifies the CPU of |
| 211 |
completion queue entries for operations that have requested |
| 212 |
notification. |
| 213 |
* Unblocking of the per-process IB provider service thread |
| 214 |
blocked within the driver. This thread returns to |
| 215 |
user-space within its process, where it causes |
| 216 |
* Unblocking of the user thread blocked within the uDAPL entry |
| 217 |
point (dat_evd_wait() or dat_cno_wait()). |
| 218 |
|
| 219 |
There are several reasons for the high number of context switches, |
| 220 |
specifically: |
| 221 |
* The fact that the IB interface delivers notifications |
| 222 |
through callbacks rather than through unblocking waiting |
| 223 |
threads; this does not match uDAPL's blocking interface. |
| 224 |
* The fact that the IB interface for blocking on a CQ doesn't |
| 225 |
have a threshhold. If it did, we could often convert a |
| 226 |
dat_evd_wait() into a wait on that CQ. |
| 227 |
* The lack of a parallel concept to the CNO within IB. |
| 228 |
|
| 229 |
These are all areas in which closer integration between the IB |
| 230 |
verbs/driver and uDAPL could allow the user thread to wait within the |
| 231 |
driver. This would allow the hardware interrupt thread to directly |
| 232 |
unblock the user thread, saving a context switch. |
| 233 |
|
| 234 |
A specific listing of the optimizations considered here are: |
| 235 |
* Allow blocking on an IB CQ. This would allow removal of the |
| 236 |
excess context switch for dat_evd_wait() in cases where |
| 237 |
there is a 1-to-1 correspondence between an EVD and a CQ and |
| 238 |
no threshold was passed to dat_evd_wait(). |
| 239 |
* Allow blocking on an IB CQ to take a threshold argument. |
| 240 |
This would allow removal of the excess context switch for |
| 241 |
dat_evd_wait() in cases where there is a 1-to-1 |
| 242 |
correspondence between an EVD and a CQ regardless of the |
| 243 |
threshold value. |
| 244 |
* Give the HCA device driver knowledge of and access to the |
| 245 |
implementation of the uDAPL EVD, and implement dat_evd_wait() |
| 246 |
as an ioctl blocking within the device driver. This would |
| 247 |
allow removal of the excess context switch in all cases for |
| 248 |
a dat_evd_wait(). |
| 249 |
* Give the HCA device driver knowledge of and access to the |
| 250 |
implementation of the uDAPL CNO, and implement dat_cno_wait() |
| 251 |
as an ioctl blocking within the device driver. This would |
| 252 |
allow removal of the excess context switch in all cases for |
| 253 |
a dat_cno_wait(), and could improve performance for blocking |
| 254 |
on OS Proxy Wait Objects related to the uDAPL CNO. |
| 255 |
|
| 256 |
See the DAPL Event Subsystem Design (doc/dapl_event_design.txt) for |
| 257 |
more details on this class of optimization. |
| 258 |
|
| 259 |
======================== |
| 260 |
Reducing Copying of Data |
| 261 |
======================== |
| 262 |
|
| 263 |
There are two primary places in which a closer integration between the |
| 264 |
IB verbs/driver and the uDAPL implementation could reducing copying |
| 265 |
costs: |
| 266 |
|
| 267 |
-- Avoidance of s/g list copy on posting |
| 268 |
|
| 269 |
Currently there are two copies involved in posting a data transfer |
| 270 |
request in uDAPL: |
| 271 |
* From the user context to uDAPL. This copy is required |
| 272 |
because the scatter/gather list formats for uDAPL and IB |
| 273 |
differ; a copy is required to change formats. |
| 274 |
* From uDAPL to the WQE. This copy is required because IB |
| 275 |
specifies that all user parameters are owned by the user |
| 276 |
upon return from the IB call, and therefore IB must keep its |
| 277 |
own copy for use during the data transfer operation. |
| 278 |
|
| 279 |
If the uDAPL data transfer dispatch operations were implemented |
| 280 |
directly on the IB hardware, these copies could be combined. |
| 281 |
|
| 282 |
-- Avoidance of Event data copy from CQ to EVD |
| 283 |
|
| 284 |
Currently there are two copies of data involved in receiving an event |
| 285 |
in a standard data transfer operation: |
| 286 |
* From the CQ on which the IB completion occurs to an event |
| 287 |
structure held within the uDAPL EVD. This is because the IB |
| 288 |
verbs provide no way to discover how many elements have been |
| 289 |
posted to a CQ. This copy is not |
| 290 |
required for dat_evd_dequeue. However, dat_evd_wait |
| 291 |
requires this copy in order to correctly implement the |
| 292 |
threshhold argument; the callback must know when to wakeup |
| 293 |
the waiting thread. In addition, copying all CQ entries |
| 294 |
(not just the one to be returned) is necessary before |
| 295 |
returning from dat_evd_wait in order to set the *nmore OUT |
| 296 |
parameter. |
| 297 |
* From the EVD into the event structure provided in the |
| 298 |
dat_evd_wait() call. This copy is required because of the |
| 299 |
DAT specification, which requires a user-provided event |
| 300 |
structure to the dat_evd_wait() call in which the event |
| 301 |
information will be returned. If dat_evd_wait() were |
| 302 |
instead, for example, to hand back a pointer to the already |
| 303 |
allocated event structure, that would eventually require the |
| 304 |
event subsystem to allocate more event structures. This is |
| 305 |
avoided in the critical path. |
| 306 |
|
| 307 |
A tighter integration between the IB verbs/driver and the uDAPL |
| 308 |
implementation would allow the avoidance of the first copy. |
| 309 |
Specifically, providing a way to get information as to the number of |
| 310 |
completions on a CQ would allow avoidance of that copy. |
| 311 |
|
| 312 |
See the uDAPL Event Subsystem Design for more details on this class of |
| 313 |
optimization. |
| 314 |
|
| 315 |
==================== |
| 316 |
Elimination of Locks |
| 317 |
==================== |
| 318 |
|
| 319 |
Currently there is only a single lock used on the critical path in the |
| 320 |
reference implementation, in dat_evd_wait() and dat_evd_dequeue(). |
| 321 |
This lock is in place because the ib_completion_poll() routine is not |
| 322 |
defined as thread safe, and both dat_evd_wait() and dat_evd_dequeue() |
| 323 |
are. If there was some way for a vendor to make ib_completion_poll() |
| 324 |
thread safe without a lock (e.g. if the appropriate hardware/software |
| 325 |
interactions were naturally safe against races), and certain other |
| 326 |
modifications made to the code, the lock might be removed. |
| 327 |
|
| 328 |
The modifications required are: |
| 329 |
* Making racing consumers from DAPL ring buffers thread safe. |
| 330 |
This is possible, but somewhat tricky; the key is to make |
| 331 |
the interaction with the producer occur through a count of |
| 332 |
elements on the ring buffer (atomically incremented and |
| 333 |
decremented), but to dequeue elements with a separate atomic |
| 334 |
pointer increment. The atomic modification of the element |
| 335 |
count synchronizes with the producer and acquires the right |
| 336 |
to do an atomic pointer increment to get the actual data. |
| 337 |
The atomic pointer increment synchronizes with the other |
| 338 |
consumers and actually gets the buffer data. |
| 339 |
* The optimization described above for avoiding copies from |
| 340 |
the CQ to the DAPL EVD Event storage queue. Without this |
| 341 |
optimization a potential race between dat_evd_dequeue() and |
| 342 |
dat_evd_wait() exists where dat_evd_dequeue will return an |
| 343 |
element advanced in the event stream from the one returned |
| 344 |
from dat_evd_wait(): |
| 345 |
|
| 346 |
dat_evd_dequeue() called |
| 347 |
|
| 348 |
EVD state checked; ok for |
| 349 |
dat_evd_dequeue() |
| 350 |
dat_evd_wait() called |
| 351 |
|
| 352 |
State changed to reserve EVD |
| 353 |
for dat_evd_wait() |
| 354 |
|
| 355 |
Partial copy of CQ to EVD Event store |
| 356 |
|
| 357 |
Dequeue of CQE from CQ |
| 358 |
|
| 359 |
Completion of copy of CQ to EVD Event store |
| 360 |
|
| 361 |
Return of first CQE copied to EVD Event store. |
| 362 |
|
| 363 |
Return of thie CQE from the middle |
| 364 |
of the copied stream. |
| 365 |
|
| 366 |
|
| 367 |
If no copy occurs, dat_evd_wait() and dat_evd_dequeue() may |
| 368 |
race, but if all operations on which they may race (access |
| 369 |
to the EVD Event Queue and access to the CQ) are thread |
| 370 |
safe, this race will cause no problems. |
| 371 |
|
| 372 |
============================ |
| 373 |
Eliminating Subroutine Calls |
| 374 |
============================ |
| 375 |
|
| 376 |
This area is the simplest, as there are many DAPL calls on the |
| 377 |
critical path that are very thin veneers on top of their IB |
| 378 |
equivalents. All of these calls are canidates for being merged with |
| 379 |
those IB equivalents. In cases where there are other optimizations |
| 380 |
that may be acheived with the call described above (e.g. within the |
| 381 |
event subsystem, the data transfer operation posting code), that call |
| 382 |
is not mentioned here: |
| 383 |
* dat_pz_create |
| 384 |
* dat_pz_free |
| 385 |
* dat_pz_query |
| 386 |
* dat_lmr_create |
| 387 |
* dat_lmr_free |
| 388 |
* dat_lmr_query |
| 389 |
* dat_rmr_create |
| 390 |
* dat_rmr_free |
| 391 |
* dat_rmr_query |
| 392 |
* dat_rmr_bind |
| 393 |
|
| 394 |
|