Debian Codegraph @

doc/dapl_vendor_specific_changes.txt

raw
1 Suggested Vendor-Specific Changes v. 0.92
2 -----------------------------------------
3
4 =================
5 Table of Contents
6 =================
7
8 * Table of Contents
9 * Introduction
10 * Referenced documents
11 * Functionality Changes
12 + Missing Functionality
13 + dat_evd_resize
14 + Ordering guarantees on connect/disconnect.
15 + Shared memory
16 + dat_cr_handoff
17 * Performance optimizations
18 + Reduction of context switches
19 [Many interrelated optimizations]
20 + Reducing copying of data
21 + Avoidance of s/g list copy on posting
22 + Avoidance of event data copy from CQ to EVD
23 + Elimination of locks
24 + Eliminating subroutine calls
25
26
27 ============
28 Introduction
29 ============
30
31 This document is a list of functionality enhancements and
32 optimizations hardware vendors porting uDAPL may want to consider as
33 part of their port. The functionality enhancements mentioned in this
34 document are situations in which HCA Vendors, with their access to
35 driver and verb-level source code, and their reduced portability
36 concerns, are in a much better position than the reference
37 implementation to implement portions of the uDAPL v. 1.0
38 specification. (Additional areas in which the reference
39 implementation, because of a lack of time or resources, did not fully
40 implement the uDAPL 1.0 specification are not addressed in this file;
41 see the file doc/dapl_unimplemented_functionality.txt, forthcoming).
42 Vendors should be guided in their implementation of these
43 functionality enhancements by their customers need for the features
44 involved.
45
46 The optimizations suggested in this document have been identified by
47 the uDAPL Reference Implementation team as areas in which performance
48 may be improved by "breaching" the IB Verbs API boundary. They are
49 inappropriate for the reference implementation (which has portability
50 as one of its primary goals) but may be appropriate for a HCA-specific
51 port of uDAPL. Note that no expected performance gain is attached to
52 the suggested optimizations. This is intentional. Vendors should be
53 guided in their performance improvements by performance evaluations
54 done in the context of a representative workload, and the expected
55 benefit from a particular optimization weighed against the cost in
56 code complexity and scheduling, before the improvement is implemented.
57 This document is intended to seed that process; it is not intended to
58 be a roadmap for that process.
59
60 We divide functionality changes into two categories
61 * Areas in which functionality is lacking in the reference
62 implementation.
63 * Areas in which the functionality is present in the reference
64 implementation, but needs improvement.
65
66 We divide performance improvements into three types:
67 * Reducing context switches
68 * Reducing copying of data (*)
69 * Eliminating subroutine calls
70
71 (*) Note that the data referred to in "reducing copying of data" is
72 the meta data describing an operation (e.g. scatter/gather list or
73 event information), not the actual data to be transferred. No data
74 transfer copies are required within the uDAPL reference
75 implementation.
76
77 ====================
78 Referenced Documents
79 ====================
80
81 uDAPL: User Direct Access Programming Library, Version 1.0. Published
82 6/21/2002. http://www.datcollaborative.org/uDAPL_062102.pdf.
83 Referred to in this document as the "DAT Specification".
84
85 InfiniBand Access Application Programming Interface Specification,
86 Version 1.2, 4/15/2002. In DAPL SourceForge repository at
87 doc/api/access_api.pdf. Referred to in this document as the "IBM
88 Access API Specification".
89
90 uDAPL Reference Implementation Event System Design. In DAPL
91 SourceForge repository at doc/dapl_event_design.txt.
92
93 uDAPL Reference Implementation Shared Memory Design. In DAPL
94 SourceForge repository at doc/dapl_shared_memory_design.txt.
95
96 uDAPL list of unimplmented functionality. In DAPL SourceForge
97 repository at doc/dapl_unimplemented_funcitonality.txt (forthcoming).
98
99 ===========================================
100 Suggested Vendor Functionality Enhancements
101 ===========================================
102
103 Missing Functionality
104 ---------------------
105 -- dat_evd_resize
106
107 The uDAPL event system does not currently implement dat_evd_resize.
108 The primary reason for this is that it is not currently possible to
109 identify EVDs with the CQs that back them. Hence uDAPL must keep a
110 separate list of events, and any changes to the size of that event
111 list would require careful synchronization with all users of that EVD
112 (see the uDAPL Event System design for more details). If the various
113 vendor specific optimizations in this document were implemented that
114 eliminated the requirement for the EVD to keep its own event list,
115 dat_evd_resize might be easily implemented by a call or calls to
116 ib_cq_resize.
117
118 -- Ordering guarantees on connect/disconnect.
119
120 The DAPL 1.1 specification specifies that if an EVD combines event
121 streams for connection events and DTO events for the same endpoint,
122 there is an ordering guarantee: the connection event on the AP occurs
123 before any DTO events, and the disconnection event occurs after all
124 successful DTO events. Since DTO events are provided by the IBM OS
125 Access API through ib_completion_poll (in response to consumer
126 request) and connection events are provided through callbacks (which
127 may race with consumer requests) there is no trivial way to implement
128 this functionality. The functionality may be implemented through
129 under the table synchronizations between EVD and EP; specifically:
130 * The first time a DTO event is seen on an endpoint, if the
131 connection event has not yet arrived it is created and
132 delivered ahead of that DTO event.
133 * When a connection event is seen on an endpoint, if a
134 connection event has already been created for that endpoint
135 it is silently discarded.
136 * When a disconnection event is seen on an endpoint, it is
137 "held" until either: a) all expected DTO events for that
138 endpoint have completed, or b) a DTO marked as "flushed by
139 disconnect" is received. At that point it is delivered.
140
141 Because of the complexity and performance overhead of implementating
142 this feature, the DAPL 1.1 reference implementation has chosen to take
143 the second approach allowed by the 1.1 specification: disallowing
144 integration of connection and data transfer events on the same EVD.
145 This fineses the problem, is in accordance with the specification, and
146 is more closely aligned with the ITWG IT-API currently in development,
147 which only allows a single event stream type for each simple EVD.
148 However, other vendors may choose to implement the functionality
149 described above in order to support more integration of event streams.
150
151 -- Shared memory implementation
152
153 The difficulties involved in the dapl shared memory implementation are
154 fully described in doc/dapl_shared_memory_design.txt. To briefly
155 recap:
156
157 The uDAPL spec describes a peer-to-peer shared memory model; all uDAPL
158 instances indicate that they want to share registration resources for
159 a section of memory do so by providing the same cookie. No uDAPL
160 instance is unique; all register their memory in the same way, and no
161 communication between the instances is required.
162
163 In contrast, the IB shared memory interface requires the first process
164 to register the memory to do so using the standard memory registration
165 verbs. All other processes after that must use the shared memory
166 registration verb, and provide to that verb the memory region handle
167 returned from the initial call. This means that the first process to
168 register the memory must communicate the memory region handle it
169 receives to all the other processes who wish to share this memory.
170 This is a master-slave model of shared memory registration; the
171 initial process (the master), is unique in its role, and it must tell
172 the slaves how to register the memory after it.
173
174 To translate between these two models, the uDAPL implementation
175 requires some mapping between the shared cookie and the memory region
176 handle. This mapping must be exclusive and must have inserts occur
177 atomically with lookups (so that only one process can set the memory
178 region handle; the others retrieve it). It must also track the
179 deregistration of the shared memory, and the exiting of the processes
180 registering the shared memory; when all processes have deregistered
181 (possibly by exitting) it must remove the mapping from cookie to
182 memory region handle.
183
184 This mapping must obviously be shared between all uDAPL
185 implementations on a given host. Implementing such a shared mapping
186 is problematic in a pure user-space implementation (as the reference
187 implementation is) but is expected to be relatively easy in vendor
188 supplied uDAFS implementations, which will presumably include a
189 kernel/device driver component. For this reason, we have chosen to
190 leave this functionality unimplemented in the reference implementation.
191
192 -- Implementation of dat_cr_handoff
193
194 Given that the change of service point involves a change in associated
195 connection qualifier, which has been advertised at the underlying
196 Verbs/driver level, it is not clear how to implement this function
197 cleanly within the reference implementation. We thus choose to defer
198 it for implementation by the hardware vendors.
199
200 =========================
201 Performance Optimizations
202 =========================
203
204
205 Reduction of context switches
206 -----------------------------
207 Currently, three context switches are required on the standard
208 uDAPL notification path. These are:
209 * Invocation of the hardware interrupt handler in the kernel.
210 Through this method the hardware notifies the CPU of
211 completion queue entries for operations that have requested
212 notification.
213 * Unblocking of the per-process IB provider service thread
214 blocked within the driver. This thread returns to
215 user-space within its process, where it causes
216 * Unblocking of the user thread blocked within the uDAPL entry
217 point (dat_evd_wait() or dat_cno_wait()).
218
219 There are several reasons for the high number of context switches,
220 specifically:
221 * The fact that the IB interface delivers notifications
222 through callbacks rather than through unblocking waiting
223 threads; this does not match uDAPL's blocking interface.
224 * The fact that the IB interface for blocking on a CQ doesn't
225 have a threshhold. If it did, we could often convert a
226 dat_evd_wait() into a wait on that CQ.
227 * The lack of a parallel concept to the CNO within IB.
228
229 These are all areas in which closer integration between the IB
230 verbs/driver and uDAPL could allow the user thread to wait within the
231 driver. This would allow the hardware interrupt thread to directly
232 unblock the user thread, saving a context switch.
233
234 A specific listing of the optimizations considered here are:
235 * Allow blocking on an IB CQ. This would allow removal of the
236 excess context switch for dat_evd_wait() in cases where
237 there is a 1-to-1 correspondence between an EVD and a CQ and
238 no threshold was passed to dat_evd_wait().
239 * Allow blocking on an IB CQ to take a threshold argument.
240 This would allow removal of the excess context switch for
241 dat_evd_wait() in cases where there is a 1-to-1
242 correspondence between an EVD and a CQ regardless of the
243 threshold value.
244 * Give the HCA device driver knowledge of and access to the
245 implementation of the uDAPL EVD, and implement dat_evd_wait()
246 as an ioctl blocking within the device driver. This would
247 allow removal of the excess context switch in all cases for
248 a dat_evd_wait().
249 * Give the HCA device driver knowledge of and access to the
250 implementation of the uDAPL CNO, and implement dat_cno_wait()
251 as an ioctl blocking within the device driver. This would
252 allow removal of the excess context switch in all cases for
253 a dat_cno_wait(), and could improve performance for blocking
254 on OS Proxy Wait Objects related to the uDAPL CNO.
255
256 See the DAPL Event Subsystem Design (doc/dapl_event_design.txt) for
257 more details on this class of optimization.
258
259 ========================
260 Reducing Copying of Data
261 ========================
262
263 There are two primary places in which a closer integration between the
264 IB verbs/driver and the uDAPL implementation could reducing copying
265 costs:
266
267 -- Avoidance of s/g list copy on posting
268
269 Currently there are two copies involved in posting a data transfer
270 request in uDAPL:
271 * From the user context to uDAPL. This copy is required
272 because the scatter/gather list formats for uDAPL and IB
273 differ; a copy is required to change formats.
274 * From uDAPL to the WQE. This copy is required because IB
275 specifies that all user parameters are owned by the user
276 upon return from the IB call, and therefore IB must keep its
277 own copy for use during the data transfer operation.
278
279 If the uDAPL data transfer dispatch operations were implemented
280 directly on the IB hardware, these copies could be combined.
281
282 -- Avoidance of Event data copy from CQ to EVD
283
284 Currently there are two copies of data involved in receiving an event
285 in a standard data transfer operation:
286 * From the CQ on which the IB completion occurs to an event
287 structure held within the uDAPL EVD. This is because the IB
288 verbs provide no way to discover how many elements have been
289 posted to a CQ. This copy is not
290 required for dat_evd_dequeue. However, dat_evd_wait
291 requires this copy in order to correctly implement the
292 threshhold argument; the callback must know when to wakeup
293 the waiting thread. In addition, copying all CQ entries
294 (not just the one to be returned) is necessary before
295 returning from dat_evd_wait in order to set the *nmore OUT
296 parameter.
297 * From the EVD into the event structure provided in the
298 dat_evd_wait() call. This copy is required because of the
299 DAT specification, which requires a user-provided event
300 structure to the dat_evd_wait() call in which the event
301 information will be returned. If dat_evd_wait() were
302 instead, for example, to hand back a pointer to the already
303 allocated event structure, that would eventually require the
304 event subsystem to allocate more event structures. This is
305 avoided in the critical path.
306
307 A tighter integration between the IB verbs/driver and the uDAPL
308 implementation would allow the avoidance of the first copy.
309 Specifically, providing a way to get information as to the number of
310 completions on a CQ would allow avoidance of that copy.
311
312 See the uDAPL Event Subsystem Design for more details on this class of
313 optimization.
314
315 ====================
316 Elimination of Locks
317 ====================
318
319 Currently there is only a single lock used on the critical path in the
320 reference implementation, in dat_evd_wait() and dat_evd_dequeue().
321 This lock is in place because the ib_completion_poll() routine is not
322 defined as thread safe, and both dat_evd_wait() and dat_evd_dequeue()
323 are. If there was some way for a vendor to make ib_completion_poll()
324 thread safe without a lock (e.g. if the appropriate hardware/software
325 interactions were naturally safe against races), and certain other
326 modifications made to the code, the lock might be removed.
327
328 The modifications required are:
329 * Making racing consumers from DAPL ring buffers thread safe.
330 This is possible, but somewhat tricky; the key is to make
331 the interaction with the producer occur through a count of
332 elements on the ring buffer (atomically incremented and
333 decremented), but to dequeue elements with a separate atomic
334 pointer increment. The atomic modification of the element
335 count synchronizes with the producer and acquires the right
336 to do an atomic pointer increment to get the actual data.
337 The atomic pointer increment synchronizes with the other
338 consumers and actually gets the buffer data.
339 * The optimization described above for avoiding copies from
340 the CQ to the DAPL EVD Event storage queue. Without this
341 optimization a potential race between dat_evd_dequeue() and
342 dat_evd_wait() exists where dat_evd_dequeue will return an
343 element advanced in the event stream from the one returned
344 from dat_evd_wait():
345
346 dat_evd_dequeue() called
347
348 EVD state checked; ok for
349 dat_evd_dequeue()
350 dat_evd_wait() called
351
352 State changed to reserve EVD
353 for dat_evd_wait()
354
355 Partial copy of CQ to EVD Event store
356
357 Dequeue of CQE from CQ
358
359 Completion of copy of CQ to EVD Event store
360
361 Return of first CQE copied to EVD Event store.
362
363 Return of thie CQE from the middle
364 of the copied stream.
365
366
367 If no copy occurs, dat_evd_wait() and dat_evd_dequeue() may
368 race, but if all operations on which they may race (access
369 to the EVD Event Queue and access to the CQ) are thread
370 safe, this race will cause no problems.
371
372 ============================
373 Eliminating Subroutine Calls
374 ============================
375
376 This area is the simplest, as there are many DAPL calls on the
377 critical path that are very thin veneers on top of their IB
378 equivalents. All of these calls are canidates for being merged with
379 those IB equivalents. In cases where there are other optimizations
380 that may be acheived with the call described above (e.g. within the
381 event subsystem, the data transfer operation posting code), that call
382 is not mentioned here:
383 * dat_pz_create
384 * dat_pz_free
385 * dat_pz_query
386 * dat_lmr_create
387 * dat_lmr_free
388 * dat_lmr_query
389 * dat_rmr_create
390 * dat_rmr_free
391 * dat_rmr_query
392 * dat_rmr_bind
393
394