Debian Codegraph @

doc/dapl_shared_memory_design.txt

raw
1 #######################################################################
2 # #
3 # DAPL Shared Memory Design #
4 # #
5 # James Lentini #
6 # jlentini at users.sourceforge.net #
7 # #
8 # Created 09/17/2002 #
9 # Updated 01/21/2005 #
10 # Version 0.04 #
11 # #
12 #######################################################################
13
14
15 Contents
16 --------
17 0. Introduction
18 1. Referenced Documents
19 2. Requirements
20 3. Interface
21 4. Implementation Options
22
23
24 Introduction
25 ------------
26
27 This document describes the design of shared memory registration for
28 the DAPL reference implementation (RI).
29
30 Implementing shared memory support completely within the DAPL RI
31 would not be an ideal solution. A more robust and efficient
32 implementation can be acheived by HCA vendors that integrate a DAT
33 provider into their software stack. Therefore the RI will not contain
34 an implementation of this feature.
35
36
37 Referenced Documents
38 --------------------
39
40 kDAPL: Kernel Direct Access Programming Library, Version 1.2.
41 uDAPL: User Direct Access Programming Library, Version 1.2.
42 Available in the DAPL SourceForge repository at
43 [doc/api/kDAPL_spec.pdf] and [doc/api/uDAPL_spec.pdf]. Collectively
44 referred to in this document as the "DAT Specification".
45
46 InfiniBand Access Application Programming Interface Specification,
47 Version 1.2, 4/15/2002. Available in the DAPL SourceForge repository
48 at [doc/api/IBM_access_api.pdf]. Referred to in this document as the
49 "IBM Access API Specification".
50
51 Mellanox IB-Verbs API (VAPI) Mellanox Software Programmers Interface
52 for InfiniBand Verbs. Available in the DAPL SourceForge repository
53 at [doc/api/MellanoxVerbsAPI.pdf]. Referred to in this document as the
54 "VAPI API Specification".
55
56 InfiniBand Architecture Specification, Volumes 1 and 2, Release
57 1.2, Available from http://www.infinibandta.org/
58 Referred to in this document as the "Infiniband Specification".
59
60
61 Requirements
62 ------------
63
64 The DAT shared memory model can be characterized as a peer-to-peer
65 model since the order in which consumers register a region is not
66 dictated by the programming interface.
67
68 The DAT API function used to register shared memory is:
69
70 DAT_RETURN
71 dat_lmr_create (
72 IN DAT_IA_HANDLE ia_handle,
73 IN DAT_MEM_TYPE mem_type,
74 IN DAT_REGION_DESCRIPTION region_description,
75 IN DAT_VLEN length,
76 IN DAT_PZ_HANDLE pz_handle,
77 IN DAT_MEM_PRIV_FLAGS mem_privileges,
78 OUT DAT_LMR_HANDLE *lmr_handle,
79 OUT DAT_LMR_CONTEXT *lmr_context,
80 OUT DAT_RMR_CONTEXT *rmr_context,
81 OUT DAT_VLEN *registered_length,
82 OUT DAT_VADDR *registered_address );
83
84 where a DAT_REGION_DESCRIPTION is defined as:
85
86 typedef union dat_region_description
87 {
88 DAT_PVOID for_va;
89 DAT_LMR_HANDLE for_lmr_handle;
90 DAT_SHARED_MEMORY for_shared_memory;
91 } DAT_REGION_DESCRIPTION;
92
93 In the case of a shared memory registration the DAT consumer will set
94 the DAT_MEM_TYPE flag to DAT_MEM_TYPE_SHARED_VIRTUAL and place a
95 cookie in the DAT_REGION_DESCRIPTION union's DAT_SHARED_MEMORY
96 member. The DAT_SHARED_MEMORY type is defined as follows:
97
98 typedef struct dat_shared_memory
99 {
100 DAT_PVOID virtual_address;
101 DAT_LMR_COOKIE shared_memory_id;
102 } DAT_SHARED_MEMORY;
103
104 Unlike the DAT peer-to-peer model, the Infiniband shared memory model
105 requires a master-slave relationship. A memory region must first be
106 registered using the Register Memory Region verb with subsequent
107 registrations made using the Register Shared Memory Region verb.
108
109 The later is implemented in the IBM OS Access API as:
110
111 ib_int32_t
112 ib_mr_shared_register_us(
113 ib_hca_handle_t hca_handle,
114 ib_mr_handle_t *mr_handle, /* IN-OUT: could be changed */
115 ib_pd_handle_t pd_handle, /* IN */
116 ib_uint32_t access_control, /* IN */
117 ib_uint32_t *l_key, /* OUT */
118 ib_uint32_t *r_key, /* OUT: if remote access needed */
119 ib_uint8_t **va ); /* IN-OUT: virt. addr. to register */
120
121 The important parameter is the memory region handle which must be the
122 same as an already registered region.
123
124 Two requirements are implied by this difference between the DAT and
125 Infiniband models. First, DAPL implementations need a way to determine
126 the first registration of a shared region. Second implementations must
127 map DAT_LMR_COOKIE values to memory region handles both within and
128 across processes. To satisfy the above requirements DAPL must maintain
129 this information in a system wide database.
130
131 The difficulty of implementing such a database at the DAT provider
132 level is the reason the RI's shared memory code is meant to be
133 temporary. Such a database is much better suited as part of the HCA
134 vendor's software stack, specifically as part of their HCA driver.
135
136 If DAPL was based on a master-slave model like InfiniBand, the
137 implementation of shared memory would be straight
138 forward. Specifically the complexity is a result of the consumer being
139 responsible for specifying the DAT_LMR_COOKIE values. If the DAPL
140 spec. were changed to allow the provider and not the consumer to
141 specify the DAT_LMR_COOKIE value, the implementation of this feature
142 would be greatly simplified. Since the DAPL API already requires
143 consumers to communicate the DAT_LMR_COOKIE values between processes,
144 such a change places minimal additional requirements on the
145 consumer. The dapl_lmr_query call could easily be adapted to allow the
146 consumer to query the provider for a given LMR's DAT_LMR_COOKIE
147 value. The only spec changes needed would be to add a DAT_LMR_COOKIE
148 member to the DAT_LMR_PARAM structure and a DAT_LMR_FIELD_LMR_COOKIE
149 constant to the DAT_LMR_PARAM_MASK enumeration. A provider could then
150 store the given LMR's memory region handle in this value, greatly
151 simplifying the implementation of shared memory in DAPL.
152
153
154 Interface
155 ---------
156
157 To allow the database implementation to easily change, the RI would use
158 a well defined interface between the memory subsystem and the
159 database. Conceptually the database would contain a single table with
160 the following columns:
161
162 [ LMR Cookie ][ MR Handle ][ Reference Count ][ Initialized ]
163
164 where the LMR Cookie column is the primary key.
165
166 The following functions would be used to access the database:
167
168 DAT_RETURN
169 dapls_mrdb_init (
170 void );
171
172 Called by dapl_init(.) to perform any necessary database
173 initialization.
174
175 DAT_RETURN
176 dapls_mrdb_exit (
177 void );
178
179 Called by dapl_fini(.) to perform any necessary database cleanup.
180
181 DAT_RETURN
182 dapls_mrdb_record_insert (
183 IN DAPL_LMR_COOKIE cookie );
184
185 If there is no record for the specified cookie, an empty record is
186 added with a reference count of 1 and the initialized field is set to
187 false. If a record already exists, the function returns an error.
188
189 DAT_RETURN
190 dapls_mrdb_record_update (
191 IN DAPL_LMR_COOKIE cookie,
192 IN ib_mr_handle_t mr_handle );
193
194 If there is a record for the specified cookie, the MR handle field is
195 set to the specified mr_handle value and the initialized field is set
196 to true. Otherwise an error is returned.
197
198 DAT_RETURN
199 dapls_mrdb_record_query (
200 IN DAPL_LMR_COOKIE cookie,
201 OUT ib_mr_handle_t *mr_handle );
202
203 If there is a record for the specified cookie and the initialized
204 field is true, the MR handle field is returned and the reference
205 count field is incremented. Otherwise an error is returned.
206
207 DAT_RETURN
208 dapls_mrdb_record_dec (
209 IN DAPL_LMR_COOKIE cookie );
210
211 If there is a record for the specified cookie, the reference count
212 field is decremented. If the reference count is zero after the
213 decrement, the record is removed from the database. Otherwise an
214 error is returned.
215
216 The generic algorithms for creating and destroying a shared memory
217 region are:
218
219 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
220 Function: CreateShared
221 Inputs:
222 ia_handle
223 pz_handle
224 address
225 length
226 lmr_cookie
227 privileges
228 Outputs:
229 lmr_handle
230 lmr_context
231 registered_address
232 registered_length
233
234 forever
235 {
236 if dapls_mrdb_record_insert(cookie) is successful
237 {
238 if dapl_lmr_create_virtual is not successful
239 dapls_mrdb_record_dec(cookie)
240 return error
241
242 else if dapls_mrdb_record_update(cookie, lmr->mr_handle) is not successful
243 dapls_mrdb_record_dec(cookie)
244 return error
245
246 else break
247 }
248 else if dapls_mrdb_record_query(cookie, mr_handle) is successful
249 {
250 if ib_mrdb_shared_register_us is not successful
251 dapls_mrdb_record_dec(cookie)
252 return error
253
254 else break
255 }
256 }
257
258 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
259
260 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
261 Function: FreeShared
262 Inputs:
263 lmr
264 Outputs:
265
266 if dapls_ib_mr_deregister(lmr) is successful
267 dapls_mrdb_record_dec(lmr->cookie)
268
269 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
270
271
272 Implementation Options
273 ----------------------
274
275 As described above the crucial functionality needed to implement
276 shared memory support is a system wide database for mapping LMR
277 cookies to memory region handles. The following designs represent some
278 of the options for implementing such a database. Adding a database
279 increases the complexity of DAPL from both an implementor and user's
280 perspective. These designs should be evaluated on the degree to which
281 they minimize the additional complexity while still providing a robust
282 solution.
283
284
285 File System Database
286 --------------------
287
288 Employing a database that is already part of the system would be
289 ideal. One option on Linux is to use the file system. An area of the
290 file system could be set aside for the creation of files to represent
291 each LMR cookie. The area of the file system could be specified
292 through a hard coded value, an environment variable, or a
293 configuration file. A good candidate would be a DAPL subdirectory of
294 /tmp.
295
296 Exclusive file creation is available through the creat(2) system call
297 in Linux. The standard I/O interface (fopen(3), etc.) does not support
298 this feature making porting difficult. However porting to other
299 environments is not a goal of this design since the entire scheme is
300 only a temporary solution.
301
302 Determining when to delete the files is a difficult problem. A
303 reference count is required to properly remove a file once all the
304 memory regions it represents are deregistered. The synchronization
305 mechanism necessary for maintaining the reference count is not easily
306 implemented. As an alternative, a script could be provided to clean up
307 the database by removing all the files. The script would need to be
308 run before any DAPL consumers were started to ensure a clean
309 database. The disadvantage of using a script is that no DAPL instances
310 can be running when it is used. Another option would be to store the
311 process ID (PID) of the process that created the file as part of the
312 file's contents. Upon finding a record for a given LMR cookie value, a
313 DAPL instance could determine if there was a process with the same PID
314 in the system. To accomplish this the kill(2) system call could be
315 used (ex. kill(pid, 0) ). This method of validating the record assumes
316 that all DAPL instances can signal one another and that the PID values
317 do not wrap before the check is made.
318
319 Another difficulty with this solution is choosing an accessible
320 portion of the file system. The area must have permissions that allow
321 all processes using DAPL to access and modify its contents. System
322 administrators are typically reluctant to allow areas without any
323 access controls. Typically such areas are on a dedicated file system
324 of a minimal size to ensure that malicious or malfunctioning software
325 does not monopolize the system's storage capacity. Since very little
326 information will be stored in each file it is unlikely that DAPL would
327 need a large amount of storage space even if a large number of shared
328 memory regions were in use. However since a file is needed for each
329 shared region, a large number of shared registrations may lead to the
330 consumption of all a file system's inodes. Again since this solution
331 is meant to be only temporary this constraint may be acceptable.
332
333 There is also the possibility for database corruption should a process
334 crash or deadlock at an inopportune time. If a process creates file x
335 and then crashes all other processes waiting for the memory handle to
336 be written to x will fail.
337
338 The database interface could be implemented as follows:
339
340 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
341 Function: dapls_mrdb_init
342 Inputs:
343
344 Outputs:
345
346 return success
347
348 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
349
350 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
351 Function: dapls_mrdb_exit
352 Inputs:
353
354 Outputs:
355
356 return success
357
358 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
359
360 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
361 Function: dapls_mrdb_record_insert
362 Inputs:
363 cookie
364 Outputs:
365
366 file_name = convert cookie to valid file name
367
368 fd = exclusively create file_name
369 if fd is invalid
370 return failure
371
372 if close fd fails
373 return failure
374
375 return success
376
377 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
378
379 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
380 Function: dapls_mrdb_record_update
381 Inputs:
382 cookie
383 mr_handle
384 Outputs:
385
386 file_name = convert cookie to valid file name
387
388 fd = open file_name
389 if fd is invalid
390 return failure
391
392 if write mr_handle to file_name fails
393 return failure
394
395 if close fd fails
396 return failure
397
398 return success
399
400 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
401
402 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
403 Function: dapls_mrdb_record_query
404 Inputs:
405 cookie
406
407 Outputs:
408 mr_handle
409
410 file_name = convert cookie to valid file name
411
412 fd = open file_name
413 if fd is invalid
414 return failure
415
416 if read mr_handle from file_name fails
417 return failure
418
419 if close fd fails
420 return failure
421
422 return success
423
424 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
425
426 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
427 Function: dapls_mrdb_record_dec
428 Inputs:
429 cookie
430 Outputs:
431
432 return success
433
434 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
435
436
437 Daemon Database
438 ---------------
439
440 The database could be maintained by a separate daemon process.
441 The DAPL instances would act as clients of the daemon server and
442 communicate with the daemon through the various IPC mechanisms
443 available on Linux: Unix sockets, TCP/IP sockets, System V message
444 queues, FIFOs, or RPCs.
445
446 As with the file system based database, process crashes can potentially
447 cause database corruption.
448
449 While the portability of this implementation will depend on the chosen
450 IPC mechanism, this approach will be at best Unix centric and possibly
451 Linux specific.
452
453 The database interface could be implemented as follows:
454
455 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
456 Function: dapls_mrdb_init
457 Inputs:
458
459 Outputs:
460
461 initialize IPC mechanism
462
463 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
464
465 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
466 Function: dapls_mrdb_exit
467 Inputs:
468
469 Outputs:
470
471 shutdown IPC mechanism
472
473 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
474
475 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
476 Function: dapls_mrdb_record_insert
477 Inputs:
478 cookie
479 Outputs:
480
481 if send insert message for cookie fails
482 return error
483
484 if receive insert response message fails
485 return error
486
487 if insert success
488 return success
489 else return error
490
491 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
492
493 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
494 Function: dapls_mrdb_record_update
495 Inputs:
496 cookie
497 mr_handle
498
499 Outputs:
500
501 if send update message for cookie and mr_handle fails
502 return error
503 else return success
504
505 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
506
507 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
508 Function: dapls_mrdb_record_query
509 Inputs:
510 cookie
511
512 Outputs:
513 mr_handle
514
515 if send query message for cookie fails
516 return error
517
518 else if receive query response message with mr_handle fails
519 return error
520
521 else return success
522
523 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
524
525 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
526 Function: dapls_mrdb_record_dec
527 Inputs:
528 cookie
529 Outputs:
530
531 if send decrement message for cookie fails
532 return error
533 else return success
534
535 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
536
537
538 Shared Memory Database
539 ----------------------
540
541 The database could be maintained in an area of memory shared by all
542 DAPL instances running on a system. Linux supports the System V shared
543 memory functions shmget(2), shmctl(2), shmat(2), and shmdt(2). A hard
544 coded key_t value could be used so that each DAPL instance attached to
545 the same piece of shared memory. The size of the database would be
546 constrained by the size of the shared memory region. Synchronization
547 could be achieved by using atomic operations targeting memory in the
548 shared region.
549
550 Such a design would suffer from the corruption problems described
551 above. If a process crashed there would be no easy way to clean up its
552 locks and roll back the database to a consistent state.
553
554 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
555 Function: dapls_mrdb_init
556 Inputs:
557
558 Outputs:
559
560 attach shared region
561
562 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
563
564 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
565 Function: dapls_mrdb_exit
566 Inputs:
567
568 Outputs:
569
570 detach shared region
571
572 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
573
574 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
575 Function: dapls_mrdb_record_insert
576 Inputs:
577 cookie
578 Outputs:
579
580 lock database
581
582 if db does not contain cookie
583 add record for cookie
584
585 unlock database
586
587 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
588
589 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
590 Function: dapls_mrdb_record_update
591 Inputs:
592 cookie
593 mr_handle
594 Outputs:
595
596 lock database
597
598 if db contains cookie
599 update record's mr_handle
600
601 unlock database
602
603 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
604
605 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
606 Function: dapls_mrdb_record_query
607 Inputs:
608 cookie
609
610 Outputs:
611 mr_handle
612
613 lock database
614
615 if db contains cookie
616 set mr_handle to record's value
617 increment record's reference count
618
619 unlock database
620
621 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
622
623 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
624 Function: dapls_mrdb_record_dec
625 Inputs:
626 cookie
627 Outputs:
628
629 lock database
630
631 if db contains cookie
632 decrement record's reference count
633
634 if reference count is 0
635 remove record
636
637 unlock database
638
639 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
640
641
642 Kernel Module Database
643 ----------------------
644
645 If the DAT library were integrated with an HCA vendor's software
646 stack, the database could be managed by the HCA driver. Placing the
647 database in the kernel would alleviate the synchronization problems
648 posed by multiple processes. Since memory registration operations
649 already involve a transition into the kernel, no extra overhead would
650 be incurred by this design.
651
652 The RI could include a kernel module with this functionality as a
653 temporary solution. The module could identify itself as a character
654 device driver and communicate with user level processes through an
655 ioctl(2). The driver could also create an entry in the proc file
656 system to display the database's contents for diagnostic purposes.
657
658 A major benefit of a kernel based implementation is that the database
659 can remain consistent even in the presence of application
660 errors. Since DAPL instances communicate with the driver by means of
661 ioctl(2) calls on a file, the driver can be arrange to be informed
662 when the file is closed and perform any necessary actions. The driver
663 is guaranteed to be notified of a close regardless of the manner in
664 which the process exits.
665
666 The database could be implemented as a dictionary using the LMR cookie
667 values as keys.
668
669 The following pseudo-code describes the functions needed by the kernel
670 module and the database interface.
671
672 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
673 Function: KernelModuleInit
674 Inputs:
675
676 Outputs:
677
678 dictionary = create_dictionary()
679 create_proc_entry()
680 create_character_device_entry()
681
682 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
683
684 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
685 Function: KernelModuleExit
686 Inputs:
687
688 Outputs:
689
690 remove_character_device_entry()
691 remove_proc_entry()
692 fee_dictionary(dictionary)
693
694 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
695
696 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
697 Function: DeviceOpen
698 Inputs:
699 file
700
701 Outputs:
702
703 dev_data = allocate device data
704
705 file->private_data = dev_data
706
707 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
708
709 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
710 Function: DeviceClose
711 Inputs:
712 file
713
714 Outputs:
715
716 dev_data = file->private_data
717
718 for each record in dev_data
719 {
720 RecordDecIoctl
721 }
722
723 deallocate dev_data
724
725 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
726
727 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
728 Function: RecordInsertIoctl
729 Inputs:
730 file
731 cookie
732
733 Outputs:
734
735 lock dictionary
736
737 if cookie is not in dictionary
738 insert cookie into dictionary
739
740
741 unlock dictionary
742
743 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
744
745 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
746 Function: RecordUpdateIoctl
747 Inputs:
748 file
749 cookie
750 mr_handle
751
752 Outputs:
753
754 dev_data = file->private_data
755
756 lock dictionary
757
758 if cookie is in dictionary
759 add record reference to dev_data
760 update mr_handle
761
762 unlock dictionary
763
764 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
765
766 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
767 Function: RecordQueryIoctl
768 Inputs:
769 file
770 cookie
771
772 Outputs:
773 mr_handle
774
775 dev_data = file->private_data
776
777 lock dictionary
778
779 if cookie is in dictionary
780 add record reference to dev_data
781 retrieve mr_handle
782
783 unlock dictionary
784
785 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
786
787 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
788 Function: RecordDecIoctl
789 Inputs:
790 file
791 cookie
792
793 Outputs:
794
795 dev_data = file->private_data
796 remove record reference from dev_data
797
798 lock dictionary
799
800 if cookie is in dictionary
801 decrement reference count
802 if reference count is 0
803 remove record
804
805 unlock dictionary
806
807
808 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
809
810 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
811 Function: dapls_mrdb_init
812 Inputs:
813
814 Outputs:
815
816 fd = open device file
817
818 if fd is invalid
819 return error
820 else
821 return success
822
823 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
824
825 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
826 Function: dapls_mrdb_exit
827 Inputs:
828
829 Outputs:
830
831 close fd
832
833 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
834
835 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
836 Function: dapls_mrdb_record_insert
837 Inputs:
838 cookie
839 Outputs:
840
841 ioctl on fd
842
843 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
844
845 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
846 Function: dapls_mrdb_record_update
847 Inputs:
848 cookie
849 mr_handle
850 Outputs:
851
852 ioctl on fd
853
854 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
855
856 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
857 Function: dapls_mrdb_record_query
858 Inputs:
859 cookie
860
861 Outputs:
862 mr_handle
863
864 ioctl on fd
865
866 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
867
868 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
869 Function: dapls_mrdb_record_dec
870 Inputs:
871 cookie
872 Outputs:
873
874 ioctl on fd
875
876 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+