Debian Codegraph @

doc/dapl_memory_management_design.txt

raw
1 #######################################################################
2 # #
3 # DAPL Memory Management Design #
4 # #
5 # James Lentini #
6 # jlentini at users.sourceforge.net #
7 # #
8 # Created 05/06/2002 #
9 # Updated 08/22/2002 #
10 # #
11 #######################################################################
12
13
14 Contents
15 -------
16 0. Introduction
17 1. Protection Zones (PZs)
18 2. Local Memory Regions (LMRs)
19 3. Remote Memory Regions (RMRs)
20
21
22 0. Introduction
23 ---------------
24
25 The memory management subsystem allows consumers to register and
26 unregister memory regions. The DAT API distinguishes between local
27 and remote memory areas. The former server as local buffers for DTO
28 operations while the later are used for RDMA operations.
29
30 Each DAT function is implemented in a file named dapl_<function name>.c.
31 For example, dat_pz_create is implemented in dapl_pz_create.c. There
32 are also dapl_<object name>_util.{h,c} files for each object. For
33 example, there are dapl_pz_util.h and dapl_pz_util.c files. The
34 use of util files follows the convention used elsewhere in the DAPL
35 reference provider. These files contain common object creation and
36 destruction code.
37
38
39 1. Protection Zones (PZs)
40 -------------------------
41
42 DAPL protection zones provide consumers with a means to associate
43 various DAPL objects with one another. The association can then be
44 validated before allowing these objects to be manipulated. The DAT
45 functions related to PZs are:
46
47 dat_pz_create
48 dat_pz_free
49 dat_pz_query
50
51 These are implemented in the DAPL reference provider by
52
53 dapl_pz_create
54 dapl_pz_free
55 dapl_pz_query
56
57 The reference implementation maps the DAPL PZ concept onto Infiniband
58 protections domains (PDs).
59
60 The DAT_PZ_HANDLE value returned to DAT consumers is a pointer to a
61 DAPL_PZ data structure. The DAPL_PZ structure is used to represent all
62 PZ objects. Code that manipulates this structure should atomically
63 increment and decrement the ref_count member to track the number of
64 objects referencing the PZ.
65
66
67 2. Local Memory Regions (LMRs)
68 ------------------------------
69
70 DAPL local memory regions represent a memory area on the host
71 system that the consumer wishes to access via local DTO operations.
72 The DAT functions related to LMRs are:
73
74 dat_lmr_create
75 dat_lmr_free
76 dat_lmr_query
77
78 These are implemented in
79
80 dapl_lmr_create
81 dapl_lmr_free
82 dapl_lmr_query
83
84 In the reference implementation, DAPL LMRs are mapped onto
85 Infiniband memory regions (MRs).
86
87 LMR creation produces two values: a DAT_LMR_CONTEXT and a
88 DAT_LRM_HANDLE.
89
90 The DAT_LMR_CONTEXT value is used to uniquely identify the LMR
91 when posting data transfer operations. These values map directly
92 to Infiniband L_KEYs.
93
94 Since some DAT functions need to translate a DAT_LMR_CONTEXT value
95 into a DAT_LMR_HANDLE (ex. dat_rmr_bind), a dictionary data structure
96 is used to associate DAT_LMR_CONTEXT values with their corresponding
97 DAT_LMR_HANDLE. Each time a new LMR is created, the DAT_LMR_HANDLE
98 should be inserted into the dictionary with the associated
99 DAT_LMR_CONTEXT as the key.
100
101 A hash table was chosen to implement this data structure. Since the
102 L_KEY values are being used by the CA hardware for indexing purposes,
103 there distribution is expected to be uniform and hence ideal for hashing.
104
105 The DAT_LMR_HANDLE value returned to DAT consumers is a pointer to
106 a DAPL_LMR data structure. The DAPL_LMR structure is used to represent
107 all LMR objects. The ref_count member should be used to track objects
108 associated with a given LMR.
109
110 The DAT API exposes the DAT_LMR_CONTEXT to consumers to allow
111 for sharing of memory registrations between multiple address spaces.
112 The mechanism by which such a feature would be implemented does not
113 yet exist. Consumers may be able to take advantage of this
114 feature on future transports.
115
116
117 3. Remote Memory Regions (RMRs)
118 -------------------------------
119
120 DAPL remote memory regions represent a memory area on the host
121 system to which the consumer wishes to allow RMDA operations. The
122 related DAT functions are
123
124 dat_rmr_create
125 dat_rmr_free
126 dat_rmr_query
127 dat_rmr_bind
128
129 which are implemented in
130
131 dapl_rmr_create
132 dapl_rmr_free
133 dapl_rmr_query
134 dapl_rmr_bind
135
136 The reference provider maps RMR objects onto Infiniband memory
137 windows.
138
139 The DAT_RMR_HANDLE value returned to DAT consumers is a pointer to
140 a DAPL_RMR data structure. The DAPL_RMR structure is used to represent
141 all RMR objects.
142
143 The API for binding a LMR to a RMR has the following function
144 signature:
145
146 DAT_RETURN
147 dapl_rmr_bind (
148 IN DAT_RMR_HANDLE rmr_handle,
149 IN const DAT_LMR_TRIPLET *lmr_triplet,
150 IN DAT_MEM_PRIV_FLAGS mem_priv,
151 IN DAT_EP_HANDLE ep_handle,
152 IN DAT_RMR_COOKIE user_cookie,
153 IN DAT_COMPLETION_FLAGS completion_flags,
154 OUT DAT_RMR_CONTEXT *rmr_context )
155
156 where a DAT_LMR_TRIPLET is defined as:
157
158 typedef struct dat_lmr_triplet
159 {
160 DAT_LMR_CONTEXT lmr_context;
161 DAT_UINT32 pad;
162 DAT_VADDR virtual_address;
163 DAT_VLEN segment_length;
164 } DAT_LMR_TRIPLET;
165
166 In the case of IB, the DAT_LMR_CONTEXT value is a L_KEY.
167 As described in the IB spec, the Bind Memory Window verb
168 takes both a L_KEY and Memory Region Handle among other
169 parameters. Therefore a data structure must be used to
170 map a DAT_LMR_CONTEXT (L_KEY) value to a DAPL_LMR so
171 that the needed memory region handle can be retrieved.
172 The LMR hash table described above is used for this
173 purpose.