Debian Codegraph @

README

raw
1 README/Release Notes
2 OFED 3.18 DAPL Release 2.1.6
3 August 2015
4
5 User space libraries/utilities for Direct Access Transport (DAT) v2.0. DAT is
6 a transport-independent, platform-independent Application Programming
7 Interface that supports RDMA (remote direct memory access) devices.
8 Note: v1.2 is no longer supported and will not be included with OFED releases
9
10 MIC support is provided with the new MCM provider and MPXYD service, since dapl-2.1.0.
11 MCM requires the Intel(R) MPSS 3.x (YOCTO) release for Linux to be installed on your system.
12 MPSS 3.x for Linux can be downloaded from: http://software.intel.com/mic-developer
13
14 For latest documentation and packages: //www.openfabrics.org/downloads/dapl/
15
16 =================
17 1.0 Release Notes
18 =================
19
20 dapl-2.1.5 changes include improvements for large scale UD communication management:
21
22 - AH caching, reduced memory footprint (grows as needed)
23 - Port space increased to 24 bits
24 - Hash table for port space, CM object management
25 - Optimized CM wire protocol for fast index lookup
26
27 Tested on 1200n 28ppn cluster, AlltoAll Intel MPI, UD mode.
28 Both static and dynamic modes, over 500m UD QP connections.
29
30 dapl-2.1.6 changes include MIC support for full offload mode
31
32 - Add support for Truescale qib devices with no CCL Direct verbs support on MIC.
33 - Enhancement for inside the box transfers without IB adapter via ibscif.
34 - Add DAPL_NETWORK_NODES, DAPL_NETWORK_PPN environment variables.
35
36 ==========
37 2.0 BUILD:
38 ==========
39
40 # NON_DEBUG build/install example for x86_64, OFED targets
41 ./configure --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include"
42 make install
43
44 # DEBUG build/install example for x86_64, using OFED targets
45 ./configure --enable-debug --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include"
46 make install
47
48 # COUNTERS build/install example for x86_64, using OFED targets
49 ./configure --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include -DDAPL_COUNTERS"
50 make install
51
52 =========================================================
53 3.0 Provider descriptions and CM results (cma, scm, ucm):
54 =========================================================
55
56 1. CMA - uses OFA rdma_cm to setup QP's. IPoIB, ARP, and SA queries required.
57
58 Provider name: ofa-v2-ib0
59 PROs: OFA rdma_cm has the most testing across many applications.
60 Supports both iWARP and IB.
61
62 CONs: Serialization of conn processing with kernel based CM service
63 Requires IPoIB ARP for name resolution, storms
64 Requires SA for path record queries for IB fabrics.
65 Conn Request private data limited to 52 bytes.
66
67 Settings for larger clusters (512+ cores):
68
69 setenv DAPL_CM_ROUTE_TIMEOUT_MS 20000
70 setenv DAPL_CM_ARP_TIMEOUT_MS 10000
71
72 2. SCM - uses sockets to exchange QP information. IPoIB, ARP, and SA queries NOT required.
73
74 Provider name (connectx): ofa-v2-mlx4_0-1
75 PROs: Each rank has own instance of socket cm. More private data with requests.
76 Doesn't require path-record lookup.
77
78 CONs: Socket resources grow with scale-out, serialization of
79 connections with kernel based tcp sockets,
80 Competes for MPI socket resources/port space and other TCP applications.
81 Sockets remain in TIMEWAIT state for minutes after closure.
82 Requires ARP for name resolution.
83 Doesn't support iWARP devices.
84
85 Settings for larger clusters (512+ cores):
86
87 setenv DAPL_ACK_RETRY 7 /* IB RC Ack retry count */
88 setenv DAPL_ACK_TIMER 20 /* IB RC Ack retry timer */
89
90 3. UCM - use's IB UD QP to exchange QP info. Sockets, ARP, IPoIB, and SA queries NOT required.
91
92 Provider name (connectx): ofa-v2-mlx4_0-1u
93 PROs: Each rank has own instance of CM in user process
94 Resources fixed per rank regardless of scale-out size
95 No serialization of user or kernel resources establishing connections,
96 Simple 3-way msg handsake, CM messages fit in inline data for lowest message latency,
97 Supports alternate paths
98 No address resolution required.
99 No path resolution required.
100
101 CONs: New provider with limited testing, a little tougher to debug.
102 Doesn't support iWARP
103
104 Settings for larger clusters (512+ cores):
105
106 setenv DAPL_UCM_REP_TIME 2000 /* REQUEST timer, waiting for REPLY in millisecs */
107 setenv DAPL_UCM_RTU_TIME 2000 /* REPLY timer, waiting for RTU in millisecs */
108 setenv DAPL_UCM_CQ_SIZE 2000 /* CM completion queue */
109 setenv DAPL_UCM_QP_SIZE 2000 /* CM message queue */
110 setenv DAPL_UCM_RETRY 7 /* REQUEST and REPLY retries */
111 setenv DAPL_ACK_RETRY 7 /* IB RC Ack retry count */
112 setenv DAPL_ACK_TIMER 20 /* IB RC Ack retry timer */
113
114 CM Performance: CPS profile for cma, scm, and ucm v2 uDAPL providers:
115 -----------------------------------------------------------------------
116 Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz (IVT)
117 Mellanox MLX4 IB FDR, no switch.
118
119 dtestcm (server/client):
120
121 cma: Connections: 313.10 usec, CPS 3193.83 Total 0.31 secs, poll_cnt=6300, Num=1000
122 scm: Connections: 167.65 usec, CPS 5964.92 Total 0.17 secs, poll_cnt=2394, Num=1000
123 ucm: Connections: 71.85 usec, CPS 13918.06 Total 0.07 secs, poll_cnt=2360, Num=1000
124
125 dapl_cm_bw: MPI uDAPL/CM profiling application (all-to-all connections, all ranks)
126
127 CMA
128 2 Connect times (10): Total 0.0049 per 0.0005 CPS=2051.38
129 4 Connect times (40): Total 0.0151 per 0.0004 CPS=2650.16
130 8 Connect times (240): Total 0.0548 per 0.0002 CPS=4380.59
131 16 Connect times (1120): Total 4.0356 per 0.0036 CPS=277.53
132 32 Connect times (4800): Total 4.4704 per 0.0009 CPS=1073.72
133
134 SCM
135 2 Connect times (10): Total 0.0029 per 0.0003 CPS=3441.31
136 4 Connect times (40): Total 0.0060 per 0.0002 CPS=6635.97
137 8 Connect times (240): Total 0.0194 per 0.0001 CPS=12383.47
138 16 Connect times (1120): Total 0.0649 per 0.0001 CPS=17246.93
139 32 Connect times (4800): Total 1.0193 per 0.0002 CPS=4708.95
140
141 UCM
142 2 Connect times (10): Total 0.0014 per 0.0001 CPS=6993.91
143 4 Connect times (40): Total 0.0045 per 0.0001 CPS=8837.87
144 8 Connect times (240): Total 0.0155 per 0.0001 CPS=15477.13
145 16 Connect times (1120): Total 0.0630 per 0.0001 CPS=17765.12
146 32 Connect times (4800): Total 0.2632 per 0.0001 CPS=18236.54
147
148 ===================================================================================================
149 4.0 BKM for installing new DAPL library on your cluster without any impact on existing OFED install:
150 ====================================================================================================
151
152 Note: example for user /home/user1, (assumes /home/user1 is exported) and MLX4 adapter, port 1
153
154 Download latest 2.1.x package: http://www.openfabrics.org/downloads/dapl/dapl-2.1.6.tar.gz
155
156 untar in /home/user1
157 cd /home/user1/dapl-2.1.6
158 ./configure LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include"
159 make
160
161 Create /home/user1/dat.conf with following 3 lines. (entries with path to new libraries):
162
163 ofa-v2-mlx4_0-1u u2.0 nonthreadsafe default /home/user1/dapl-2.1.5/dapl/udapl/.libs/libdaploucm.so.2 dapl.2.0 "mlx4_0 1" ""
164 ofa-v2-mlx4_0-1m u2.0 nonthreadsafe default /home/user1/dapl-2.1.5/dapl/udapl/.libs/libdaplomcm.so.2 dapl.2.0 "mlx4_0 1" ""
165 ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default /home/user1/dapl-2.1.5/dapl/udapl/.libs/libdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
166 ofa-v2-ib0 u2.0 nonthreadsafe default /home/user1/dapl-2.1.5/dapl/udapl/.libs/libdaplcma.so.1 dapl.2.0 "ib0 0" ""
167
168 Run uDAPL application or Intel MPI that uses uDAPL, with (assuming mlx4_0 adapters) following:
169
170 setenv DAT_OVERRIDE=/home/user1/dat.conf
171 setenv LD_LIBRARY_PATH=/home/user1/dapl-2.1.5/dapl/udapl/.libs:$LD_LIBRARY_PATH
172
173 If running Intel MPI and uDAPL IB UD cm, set the following (recommended):
174
175 setenv I_MPI_DAPL_PROVIDER=ofa-v2-mlx4_0-1u
176
177 If running Intel MPI and uDAPL IB mcm with MIC, set the following:
178
179 setenv I_MPI_DAPL_PROVIDER=ofa-v2-mlx4_0-1m
180
181 If running Intel MPI and uDAPL socket cm, set the following:
182
183 setenv I_MPI_DAPL_PROVIDER=ofa-v2-mlx4_0-1
184
185
186 If running Intel MPI and uDAPL rdma_cm, set the following:
187
188 setenv I_MPI_DAPL_PROVIDER=ofa-v2-ib0
189
190
191 ============================================================
192 5.0 MCM Provider, MPXYD Daemon (CCL-proxy) Build and Install
193 ============================================================
194
195 MCM is a new uDAPL provider that is an extension to standard DAT 2.0 libraries. The purpose of this service
196 is to proxy RDMA writes from the MIC to the HOST to improve large IO performance. The provider will support
197 MIC to MIC, HOST to HOST, and MIC to HOST environments. The mcm client will NOT use MPXYD when running on the host.
198 It requires a new MPXYD daemon service when clients are running on a MIC KNC adapter. This package installs all the
199 host side libraries and daemon service. The MIC libraries must be built and moved over to MIC adapter. This verion
200 is currently included with MPSS and all libraries and services will be installed by default.
201
202 Current release package: dapl-2.1.6.tar.gz
203
204 * Sample host build from source package (ofed must be installed)
205
206 ./configure --enable-mcm --prefix=/usr --libdir=/usr/lib64 --sysconfdir=/etc
207 make
208 make install
209
210 * Sample host rpmbuild/update from release tarball, /root:
211
212 rpmbuild -ta dapl-2.1.6.tar.gz
213 rpm -U /root/rpmbuild/RPMS/x86_64/dapl*
214
215 * Sample MIC build from source package for MPSS 3.x (MPSS must be installed)
216 * Assume /opt is nfs mounted across cluster
217
218 source /opt/mpss/3.x/environment-setup-k1om-mpss-linux
219 ./configure --enable-mcm --prefix /opt/dapl/mic --host=x86_64-k1om-linux
220 make
221 make install
222
223 copy /opt/dapl/mic/* files out to all MIC cards
224
225 * Cluster deployment
226
227 (1) Build once on the head or on one of the nodes (with MPSS) as described in the above steps.
228
229 (2) HOST: Install dapl libraries and mpxyd service, "rpm -U" all dapl RPM files on host nodes:
230
231 (3) MIC: Setup dapl overlay for new package (/opt/intel/dapl):
232
233 Create /etc/mpss/conf.d/dapl.conf with following entry:
234
235 Overlay Filelist /opt/dapl /opt/dapl/dapl.filelist on
236
237 Create /opt/dapl/dapl.filelist with following entries:
238
239 file /etc/dat.conf mic/etc/dat.conf 755 0 0
240 file /usr/bin/dtest mic/bin/dtest 755 0 0
241 file /usr/bin/dtestx mic/bin/dtestx 755 0 0
242 file /usr/bin/dtestcm mic/bin/dtestcm 755 0 0
243 file /usr/bin/dapltest mic/bin/dapltest 755 0 0
244 file /usr/lib64/libdat.so.2.0.0 mic/lib/libdat.so.2.0.0 755 0 0
245 file /usr/lib64/libdaplofa.so.2.0.0 mic/lib/libdaplofa.so.2.0.0 755 0 0
246 file /usr/lib64/libdaplomcm.so.2.0.0 mic/lib/libdaplomcm.so.2.0.0 755 0 0
247 file /usr/lib64/libdaploscm.so.2.0.0 mic/lib/libdaploscm.so.2.0.0 755 0 0
248 file /usr/lib64/libdaploucm.so.2.0.0 mic/lib/libdaploucm.so.2.0.0 755 0 0
249
250 slink /usr/lib64/libdat.so libdat.so.2.0.0 777 0 0
251 slink /usr/lib64/libdat.so.2 libdat.so.2.0.0 777 0 0
252 slink /usr/lib64/libdaplofa.so libdaplofa.so.2.0.0 777 0 0
253 slink /usr/lib64/libdaplofa.so.2 libdaplofa.so.2.0.0 777 0 0
254 slink /usr/lib64/libdaplomcm.so libdaplomcm.so.2.0.0 777 0 0
255 slink /usr/lib64/libdaplomcm.so.2 libdaplomcm.so.2.0.0 777 0 0
256 slink /usr/lib64/libdaploscm.so libdaploscm.so.2.0.0 777 0 0
257 slink /usr/lib64/libdaploscm.so.2 libdaploscm.so.2.0.0 777 0 0
258 slink /usr/lib64/libdaploucm.so libdaploucm.so.2.0.0 777 0 0
259 slink /usr/lib64/libdaploucm.so.2 libdaploucm.so.2.0.0 777 0 0
260
261 Reboot or restart MPSS and ofed-mic services
262
263 Check for dapl overlay
264 micctrl --config
265
266 * Setup for non-root CCL Proxy testing, MPXYD running as process with different service port from your /home directory:
267
268 Using build instructions above, change prefix as follow and "make install":
269
270 Build MIC:
271 --prefix=/home/username/ccl-proxy-mic
272
273 Build host:
274 --prefix=/home/username/ccl-proxy-host
275
276 edit /home/username/ccl-proxy-host/etc/mpxyd.conf and change the following entries:
277
278 log_file /var/log/mpxyd.log to log_file /tmp/username/mpxyd.log
279 lock_file /var/log/mpxyd.pid to lock_file /tmp/username/mpxyd.log
280 scif_port_id 68 to scif_port_id 1068
281
282 start the mpxyd process on each node
283
284 ssh node1-hostname /home/username/ccl-proxy-host/sbin/mpxyd -P -O /home/username/ccl-proxy-host/etc/mpxyd.conf&
285
286 Note: override default port id using following environment variable:
287
288 export DAPL_MCM_PORT_ID=1068
289
290 * Notes
291
292 (1) Modify "/etc/mpxyd.conf" to change the settings for the proxy. Especially, try different values
293 of "buffer_segment_size" for performance tuning. Use a smaller value for "buffer_pool_mb"
294 to reduce the memory foorprint of mpxyd. Use a larger value for "scif_listen_qlen" to run
295 more MPI ranks per card. Also modify mcm_affinity_base to the desired CPU_id to insure
296 socket to adapter affinity. Best performance when HCA, MIC, and CPU are on same socket.
297 Default settings are on CPU socket 0.
298
299 (2) By default, only writes originated from MIC is proxied. However, it is also possible to proxy
300 host-originated writes (e.g. for debugging purpose). To do this, set the environment variable
301 "DAPL_MCM_ALWAYS_PROXY=1". This variable applies to the provider, not the proxy.
302
303 * Use the MCM provider with Intel MPI 5.1 or greater for best out of box experiences with MIC.
304
305 (1) Recommended settings:
306
307 export I_MPI_MIC=1
308 export I_MPI_DEBUG=2
309 export I_MPI_FALLBACK=0
310
311 =============================
312 6.0 Environment Variables
313 =============================
314
315 - IB UD options using UCM provider, large scale settings (Xeon)
316
317 export DAPL_NETWORK_NODES= /* set to active nodes on network for CM */
318 export DAPL_NETWORK_PPN= /* set to active processes per node for CM */
319
320 /* The following will be adjusted by provider based on NODES, PPN */
321 export DAPL_UCM_REP_TIME=8000 /* REQUEST timer, waiting on REPLY, msecs, default = 800 */
322 export DAPL_UCM_RTU_TIME=8000 /* REPLY timer, waiting for RTU in msecs, default=400 */
323 export DAPL_UCM_RETRY=7 /* REQUEST & REPLY retries, default = 7 */
324 export DAPL_UCM_QP_SIZE=4000 /* CM req/reply work queue size, default = 500 entries */
325 export DAPL_UCM_CQ_SIZE=4000 /* CM req/reply completion queue size, default = 500 entries */
326 export DAPL_UCM_TX_BURST=100 /* CM signal rate on send messages */
327 export DAPL_UCM_ENTRY_BITS=11 /* default = 11-bit, 2KB entries, allocation blocks */;
328 export DAPL_UCM_ARRAY_BITS=18 /* default = 18 bit, 256KB total */
329
330 - IB RC options using SCM provider
331
332 export DAPL_SCM_NETDEV=ib0 /* default is first non-loopback netdev */
333
334 - Other IB settings for all providers:
335
336 export DAPL_MAX_INLINE=64 /* IB RC inline optimization, best small msg latency, def=64 */
337 export DAPL_ACK_RETRY=7 /* IB RC Ack retry count, default 7 */
338 export DAPL_ACK_TIMER=20 /* IB RC Ack retry timer, 5 bits, 4.096us*2^ack_timer. 16== 268ms, 20==4.2s */
339 export DAPL_IB_MTU=2048 /* IB MTU size, default = 2048 */
340 export DAPL_RNR_TIMER=12 /* 5 bits, 12 =.64ms, 28 =163ms, 31 =491ms */
341 export DAPL_RNR_RETRY=7 /* 3 bits, 7 == infinite */
342 export DAPL_IB_PKEY= 0 /* override IB partition key, default is pkey index 0 */
343 export DAPL_IB_SL=0 /* override IB Sevice level, default = 0 */
344
345 - Other options:
346 export DAPL_WR_MAX=500 /* used to reduce max qp depth on all IB providers, default = dev attributes */
347
348 Debug logging and Counter settings ( --enable-counters)
349
350 export DAPL_DBG_SYS_MEM=10 /* threshold for low sys memory warning, def = 10 percent */
351 export DAPL_DBG_TYPE=0x0000003 /* set log, monitor, and error checking, default = warnings and errors */
352
353 DAPL_DBG_TYPE bit settings as follow:
354
355 DAPL_DBG_TYPE_ERR = 0x0001,
356 DAPL_DBG_TYPE_WARN = 0x0002,
357 DAPL_DBG_TYPE_EVD = 0x0004,
358 DAPL_DBG_TYPE_CM = 0x0008,
359 DAPL_DBG_TYPE_EP = 0x0010,
360 DAPL_DBG_TYPE_UTIL = 0x0020,
361 DAPL_DBG_TYPE_CALLBACK = 0x0040,
362 DAPL_DBG_TYPE_DTO_COMP_ERR = 0x0080,
363 DAPL_DBG_TYPE_API = 0x0100,
364 DAPL_DBG_TYPE_RTN = 0x0200,
365 DAPL_DBG_TYPE_EXCEPTION = 0x0400,
366 DAPL_DBG_TYPE_SRQ = 0x0800,
367 DAPL_DBG_TYPE_CNTR = 0x1000,
368 DAPL_DBG_TYPE_CM_LIST = 0x2000,
369 DAPL_DBG_TYPE_THREAD = 0x4000,
370 DAPL_DBG_TYPE_CM_EST = 0x8000,
371 DAPL_DBG_TYPE_CM_WARN = 0x10000,
372 DAPL_DBG_TYPE_EXTENSION = 0x20000,
373 DAPL_DBG_TYPE_CM_STATS = 0x40000,
374 DAPL_DBG_TYPE_CM_ERRS = 0x80000, /* print any cm errors on device close */
375 DAPL_DBG_TYPE_LINK_ERRS = 0x100000, /* print any link errors on device close */
376 DAPL_DBG_TYPE_LINK_WARN = 0x200000, /* print any link warning on device close */
377 DAPL_DBG_TYPE_DIAG_ERRS = 0x400000, /* print any diag_counter errors on dev close */
378 DAPL_DBG_TYPE_SYS_WARN = 0x800000, /* print low mem warning during alloc, reg_mem */
379 DAPL_DBG_TYPE_VER = 0x1000000, /* print dapl ver and build date during dev open */
380
381 =============================
382 7.0 SAMPLE uDAPL APPLICATION:
383 =============================
384
385 There are 2 sample programs, with manpages, provided with this package.
386
387 (dapl/test/dtest/)
388
389 NAME
390 dtest - simple uDAPL send/receive and RDMA test
391
392 SYNOPSIS
393 dtest [-P provider] [-b buf size] [-B burst count][-v] [-c] [-p] [-d] [-s]
394
395 dtest [-P provider] [-b buf size] [-B burst count][-v] [-c] [-p] [-d] [-h HOSTNAME]
396
397 DESCRIPTION
398 dtest is a simple test used to exercise and verify the uDAPL interfaces. At least two instantia-
399 tions of the test must be run. One acts as the server and the other the client. The server side of
400 the test, once invoked listens for connection requests, until timing out or killed. Upon receipt
401 of a cd connection request, the connection is established, the server and client sides exchange
402 information necessary to perform RDMA writes and reads.
403
404 OPTIONS
405 -P=PROVIDER
406 use PROVIDER to specify uDAPL interface using /etc/dat.conf (default OpenIB-cma)
407
408 -b=BUFFER_SIZE
409 use buffer size BUFFER_SIZE for RDMA(default 64)
410
411 -B=BURST_COUNT
412 use busrt count BURST_COUNT for interations (default 10)
413
414 -v, verbose output(default off)
415
416 -c, use consumer notification events (default off)
417
418 -p, use polling (default wait for event)
419
420 -d, delay in seconds before close (default off)
421
422 -s, run as server (default - run as server)
423
424 -h=HOSTNAME
425 use HOSTNAME to specify server hostname or IP address (default - none)
426
427 EXAMPLES
428 dtest -P OpenIB-cma -v -s
429 Starts a server process with debug verbosity using provider OpenIB-cma.
430
431 dtest -P OpenIB-cma -h server1-ib0
432
433 Starts a client process, using OpenIB-cma provider to connect to hostname server1-ib0.
434
435 SEE ALSO
436 dapltest(1)
437
438 AUTHORS
439 Arlin Davis
440 <[email protected]>
441
442 BUGS
443
444 /dapl/test/dapltest/
445
446 NAME
447 dapltest - test for the Direct Access Programming Library (DAPL)
448
449 DESCRIPTION
450 Dapltest is a set of tests developed to exercise, characterize, and verify the DAPL interfaces
451 during development and porting. At least two instantiations of the test must be run. One acts as
452 the server, fielding requests and spawning server-side test threads as needed. Other client invo-
453 cations connect to the server and issue test requests. The server side of the test, once invoked,
454 listens continuously for client connection requests, until quit or killed. Upon receipt of a con-
455 nection request, the connection is established, the server and client sides swap version numbers
456 to verify that they are able to communicate, and the client sends the test request to the server.
457 If the version numbers match, and the test request is well-formed, the server spawns the threads
458 needed to run the test before awaiting further connections.
459
460 USAGE
461 dapltest [ -f script_file_name ] [ -T S|Q|T|P|L ] [ -D device_name ] [ -d ] [ -R HT|LL|EC|PM|BE ]
462
463 With no arguments, dapltest runs as a server using default values, and loops accepting requests
464 from clients.
465
466 The -f option allows all arguments to be placed in a file, to ease test automation.
467
468 The following arguments are common to all tests:
469
470 [ -T S|Q|T|P|L ]
471 Test function to be performed:
472
473 S - server loop
474
475 Q - quit, client requests that server wait for any outstanding tests to complete, then
476 clean up and exit
477
478 T - transaction test, transfers data between client and server
479
480 P - performance test, times DTO operations
481
482 L - limit test, exhausts various resources, runs in client w/o server interaction
483 Default: S
484
485 [ -D device_name ]
486 Specifies the interface adapter name as documented in the /etc/dat.conf static configura-
487 tion file. This name corresponds to the provider library to open. Default: none
488
489 [ -d ] Enables extra debug verbosity, primarily tracing of the various DAPL operations as they
490 progress. Repeating this parameter increases debug spew. Errors encountered result in the
491 test spewing some explanatory text and stopping; this flag provides more detail about what
492 lead up to the error. Default: zero
493
494 [ -R BE ]
495 Indicate the quality of service (QoS) desired. Choices are:
496
497 HT - high throughput
498
499 LL - low latency
500
501 EC - economy (neither HT nor LL)
502
503 PM - premium
504
505 BE - best effort Default: BE
506
507 Usage - Quit test client
508
509 dapltest [Common_Args] [ -s server_name ]
510
511 Quit testing (-T Q) connects to the server to ask it to clean up and
512 exit (after it waits for any outstanding test runs to complete).
513 In addition to being more polite than simply killing the server,
514 this test exercises the DAPL object teardown code paths.
515 There is only one argument other than those supported by all tests:
516
517 -s server_name Specifies the name of the server interface.
518 No default.
519
520 Usage - Transaction test client
521
522 dapltest [Common_Args] [ -s server_name ]
523 [ -t threads ] [ -w endpoints ] [ -i iterations ] [ -Q ]
524 [ -V ] [ -P ] OPclient OPserver [ op3,
525
526 Transaction testing (-T T) transfers a variable amount of data between
527 client and server. The data transfer can be described as a sequence of
528 individual operations; that entire sequence is transferred ’iterations’
529 times by each thread over all of its endpoint(s).
530
531 The following parameters determine the behavior of the transaction test:
532
533 -s server_name Specifies the name or IP address of the server interface.
534 No default.
535
536 [ -t threads ] Specify the number of threads to be used.
537 Default: 1
538
539 [ -w endpoints ] Specify the number of connected endpoints per thread.
540 Default: 1
541
542 [ -i iterations ] Specify the number of times the entire sequence
543 of data transfers will be made over each endpoint.
544 Default: 1000
545
546 [ -Q ] Funnel completion events into a CNO.
547 Default: use EVDs
548
549 [ -V ] Validate the data being transferred.
550 Default: ignore the data
551
552 [ -P ] Turn on DTO completion polling
553 Default: off
554
555 OP1 OP2 [ OP3, ... ]
556 A single transaction (OPx) consists of:
557
558 server|client Indicates who initiates the
559 data transfer.
560
561 SR|RR|RW Indicates the type of transfer:
562 SR send/recv
563 RR RDMA read
564 RW RDMA write
565 Defaults: none
566
567 [ seg_size [ num_segs ] ]
568 :
569
570 Indicates the amount and format
571 of the data to be transferred.
572 Default: 4096 1
573 (i.e., 1 4KB buffer)
574
575 [ -f ] For SR transfers only, indicates
576 that a client’s send transfer
577 completion should be reaped when
578 the next recv completion is reaped.
579 Sends and receives must be paired
580 (one client, one server, and in that
581 order) for this option to be used.
582 Restrictions:
583
584 Due to the flow control algorithm used by the transaction test, there
585 must be at least one SR OP for both the client and the server.
586
587 Requesting data validation (-V) causes the test to automatically append
588 three OPs to those specified. These additional operations provide
589 synchronization points during each iteration, at which all user-specified
590 transaction buffers are checked. These three appended operations satisfy
591 the "one SR in each direction" requirement.
592
593 The transaction OP list is printed out if -d is supplied.
594
595 Usage - Performance test client
596
597 dapltest [Common_Args] -s server_name [ -m p|b ]
598 [ -i iterations ] [ -p pipeline ] OP
599
600 Performance testing (-T P) times the transfer of an operation.
601 The operation is posted ’iterations’ times.
602
603 The following parameters determine the behavior of the transaction test:
604
605 -s server_name Specifies the name or IP address of the server interface.
606 No default.
607
608 -m b|p Used to choose either blocking (b) or polling (p)
609 Default: blocking (b)
610 [ -i iterations ] Specify the number of times the entire sequence
611 of data transfers will be made over each endpoint.
612 Default: 1000
613
614 [ -p pipeline ] Specify the pipline length, valid arguments are in
615 the range [0,MAX_SEND_DTOS]. If a value greater than
616 MAX_SEND_DTOS is requested the value will be
617 adjusted down to MAX_SEND_DTOS.
618 Default: MAX_SEND_DTOS
619
620 OP Specifies the operation as follow:
621
622 RR|RW Indicates the type of transfer:
623 RR RDMA read
624 RW RDMA write
625 Defaults: none
626
627 [ seg_size [ num_segs ] ]
628 Indicates the amount and format
629 of the data to be transferred.
630 Default: 4096 1
631 (i.e., 1 4KB buffer)
632 Usage - Limit test client
633
634 Limit testing (-T L) neither requires nor connects to any server
635 instance. The client runs one or more tests which attempt to
636 exhaust various resources to determine DAPL limits and exercise
637 DAPL error paths. If no arguments are given, all tests are run.
638
639 Limit testing creates the sequence of DAT objects needed to
640 move data back and forth, attempting to find the limits supported
641 for the DAPL object requested. For example, if the LMR creation
642 limit is being examined, the test will create a set of
643 {IA, PZ, CNO, EVD, EP} before trying to run dat_lmr_create() to
644 failure using that set of DAPL objects. The ’width’ parameter
645 can be used to control how many of these parallel DAPL object
646 sets are created before beating upon the requested constructor.
647 Use of -m limits the number of dat_*_create() calls that will
648 be attempted, which can be helpful if the DAPL in use supports
649 essentailly unlimited numbers of some objects.
650 The limit test arguments are:
651
652 [ -m maximum ] Specify the maximum number of dapl_*_create()
653 attempts.
654 Default: run to object creation failure
655
656 [ -w width ] Specify the number of DAPL object sets to
657 create while initializing.
658 Default: 1
659
660 [ limit_ia ] Attempt to exhaust dat_ia_open()
661
662 [ limit_pz ] Attempt to exhaust dat_pz_create()
663
664 [ limit_cno ] Attempt to exhaust dat_cno_create()
665
666 [ limit_evd ] Attempt to exhaust dat_evd_create()
667
668 [ limit_ep ] Attempt to exhaust dat_ep_create()
669
670 [ limit_rsp ] Attempt to exhaust dat_rsp_create()
671
672 [ limit_psp ] Attempt to exhaust dat_psp_create()
673
674 [ limit_lmr ] Attempt to exhaust dat_lmr_create(4KB)
675
676 [ limit_rpost ] Attempt to exhaust dat_ep_post_recv(4KB)
677
678 [ limit_size_lmr ] Probe maximum size dat_lmr_create()
679
680 Default: run all tests
681 EXAMPLES
682 dapltest -T S -d -D OpenIB-cma
683
684 Starts a server process with debug verbosity.
685
686 dapltest -T T -d -s host1-ib0 -D OpenIB-cma -i 100 client SR 4096 2 server SR 4096 2
687
688 Runs a transaction test, with both sides
689 sending one buffer with two 4KB segments,
690 one hundred times.
691
692 dapltest -T P -d -s host1-ib0 -D OpenIB-cma -i 100 SR 4096 2
693
694 Runs a performance test, with the client
695 sending one buffer with two 4KB segments,
696 one hundred times.
697
698 dapltest -T Q -s host1-ib0 -D OpenIB-cma
699
700 Asks the server to clean up and exit.
701
702 dapltest -T L -D OpenIB-cma -d -w 16 -m 1000
703
704 Runs all of the limit tests, setting up
705 16 complete sets of DAPL objects, and
706 creating at most a thousand instances
707 when trying to exhaust resources.
708
709 dapltest -T T -V -d -t 2 -w 4 -i 55555 -s linux3 -D OpenIB-cma client RW 4096 1 server RW 2048 4
710 client SR 1024 4 server SR 4096 2 client SR 1024 3 -f server SR 2048 1 -f
711
712 Runs a more complicated transaction test,
713 with two thread using four EPs each,
714 sending a more complicated buffer pattern
715 for a larger number of iterations,
716 validating the data received.
717
718 =============================
719 8.0 Summary of Fixes/Changes:
720 =============================
721
722 Release 2.1.6 (OFED 3.18-1)
723 ucm: add cluster size environments to adjust CM timers
724 mpxyd: proxy_in data transfers can improperly start before RTU received
725 mcm: forward open/query for MFO devices in query only mode
726 mpxyd: byte swap incorrect on WRC wr_len
727 dtest: remove ERR message from flush QP function
728 dapltest: Quit command with "-n port" number will core dump
729 config: update dat.conf for MFO qib devices, 2 adapters/ports
730 mpxyd: add MFO support on proxy side
731 mcm: add MFO proxy commands, device, and CM support
732 mcm: add MFO support to openib_common code base
733 mcm: add full offload (MFO) mode to provider to support qib on MIC
734 dtest: pre-allocated buffer too small for RMR, DTO ops timeout
735 mpxyd: fix buffer initialization when no-inline support is active
736 mpxyd: reduce log level on qp_flush to CM level
737 mcm: intra-node proxy missing LID setup on rejects
738 mcm: add intra-node support via ibscif device and mcm provider
739 mcm: provide MIC address info with proxy device open
740 mcm: add device info to non-debug log
741 common: add DAPL_DTO_TYPE_EXTENSION_IMM for rdma_write_imm DTO type checking
742 mpxyd: fix up some of the PI logging
743 dtest: modify rdma_write_with_msg to support uni-direction streaming
744 mcm,mpxyd: fix dreq processing to defer QP flush when proxy WRs still pending
745 mpxyd: update byte_len and comp_cnt for PO to remote HST communications
746 mcm: bug fixes for non-inline devices
747 mcm: return CM_rej with CM_req_in errors
748 mpxyd,mcm: RDMA write with immed data not signaled on request side
749 mcm: add WC opcode and wc_flags in debug log message
750 mpxyd: set options bug fix for mcm_ib_inline
751 Update release notes with latest CM times
752
753 Release 2.1.5 (OFED 3.18 RC3)
754 update release notes, readme
755 dat.conf: update comments regarding versions
756 dtest: add logging of provider private data size with -v
757 scm: remove use of msg.resv field for process id logging
758 cma: report correct CM req private data size on query
759 mpxyd: memset ib_wr structure before post_send on WC and WR requests
760 mcm: add HST side provider support for device without inline data capability
761 ucm: CM changes for UD extended port space and indexer
762 ucm: add device support for new port space hash table
763 ucm: allocate/free AH hash table for UD endpoint types
764 ucm: check for AH caching when destroying via UD extension
765 ucm: optimizations for large scale UD communication management
766 mpxyd: use wr opcode instead of wc opcode to support logging on error cases
767 mcm: HST->MXS mode, using RDMA_WRITE_WITH_IMM, fails with dtest -w
768 dapl: aarch64 support for linux
769 dapltest: add scripts to dist, set default device to IPoIB
770 mpxyd: add wc_flags to proxy work completions
771
772 Release 2.1.4 (OFED 3.18 RC1)
773 mpxyd: fix typo in configuration file
774 cma: RR attributes moved to common ib_cm struct
775 mpxyd: tx thread incorrectly sleeps with negative pi_rw_cnt value
776 dat.conf: add entries for True Scale qib device
777 mpxyd: add support for devices without inline data support
778 ucm: long disconnect times with many-to-one applications
779 openib: add inline data support check during device open
780 cleanup ib/cm attribute management across openib providers
781 dapltest: fix -Werror=format-security issue with printf
782 Release 2.1.3 (targeting OFED 3.18)
783 dapl: mpxyd service changes to support multi-thread single-core option
784 dapl: add rdma_write_imm and write only option to dtest
785 ucm: add time wait override capability for CM services
786 common: dapl_ep_free must serialize CM object destroy
787 dtestx: allow scale up to 1000 EP's
788 ucm: RTU not retransmitted in TIMEWAIT state
789 mpxyd: increase max open files for service
790 mpxyd: DTO completion ERR: status 12, op RDMA_WRITE running MPI alltoall test
791 mcm: HST->MXS mode incorrectly signals multiple fragments per WR
792 mcm: add segmentation to HST->MXS mode for improved performance
793 mpxyd: set global seg_sz to 128KB for proxy data service
794 openib: add port_num to provider named attributes
795 mcm: provide CPU family/model attribute on both host and mic sides
796 dtestx: update IB extension example test with new v2.0.9 features
797 dtest: add dtestsrq for SRQ example and provider testing
798 common: add srq support for openib verbs providers
799 openib: add IB UD cm_free/ah_free extension support in UCM provider
800 openib: add new TIMEWAIT state for CM
801 extension: add IB UD extensions to reduce provider CM and AH memory footprint
802 mpxyd/mcm: add provider specific attribute DAT_IB_PROXY_VERSION
803 mpxyd: log warning if running in COMPAT mode
804 add provider and proxy support for GUID across platform
805 common: return appropriate handles with affiliated EP and EVD async events
806
807 Release 2.1.2 (OFED 3.12-1)
808 mpxyd: add global routing support for proxy connections
809 mcm: only call mix_get_attr if running on MIC
810 openib: modify check for link_layer to handle unspecified
811 dapl: add support for the s390x platform
812 dtest server exchange connection info with client
813 mpxyd: 2 MICs in same numa_node will overlap CPU affinity, don't reset base
814 mcm: implement proxy mix_prov_attr function, add fields CPU model and family
815 mpxyd: tx thread may not be signaled on small segment writes
816
817 Release 2.1.1 (OFED 3.12-1 RC1)
818 common: add provider name to log messages
819 mpxyd: log warning message if numa_node invalid include debuginfo with build
820 build: include debuginfo with build
821 mpxyd: tx thread doesn't sleep during no pending IO state
822 mpxyd: change MIC cpu_mask to per numa node instead of adapter
823 mpxyd: set to MXS mode if device numa_node is invalid (-1)
824 mpxyd: MXS based alltoall benchmark hangs or returns post_send timeout
825 mpxyd: add IO profile capabilities to help debug alltoall stall cases
826 mpxyd: retry stalled inline post_send, init m_idx only when signaled
827
828 Release 2.1.0 (OFED 3.12-1, MIC support added)
829 build: add missing NEWS file
830 update autogen.sh
831 add MCM provider and MPXYD service to build
832 mpxyd: service startup script and configuration file
833 add readme for MCM provider and MPXYD service
834 update Copyright dates
835 add new MIC RDMA proxy service daemon (MPXYD)
836 add new dapl MIC provider (MCM) to support MIC RDMA proxy services
837 MCM: new MIC provider and proxy service definitions
838 cleanup build warnings
839 common: add CQ,QP,MR abstractions for new MIC provider and data proxy service
840 openib: cleanup, use inet_ntop for GIDs, remove some logs, destroy pipes on release
841 common: new dapls_evd_cqe_to_event call, cqe to event
842 common: init ring_buffer, assign hd/tl pos in range
843 allow log level changes during device open
844 ucm: fix cm rbuf setup, include grh pad on initialization
845 ucm: remove duplicate async_event code, use common async event call
846 new lightweight open_query/close_query IB extension for fast attribute query
847 dtestcm: add more detailed debug during disconnect phase
848 cma: long delays when opening cma provider with no IPoIB configured
849 common: new debug levels for low system memory, IA stats, and package info
850 build: remove library check for mverbs with --enable-fca
851 IB extension: segfault in create collective group with non-vector type IA handle"
852 build: change configure help to correctly state collective default=none
853
854 Release 2.0.42 fixes (OFED 3.12 GA)
855 dapltest: increase DTO evd size to prevent CQ overflow on limit_rpost test
856 dapltest: RSP limit test fails. Creation of reserved SP moves EP state to DAT_EP_STATE_RESERVED in error cases.
857 dapl: fix string bug in dapls_dto_op_str
858
859 Release 2.0.41 fixes (OFED 3.12 RC1)
860 dapltest: change server port, from 45278 to 62000, out of registered IANA range
861 dat: lower log level on load errors of provider library
862 dat: dat_ia_open should close provider after failure
863 dapltest: set default limit max to 1000
864 openib: add new provider specific attributes
865 dapltest: update scripts for regression testing purposes
866 dapltest: Add final send/recv "sync" for transaction tests.
867
868 Release 2.0.40 fixes (OFED 3.12)
869 dist: ib collective extension include files missing
870 dapltest: the quit command is missing changes for -n option
871 dat.conf: remove v1, add Mellanox Connect-IB and Intel Xeon Phi MIC
872 NULL undefined on Fedora, incorrectly using kernel stddef.h
873
874 Release 2.0.39 fixes (OFED 3.5-2 GA)
875 dapltest: fix endian swap issue with performance test
876 scm: getifaddrs modfications for better out of the box experience
877 ucm, scm: UD mode triggers list_head assert with large scale alltoall test
878
879 Release 2.0.38
880 dapltest: add -n parameter to override default server port number (45278)
881 ucm,scm: UD mode creates many CR objects per EP that needs cleaned up
882 cma: add DAPL_CM_TOS environment variable to enable passing a TOS to the RDMA CM
883
884 Release 2.0.37
885 common: add support for ia name during dat_ia_query
886 common: dapl_os_atomic_inc/dec() not working as expected on ppc64 machines.
887 dapltest: ppc64 endian issue with exchanged mem handle and address
888
889 Release 2.0.36
890 scm: increase ACK timeout to 20 for a default value to match other providers.
891 common: allow qp modify in init state
892 common: check for valid states during ep posting
893 dat.conf: keep list of providers in order for backward compatibility
894 ucm: record and silently drop a duplicate reject CM message
895 windows: new version of getlocalipaddr not portable
896 dapltest: DFLT_QLEN is defined in multiple tests
897
898 Release 2.0.35
899 config/build: remove post/postun hacking used to modify dat.conf
900 config: clean up help option displays with ext-type options
901 windows: Provide auto-detect between RoCE and Infiniband for Windows.
902 ucm: update UD cm provider to support new CM stat and error counters
903 scm: update socket cm provider to support new CM stat and error counters
904 commom: add cm, link, and diag event counters in IB extended builds
905 scm: use ioctl SIOCIFCONF to get complete list of configured netdev interfaces
906 ucm: UD send failures at scale, ucm_send ERR: get_smsg(hd=149,tl=150)
907 scm: fix retry count on connection pending timeout
908 ucm: cleanup debug message, ntohl on p_size is incorrect
909 cma, scm, ucm: allow EP (QP) creation without EVD (CQ)
910 common: add DAPL_DBG_TYPE_CM_STATS (0x40000) to debug log options
911 common: dapls_ep_flush_cq will segfault when no CQ is attached to EP
912 common: ep_create should allow max_request_iov attribute setting of zero
913 common: add check for NULL handle on ext calls, SRQ free, and helper functions
914 common: add missing sub-types to dat_strerror()
915 common: extended CR event processing missing rejects on errors
916 ucm: incorrectly sends user reject during CR callback errors
917 common: change dbg level on CR callback if not listening on SP
918 scm: incorrectly sends user reject during CR callback errors
919 dat: add check for NULL handle on IA calls
920 cma,scm,ucm: extra reference on EP, with RSP, causes dat_ep_free() to hang
921 common: RSP service points incorrectly freed during CR callback
922 common: clean up dat_rsp_create log message
923 common: cleanup debug message on EVD overflows
924 scm: return correct event error code when remote host refuses requests
925 dapltest: server CR EVD is too small for multi-client configurations.
926 Common: CR EVD overflow causes segfault.
927
928 Release 2.0.34
929 scm: change debug message level for listen/bind errors
930 common: increase default IB ack timer from 16 to 20
931 common: remote ia address null pointer creates seg fault
932 common: posting events on full queue returns wrong error code
933 common: dat_ep_modify seg faults with null ep_param ptr
934 common: dat_evd_free seg faults with resized software EVD
935 common: remove assert for incorrect events during cm_request
936 dat: dat_cno_query with NULL cno_handle causes segmentation fault
937 scm: dat_psp_create returns wrong error code on bind/listen failure
938 scm: socket connect request count is reset improperly on retry
939 scm: when hostname has loopback addr assigned, default to eth0 instead of failing
940 scm: add port number to error log during hca_open failures
941 common: query calls return incorrect IA handle to consumer
942 common: srq create asserts with !dapl_llist_is_empty(head) failed
943
944 Release 2.0.33
945 scm,ucm: fix compatibility issues and set minimum protocol support
946 build: link librdmacm dependency to ib_acm usage for ucm and scm providers
947 build: add selective enable/disable-xxx build switch for each provider
948 build: add extended header files to EXTRA_DIST and fix missing backslash
949 build: set IB extended coll-type to none by default
950 common: change errno mapping of EINVAL to DAT_INVALID_PARAMETER
951 build: add IB collective and FCA provider to dapl build package as an option
952 common: add new dapls_evd_post_event_ext call for extended events
953 ucm: add support for IB collective providers
954 scm: add support for IB collective providers
955 cma: add support for IB collective providers
956 common: add supported collective types in named attributes for query
957 common: add collective call mappings via standard dapli_post_ext()
958 common: new debug bitmask definition for extension logging
959 common: new IB collective provider for Mellanox Fabric Collective Agent
960 dat: add definitions for MPI offloaded collectives in IB transport extensions
961 common: cleanup debug messages when building with ibacm feature
962
963 Release 2.0.32 fixes (OFED 1.5.3 GA):
964
965 cma: reduce output log level in disconnect from WARN to CM_WARN
966 ucm: delay freeing of active side UD cm object in case RTU is dropped
967 ucm: cm object needs to be on work queue before req sent on wire
968 ucm,scm: remove use of usec_sleep delays and use events for disc and destroy
969 common: reduce default max inline data size because of performance anomaly
970 common: dapls_evd_dto_wait() dbg message should print status and not errno
971 ucm, scm: exchange max_qp_rd_atom and limit outstanding requests
972 scm: retry socket connect on ECONNREFUSED under heavy load
973 common: qp modify RTR using wrong ep attribute parameter for dest_rd_atomic
974
975 Release 2.0.31 fixes (OFED 1.5.3 RC1):
976
977 common: clean up build warning for unused variable event_ptr
978 scm, ucm: set RAI_NOROUTE flag with rdma_getaddrinfo() call to avoid blocking.
979 cma: definition for dapl_sp_remove_ep() is missing in cm.c
980 libdat: static provider entries created for local SR database not freed
981 libdat: memory leak in static registration during parsing
982 common: increase default IB inline send threshold to 400
983 common cq: a mixup of errno and the -1 return from poll in dapls_wait_comp_channel
984 ucm: release UD cm objects after AH is exchanged to avoid duplicate request drops
985 ucm: decrease timeout retry count for disconnect requests
986 ucm: hold lock when sending cm_msgs to sync timer start with packet send
987 ucm: add debugging to include process id for better scale up debug aids
988 cma: disconnect can block for excessive times waiting for rdma_cm DREP timeout
989 ucm: configure the recv channel FD to non-blocking
990 windows: Missing librdmacm include path for build
991 debug build: only timestamp if sending to stdout to avoid performance hit
992 common: print out errors on free build and not just debug builds
993 cma: fix debug build issue
994 scm, ucm: MPI spawn test on oversubcribed server taking excessive time to complete
995 common: add high resolution time stamps and thread id to sdtout debug logs
996 common: modify debug in dat_evd_dequeue to reduce noise, only output on non-empty
997 cma: rdma_destroy_id called twice during device open bind error
998 common: dat_evd_dequeue (poll_cq) fails with invalid parameter after EP (qp) free
999 ucm: allow configuration of CM burst (signal) threshold on posting
1000 cma: fix debug build
1001 windows: debug version of windows does not build.
1002 Allow DAPL out of band connection models to use ibacm to obtain path record data.
1003 ucm: add missing map file for UCM provider
1004 ibal: delay QP transition during disconnect phase
1005 Revert "ibal: delay QP transition during disconnect phase"
1006 ibal: delay QP transition during disconnect phase
1007 common: restructure EVD processing to handle EP destruction phase
1008 ibal: sync QP destruction and device close
1009 ucm: remove unnecessary debug warning in async callback
1010
1011 v1.2 Package:
1012
1013 Release 1.2.19 fixes (OFED 1.5.2 GA):
1014
1015 common, cma: disconnect and cleanup CR linkings after DTO error on EP
1016 common: race conditions with DTO error, disconnect and dapl_reset_ep
1017 common: add new dapl_os_sleep_usec() function
1018 configure: need a false conditional for verbs attr.link_layer member check
1019 config: add conditional check for new verbs port_attr.link_layer
1020 cma, scm: new provider entries for Mellanox RDMA over Ethernet device for uDAPL v1.2
1021 cma: memory leak of verbs CQ and completion channels created during dat_ia_open
1022 cma: memory leak of FD's (pipe) created during dat_evd_create
1023
1024
1025 --- HISTORY -----------
1026
1027 OFED 1.5.1 RELEASE NOTES
1028 uDAPL v1 (1.2.16-1) and v2 (2.0.27-1)
1029
1030 ----------------
1031
1032 * New Features (v2 only) - UCM provider with IB UD based CM per process.
1033 More scalable then rdma_cm (cma) or socket cm (scm).
1034 ----------------
1035
1036 * Bug Fixes
1037
1038 V2.0 Package
1039
1040 Release 2.0.27
1041 windows: add scm makefile
1042 windows does not require rdma_cma_abi.h, move the include from common code
1043 windows patch to fix IB_INVALID_HANDLE name collision
1044 scm: dat_ep_connect fails on 32bit servers
1045 undefined symbol: dapls_print_cm_list
1046 cleanup CM object lock before freeing CM object memory
1047 destroy verbs completion channels created via ia_open or ep_create.
1048 package: update Copyright file and include the 3 license files in distribution
1049 common: when copying private_data out of rdma_cm events, use the
1050 cma: fix referencing freed address
1051 dapl: move close device after async thread is done
1052
1053 Release 2.0.26
1054 openib_common: add check for both gid and global routing in RTR
1055 openib_common: remote memory read privilege set multi times
1056 ucm, scm: DAPL_GLOBAL_ROUTING enabled causes segv
1057
1058 Release 2.0.25
1059 winof scm: initialize opt for NODELAY setsockopt
1060 winof cma: windows definition for EADDRNOTAVAIL missing
1061 scm: client side setsockopt NODELAY fails if data arrives before setting
1062 cma: setup_listener Cannot assign requested address
1063 common: seg fault in dapl_evd_wait with multi-thread application using CNO's.
1064 ucm: inbound DREQ/DREP handshake should transition QP.
1065 winof: Remove duplicate include of comp_channel.cpp from cm.c as it is
1066 included in opensm_ucb/device.c.
1067
1068 Release 2.0.24
1069 winof: Utilize WinOF version of inet_ntop() for Windows OSes which do not
1070 support inet_ntop().
1071 ucm: windows build issue with new CQ completion channel
1072 winof: add ucm provider to windows build
1073 winof: add missing build files for ibal, scm
1074 scm: connection peer resets under heavy load, incorrect event on error
1075 ucm: increase default reply and rtu timeout values.
1076 ucm: change some debug message levels and add check for valid UD REPLY during retries.
1077 ucm: increase timers during subsequent retries
1078 ucm, scm: address handles need destroyed when freeing Endpoints with UD QP's.
1079 openib_common: ignore pd free errors, clear pd_handle and return.
1080 ucm: using UD type QP's, ucm reports wrong reject event when user rejects AH resolution request.
1081 ucm, scm, cma: Fix CNO support on DTO type EVD's
1082 ucm: fix lock init bug in ucm_cm_find
1083 ucm: fix build problem with latest windows ucm changes
1084 ucm: The HCA should not be closed until all resources have been released.
1085 ucm: Fix build warning when compiling on 32-bit systems.
1086 ucm: Trying to deregister the same memory region twice leads to an
1087 dat: reduce debug message level when parsing for location of dat.conf
1088 ucm: update ucm provider for windows environment
1089 ucm: add timer/retry CM logic to the ucm provider
1090
1091 Release 2.0.23
1092 cma: cannot reuse the cm_id and qp for new connection, must reallocate a new one.
1093 scm, cma: update DAPL cm protocol revision with latest address/port changes
1094 ucm: modify IB address format to align better with sockaddr_in6
1095 Add definition for getpid similar to that used by the other dtest apps.
1096 WinOF provides a common implementation of gettimeofday that should
1097 The completion manager was updated to provide an abstraction that
1098 dtestcm: remove IB verb definitions
1099 dtest, dtestx: remove IB verb definitions
1100 scm: tighten up socket options to insure similiar behavior on Windows and Linux.
1101 cma: improve serialization of destroy and event processing
1102 scm: improve serialization of destroy and state changes
1103 common: no cleanup/release code for timer thread
1104 scm, cma: dapli_thread doesn't always get teminated on library close.
1105 ucm: tighten up locking with CM processing, state changes
1106 ucm: For UD type QP's, return CR p_data with CONN_EST event on passive side.
1107 ucm: cleanup extra cr/lf
1108 ucm: fix issues with UD QP's.
1109 winof: Convert windows version of dapl and dat libaries to use private heaps.
1110 dtest, dtestx: modifications for UD QP testing with ucm provider.
1111 scm, ucm: UD QP support was broken when porting to common openib code base.
1112 cma: cleanup warning with unused local variable, ret, in disconnect
1113 cma: remove debug message after rdma_disconnect failure
1114 scm: socket errno check needs O/S dependent wrapper
1115 dapltest: update script files for WinOF
1116 cma: conditional check for new rdma_cm definition.
1117
1118 Release 2.0.22
1119 dapltest: add mdep processor yield and use with dapltest
1120 ucm: Add new provider using a DAPL based IB-UD cm mechanism for MPI implementations.
1121
1122 Release 2.0.21
1123 scm: Fix disconnect. QP's need to move to ERROR state in
1124 modify dtest.c to cleanup CNO wait code and consolidate into
1125 CNO events, once triggered will not be returned during the cno wait.
1126 CNO support broken in both CMA and SCM providers.
1127 common osd: include winsock2.h for IPv6 definitions.
1128 common osd: include w2tcpip.h for sockaddr_in6 definitions.
1129 DAPL introduced the concept of directly waiting on the CQ for
1130 dapltest: Implement a malloc() threshold for the completion reaping.
1131 scm: handle connected state when freeing CM objects
1132 scm, dtest: changes for winof gettimeofday and FD_SETSIZE settings.
1133 scm: set TCP_NODELAY sockopt on the server side for sends.
1134 remove obsolete files in dapl/udapl source tree
1135 dtestcm: add UD type QP option to test
1136 scm: destroy QP called before disconnect
1137 cma: add support for rdma_cm TIME_WAIT event.
1138 scm: remove old udapl_scm code replaced by openib_scm.
1139 winof: fix issues after consolidating cma, scm code base.
1140 cma: lock held when exiting as a result of a rdma_create_event_channel failure.
1141 windows: all dlist functions have been moved to the header file.
1142 dtestcm windows: add build infrastructure for new dtestcm test suite
1143 openib_common: reorganize code base to share common mem, cq, qp, dto functions
1144 scm: fixes and optimizations for connection scaling
1145 scm: double the default fd_set_size
1146 scm: EP reference in CR should be cleared during ep_destroy
1147 dtestx: fix conn establishment event checking
1148 dtestcm: new test to measure dapl connection rates.
1149
1150 Release 2.0.20
1151 common,scm: add debug capabilities to print in-process CM lists
1152 scm: disconnect EP before cleaning up orphaned CR's during dat_ep_free
1153 dapltest: windows scripts updated
1154 scm: private data is not handled properly via CR rejects.
1155 scm: cleanup orphaned UD CR's when destroying the EP
1156 scm: provider specific query for default UD MTU is wrong.
1157 scm: update CM code to shutdown before closing socket
1158 dapltest: windows script dt-cli.bat updated
1159 dapl/windows cma provider: add support for network devices based on index
1160 openib: remove 1st gen provider, replaced with openib_cma and openib_scm
1161 dapltest: update windows script files
1162 dapltest: windows batch files in sripts directory
1163 windows_osd/linux_osd: new dapl_os_gettid macro to return thread id
1164 windows: missing build files for common and udapl sub-directories
1165 windows: add build files for openib_scm, remove /Wp64 build option.
1166 scm: multi-hca CM processing broken. Need cr thread wakeup mechanism per HCA.
1167 dtest: add connection timers on client side
1168 linux_osd: use pthread_self instead of getpid for debug messages
1169 windows ibal-scm: dapl/dirs file needs updated to remove ibal-scm
1170
1171 v1.2 Package:
1172
1173 Release 1.2.16
1174 package: update Copyright file and include the 3 license files in distribution
1175 cma: max sge incorrectly decremented during ibv_device_query
1176
1177 Release 1.2.15
1178 dtest, dapltest: conflict with dapl-2 utils package, change to dapl1, dapltest1
1179 scm: fix compiler warning, unused variable
1180
1181 ----------------
1182
1183 * BKM for running new DAPL library on your cluster without any impact on existing OFED installation:
1184
1185 Note: example for user /home/user1, (assumes /home/user1 is exported) and MLX4 adapter, port 1
1186
1187 Download latest 2.x package: http://www.openfabrics.org/downloads/dapl/dapl-2.0.25.tar.gz
1188
1189 untar in /home/user1
1190 cd /home/user1/dapl-2.0.25
1191 ./configure && make (build on node with OFED 1.3 or higher installed, dependency on verb/rdma_cm libraries)
1192
1193 create /home/user1/dat.conf with following 3 lines. (entries with path to new libraries):
1194
1195 ofa-v2-ib0 u2.0 nonthreadsafe default /home/user1/dapl-2.0.19/dapl/udapl/.libs/libdaplcma.so.1 dapl.2.0 "ib0 0" ""
1196 ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default /home/user1/dapl-2.0.19/dapl/udapl/.libs/libdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
1197 ofa-v2-mlx4_0-1u u2.0 nonthreadsafe default /home/user1/dapl-2.0.19/dapl/udapl/.libs/libdaploucm.so.2 dapl.2.0 "mlx4_0 1" ""
1198
1199 Run uDAPL application or an MPI that uses uDAPL, with (assuming MLX4 connectx adapters) following:
1200
1201 setenv DAT_OVERRIDE=/home/user1/dat.conf
1202
1203 If running Intel MPI and uDAPL socket cm, set the following:
1204
1205 setenv I_MPI_DEVICE=rdssm:ofa-v2-mlx4_0-1
1206
1207 or if running Intel MPI and uDAPL IB UD cm, set the following:
1208
1209 setenv I_MPI_DEVICE=rdssm:ofa-v2-mlx4_0-1u
1210
1211 or if running Intel MPI and uDAPL rdma_cm, set the following:
1212
1213 setenv I_MPI_DEVICE=rdssm:ofa-v2-ib0
1214
1215 -------------------------
1216
1217 OFED 1.4.1 RELEASE NOTES
1218
1219 NEW SINCE OFED 1.4 - new versions of uDAPL v1 (1.2.14-1) and v2 (2.0.19-1)
1220
1221 * New Features - optional counters, must be configured/built with -DDAPL_COUNTERS
1222
1223 * Bug Fixes
1224
1225 v2 - scm, cma: dat max_lmr_block_size is 32 bit, verbs max_mr_size is 64 bit
1226 v2 - scm, cma: use direct SGE mappings from dat_lmr_triplet to ibv_sge
1227 v2 - dtest: add flush EVD call after data transfer errors
1228 v2 - scm: increase default MTU size from 1024 to 2048
1229 v2 - dapltest: reset server listen ports to avoid collisions during long runs
1230 v2 - dapltest: avoid duplicating ports, increment based on ep/thread count
1231 v2 - dapltest: fix assumptions that multiple EP's will connect in order
1232 v2 - common: sync missing with when removing items off of EVD pending queue
1233 v2 - scm: reduce open time with thread start up
1234 v2 - scm: getsockopt optlen needs initialized to size of optval
1235 v2 - scm: cr_thread cleanup
1236 v2 - OFED and WinOF code sync
1237 v2 - scm: remove unnecessary query gid/lid from connection phase code.
1238 v2 - scm: add optional 64-bit counters, build with -DDAPL_COUNTERS.
1239 v1,v2 - spec files missing Requires(post) statements for sed/coreutils
1240 v1,v2 - dtest/dapltest: use $(top_builddir) for .la files during test builds
1241 v1,v2 - scm: remove unecessary thread when using direct objects
1242 v1,v2 - Fix SuSE 11 build issues, asm/atomic.h no longer exists
1243
1244 * Build Notes:
1245
1246 # NON_DEBUG build/install example for x86_64, OFED targets
1247 ./configure --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include"
1248 make install
1249
1250 # DEBUG build/install example for x86_64, using OFED targets
1251 ./configure --enable-debug --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include"
1252 make install
1253
1254 # COUNTERS build/install example for x86_64, using OFED targets
1255 ./configure --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include -DDAPL_COUNTERS"
1256 make install
1257
1258 * BKM for running new DAPL library on your cluster without any impact on existing OFED installation:
1259
1260 Note: example for user /home/user1, (assumes /home/user1 is exported) and MLX4 adapter, port 1
1261
1262 Download latest 2.x package: http://www.openfabrics.org/downloads/dapl/dapl-2.0.19.tar.gz
1263
1264 untar in /home/user1
1265 cd /home/user1/dapl-2.0.19
1266 ./configure && make (build on node with OFED 1.3 or higher installed, dependency on verb/rdma_cm libraries)
1267
1268 create /home/user1/dat.conf with following 2 lines. (entries with path to new libraries):
1269
1270 ofa-v2-ib0 u2.0 nonthreadsafe default /home/user1/dapl-2.0.19/dapl/udapl/.libs/libdaplcma.so.1 dapl.2.0 "ib0 0" ""
1271 ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default /home/user1/dapl-2.0.19/dapl/udapl/.libs/libdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
1272
1273 Run uDAPL application or an MPI that uses uDAPL, with (assuming MLX4 connectx adapters) following:
1274
1275 setenv DAT_OVERRIDE=/home/user1/dat.conf
1276
1277 If running Intel MPI and uDAPL socket cm, set the following:
1278
1279 setenv I_MPI_DEVICE=rdssm:ofa-v2-mlx4_0-1
1280
1281 if running Intel MPI and uDAPL rdma_cm, set the following:
1282
1283 setenv I_MPI_DEVICE=rdssm:ofa-v2-ib0
1284
1285 -------------------------
1286
1287 OFED 1.4 RELEASE NOTES
1288
1289 NEW SINCE OFED 1.3.1 - new versions of uDAPL v1 (1.2.12-1) and v2 (2.0.15-1)
1290
1291 * New Features
1292
1293 1. The new socket CM provider, introduced in 1.2.8 and 2.0.11 packages,
1294 assumes homogeneous cluster and will setup the QP's based on local HCA port
1295 attributes and exchanges QP information via socket's using the hostname of
1296 each node. IPoIB and rdma_cm are NOT required for this provider. QP attributes
1297 can be adjusted via the following environment parameters:
1298
1299 DAPL_ACK_TIMER (default=16 5 bits, 4.096us*2^ack_timer. 16 == 268ms)
1300 DAPL_ACK_RETRY (default=7 3 bits, 7 * 268ms = 1.8 seconds)
1301 DAPL_RNR_TIMER (default=12 5 bits, 12 == 64ms, 28 == 163ms, 31 == 491ms)
1302 DAPL_RNR_RETRY (default=7 3 bits, 7 == infinite)
1303 DAPL_IB_MTU (default=1024 limited to active MTU max)
1304
1305 The new socket cm entries in /etc/dat.conf provide a link to the actual HCA
1306 device and port. Example v1 and v2 entries for a Mellanox connectx device, port 1:
1307
1308 OpenIB-mlx4_0-1 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "mlx4_0 1" ""
1309 ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
1310
1311 This new socket cm provider, was successfully tested on the TATA CRL cluster
1312 (#8 on Top500) with Intel MPI, achieving a HPLinpack score of 132.8TFlops on
1313 1798 nodes, 14384 cores at ~76.9% of peak. DAPL_ACK_TIMER was increased to 21
1314 for this scale.
1315
1316 2. New v2 definitions for IB unreliable datagram extension (only supported in
1317 scm provider, libdaploscm.so.2)
1318
1319 Extended EP dat_service_type, with DAT_IB_SERVICE_TYPE_UD
1320 Add IB extension call dat_ib_post_send_ud().
1321 Add address handle definition for UD calls.
1322 Add IB event definitions to provide remote AH via connect and connect requests
1323 See dtestx (-d) source for example usage model
1324
1325 * Bug Fixes
1326
1327 v1,v2 - dapltest: trans test moves to cleanup stage before rdma_read processing is complete
1328 v1,v2 - Fix static registration (dat.conf) to include sysconfdir override
1329 v1,v2 - dat.conf: add default iwarp entry for eth2
1330 v1,v2 - dapl: adjust max_rdma_read_iov to 1 for iWARP devices
1331 v1,v2 - dtest: reduce default IOV's for ep_create to support iWARP
1332 v1,v2 - dtest: fix 32-bit build issues
1333 v1,v2 - build: $(DESTDIR) prepend needed on install hooks for dat.conf
1334 v2 - scm: UD shares EP;s which requires serialization
1335 v2 - dapl: fixes for IB UD extensions in common code and socket cm provider.
1336 v2 - dapl: add provider specific attribute query option for IB UD MTU size
1337 v2 - dapl build: add correct CFLAGS, set non-debug build by default for v2
1338 v2 - dtestx: fix stack corruption problem with hostname strcpy
1339 v2 - dapl extension: dapli_post_ext should always allocate cookie for requests.
1340 v2 - dapltest: manpage - rdma write example incorrect
1341 v1,v2 - dat, dapl, dtest, dapltest, providers: fix compiler warnings in dat common code
1342 v1,v2 - dapl cma: debug message during query needs definition for inet_ntoa
1343 v1,v2 - dapl scm: fix corner case that delivers duplicate disconnect events
1344 v1,v2 - dat: include stddef.h for NULL definition in dat_platform_specific.h
1345 v1,v2 - dapl: add debug messages during async and overflow events
1346 v1,v2 - dapltest: add check for duplicate disconnect events in transaction test
1347 v1,v2 - dapl scm: use correct device attribute for max_rdma_read_out, max_qp_init_rd_atom
1348 v1,v2 - dapl scm: change IB RC qp inline and timer defaults.
1349 v1,v2 - dapl scm: add mtu adjustments via environment, default = 1024.
1350 v1,v2 - dapl scm: change connect and accept to non-blocking to avoid blocking user thread.
1351 v1,v2 - dapl scm: update max_rdma_read_iov, max_rdma_write_iov EP attributes during query
1352 v1,v2 - dat: allow TYPE_ERR messages to be turned off with DAT_DBG_TYPE
1353 v1,v2 - dapl: remove needless terminating 0 in dto_op_str functions.
1354 v1,v2 - dat: remove reference to doc/dat.conf in makefile.am
1355 v1,v2 - dapl scm: fix ibv_destroy_cq busy error condition during dat_evd_free.
1356 v1,v2 - dapl scm: add stdout logging for uname and gethostbyname errors during open.
1357 v1,v2 - dapl scm: support global routing and set mtu based on active_mtu
1358 v1,v2 - dapl: add opcode to string function to report opcode during failures.
1359 v1,v2 - dapl: remove unused iov buffer allocation on the endpoint
1360 v1,v2 - dapl: endpoint pending request count is wrong
1361
1362 -------------------------
1363
1364 OFED 1.3.1 RELEASE NOTES
1365
1366 NEW SINCE OFED 1.3 - new versions of uDAPL v1 (1.2.7-1) and v2 (2.0.9-1)
1367
1368 * New Features - None
1369
1370 * Bug Fixes
1371 v2 - add private data exchange with reject
1372 v1,v2 - better error reporting in non-debug builds
1373 v1,v2 - update only OFA entries in dat.conf, cooperate with non-ofa providers
1374 v1,v2 - support for zero byte operations, iov==NULL
1375 v1,v2 - multi-transport support for inline data and private data differences
1376 v1,v2 - fix memory leaks and other reported bugs since OFED 1.3
1377 v1,v2 - dtest,dtestx,dapltest build issues on RHEL5.1
1378 v1,v2 - long delay during dat_ia_open when DNS not configured
1379 v1,v2 - use rdma_read_in/out from ep_attr per consumer instead of HCA max
1380
1381 -------------------------
1382
1383 OFED 1.3 RELEASE NOTES
1384
1385 NEW SINCE OFED 1.2
1386
1387 * New Features
1388
1389 1. Add v2.0 library support for new 2.0 API Specification
1390 2. Separate v1.2 library release to co-exist with v2.0 libraries.
1391 3. New dat.conf with both 1.2 and 2.0 support
1392 4. New v2.0 dtestx utilities to test IB extensions
1393
1394 * Bug Fixes
1395
1396 v1.2 and v2.0
1397 - uDAT: static/dynamic registry parsing fixes
1398 - uDAPL: provider fixes for dat_psp_create_any
1399 - dtest/dapltest: change default provider names to sync with dat.conf
1400 - openib_cma: issues with destroy_cm_id and init/resp exchange
1401 - dapltest: use gettimeofday instead of get_cycles for better portability
1402 - dapltest: endian issue with mem_handle, mem_address
1403 - dapltest fix to include inet_ntoa definitions
1404 - fix build problems on 32-bit and 64-bit PowerPC
1405 - cleanup packaging
1406
1407 v2.0
1408 - set default config options to match spec file, --enable-debug --enable-ext-type=ib
1409 - use unique devel target names, libdat2.so, /usr/include/dat2
1410 - dtestx fix memory leak, freeaddrinfo after getaddrinfo
1411 - Fix for IB extended DTO cookie deallocation on inbound rdma_Write_immed
1412 - WinOF: Update OFED code base to include WinOF changes, work from same code base
1413 - WinOF: add DAT_API definition, __stdcall for windows, nothing for linux
1414 - dtest: add dat_evd_query to check correct size
1415 - openib_cma: add macro to convert SID to PORT
1416 - dtest: endian support for exchanging RMR info
1417 - openib_cma: lower default settings, inline and RDMA init/resp
1418 - openib_cma: missing ia_query for max_iov_segments_per_rdma_write
1419
1420 v1.2
1421 - openib_cma: turn down dbg noise level on rejects
1422 - dtest: typo in memset
1423
1424
1425 BUILD: v1 and v2 uDAPL source install/build instructions (redhat example):
1426
1427 # cd to distribution SRPMS directory
1428 cd /tmp/OFED-1.3/SRPMS
1429 rpm -i dapl-1.2*.rpm
1430 rpm -i dapl-2.0*.rpm
1431 cd /usr/src/redhat/SOURCES
1432 tar zxf dapl-1.2*.tgz
1433 tar zxf dapl-2.0*.tgz
1434
1435 # NON_DEBUG build example for x86_64, using OFED targets
1436
1437 ./configure --prefix /usr --sysconf=/etc --libdir /usr/lib64
1438 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include"
1439
1440 # build and install
1441
1442 make
1443 make install
1444
1445 # DEBUG build example for x86_64, using OFED targets
1446
1447 ./configure --enable-debug --prefix /usr --sysconf=/etc --libdir /usr/lib64
1448 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include"
1449
1450 # build and install
1451
1452 make
1453 make install
1454
1455 # DEBUG messages: set environment variable DAPL_DBG_TYPE, default
1456 mapping is 0x0003
1457
1458 DAPL_DBG_TYPE_ERR = 0x0001,
1459 DAPL_DBG_TYPE_WARN = 0x0002,
1460 DAPL_DBG_TYPE_EVD = 0x0004,
1461 DAPL_DBG_TYPE_CM = 0x0008,
1462 DAPL_DBG_TYPE_EP = 0x0010,
1463 DAPL_DBG_TYPE_UTIL = 0x0020,
1464 DAPL_DBG_TYPE_CALLBACK = 0x0040,
1465 DAPL_DBG_TYPE_DTO_COMP_ERR= 0x0080,
1466 DAPL_DBG_TYPE_API = 0x0100,
1467 DAPL_DBG_TYPE_RTN = 0x0200,
1468 DAPL_DBG_TYPE_EXCEPTION = 0x0400,
1469 DAPL_DBG_TYPE_SRQ = 0x0800,
1470 DAPL_DBG_TYPE_CNTR = 0x1000
1471
1472 -------------------------
1473
1474 OFED 1.2 RELEASE NOTES
1475
1476 NEW SINCE Gamma 3.2 and OFED 1.1
1477
1478 * New Features
1479
1480 1. Added dtest and dapltest to the openfabrics build and utils rpm.
1481 Includes manpages.
1482 2. Added following enviroment variables to configure connection management
1483 timers (default settings) for larger clusters:
1484
1485 DAPL_CM_ARP_TIMEOUT_MS 4000
1486 DAPL_CM_ARP_RETRY_COUNT 15
1487 DAPL_CM_ROUTE_TIMEOUT_MS 4000
1488 DAPL_CM_ROUTE_RETRY_COUNT 15
1489
1490 * Bug Fixes
1491
1492 + Added support for new ib verbs client register event. No extra
1493 processing required at the uDAPL level.
1494 + Fix some issues supporting create qp without recv cq handle or
1495 recv qp resources. IB verbs assume a recv_cq handle and uDAPL
1496 dapl_ep_create assumes there is always recv_sge resources specified.
1497 + Fix some timeout and long disconnect delay issues discovered during
1498 scale-out testing. Added support to retry rdma_cm address and route
1499 resolution with configuration options. Provide a disconnect call
1500 when receiving the disconnect request to guarantee a disconnect reply
1501 and event on the remote side. The rdma_disconnect was not being called
1502 from dat_ep_disconnect() as a result of the state changing
1503 to DISCONNECTED in the event callback.
1504 + Changes to support exchanging and validation of the device
1505 responder_resources and the initiator_depth during conn establishment
1506 + Fix some build issues with dapltest on 32 bit arch, and on ia64 SUSE arch
1507 + Add support for multiple IB devices to dat.conf to support IPoIB HA failover
1508 + Fix atomic operation build problem with ia64 and RHEL5.
1509 + Add support to return local and remote port information with dat_ep_query
1510 + Cleanup RPM specfile for the dapl package, move to 1.2-1 release.
1511
1512 NEW SINCE Gamma 3.1 and OFED 1.0
1513
1514 * BUG FIXES
1515
1516 + Update obsolete CLK_TCK to CLOCKS_PER_SEC
1517 + Fill out some unitialized fields in the ia_attr structure returned by
1518 dat_ia_query().
1519 + Update dtest to support multiple segments on rdma write and change
1520 makefile to use OpenIB-cma by default.
1521 + Add support for dat_evd_set_unwaitable on a DTO evd in openib_cma
1522 provider
1523 + Added errno reporting (message and return codes) during open to help
1524 diagnose create thread issues.
1525 + Fix some suspicious inline assembly EIEIO_ON_SMP and ISYNC_ON_SMP
1526 + Fix IA64 build problems
1527 + Lower the reject debug message level so we don't see warnings when
1528 consumers reject.
1529 + Added support for active side TIMED_OUT event from a provider.
1530 + Fix bug in dapls_ib_get_dat_event() call after adding new unreachable
1531 event.
1532 + Update for new rdma_create_id() function signature.
1533 + Set max rdma read per EP attributes
1534 + Report the proper error and timeout events.
1535 + Socket CM fix to guard against using a loopback address as the local
1536 device address.
1537 + Use the uCM set_option feature to adjust connect request timeout
1538 retry values.
1539 + Fix to disallow any event after a disconnect event.
1540
1541 * OFED 1.1 uDAPL source build instructions:
1542
1543 cd /usr/local/ofed/src/openib-1.1/src/userspace/dapl
1544
1545 # NON_DEBUG build configuration
1546
1547 ./configure --disable-libcheck --prefix /usr/local/ofed
1548 --libdir /usr/local/ofed/lib64 LDFLAGS=-L/usr/local/ofed/lib64
1549 CPPFLAGS="-I../libibverbs/include -I../librdmacm/include"
1550
1551 # build and install
1552
1553 make
1554 make install
1555
1556 # DEBUG build configuration
1557
1558 ./configure --disable-libcheck --enable-debug --prefix /usr/local/ofed
1559 --libdir /usr/local/ofed/lib64 LDFLAGS=-L/usr/local/ofed/lib64
1560 CPPFLAGS="-I../libibverbs/include -I../librdmacm/include"
1561
1562 # build and install
1563
1564 make
1565 make install
1566
1567 # DEBUG messages: set environment variable DAPL_DBG_TYPE, default
1568 mapping is 0x0003
1569
1570 DAPL_DBG_TYPE_ERR = 0x0001,
1571 DAPL_DBG_TYPE_WARN = 0x0002,
1572 DAPL_DBG_TYPE_EVD = 0x0004,
1573 DAPL_DBG_TYPE_CM = 0x0008,
1574 DAPL_DBG_TYPE_EP = 0x0010,
1575 DAPL_DBG_TYPE_UTIL = 0x0020,
1576 DAPL_DBG_TYPE_CALLBACK = 0x0040,
1577 DAPL_DBG_TYPE_DTO_COMP_ERR= 0x0080,
1578 DAPL_DBG_TYPE_API = 0x0100,
1579 DAPL_DBG_TYPE_RTN = 0x0200,
1580 DAPL_DBG_TYPE_EXCEPTION = 0x0400,
1581 DAPL_DBG_TYPE_SRQ = 0x0800,
1582 DAPL_DBG_TYPE_CNTR = 0x1000
1583
1584
1585 Note: The udapl provider library libdaplscm.so is untested and
1586 unsupported, thus customers should not use it.
1587 It will be removed in the next OFED release.
1588
1589 DAPL GAMMA 3.1 RELEASE NOTES
1590
1591 This release of the DAPL reference implementation
1592 is timed to coincide with the first release of the
1593 Open Fabrics (www.openfabrics.org) software stack.
1594 This release adds support for this new stack, which
1595 is now the native Linux RDMA stack.
1596
1597 This release also adds a new licensing option. In
1598 addition to the Common Public License and BSD License,
1599 the code can now be licensed under the terms of the GNU
1600 General Public License (GPL) version 2.
1601
1602 NEW SINCE Gamma 3.0
1603
1604 - GPL v2 added as a licensing option
1605 - OpenFabrics (aka OpenIB) gen2 verbs support
1606 - dapltest support for Solaris 10
1607
1608 * BUG FIXES
1609
1610 + Fixed a disconnect event processing race
1611 + Fix to destroy all QPs on IA close
1612 + Removed compiler warnings
1613 + Removed unused variables
1614 + And many more...
1615
1616 DAPL GAMMA 3.0 RELEASE NOTES
1617
1618 This is the first release based on version 1.2 of the spec. There
1619 are some components, such a shared receive queues (SRQs), which
1620 are not implemented yet.
1621
1622 Once again there were numerous bug fixes submitted by the
1623 DAPL community.
1624
1625 NEW SINCE Beta 2.06
1626
1627 - DAT 1.2 headers
1628 - DAT_IA_HANDLEs implemented as small integers
1629 - Changed default device name to be "ia0a"
1630 - Initial support for Linux 2.6.X kernels
1631 - Updates to the OpenIB gen 1 provider
1632
1633 * BUG FIXES
1634
1635 + Updated Makefile for differentiation between OS releases.
1636 + Updated atomic routines to use appropriate API
1637 + Removed unnecessary assert from atomic_dec.
1638 + Fixed bugs when freeing a PSP.
1639 + Fixed error codes returned by the DAT static registry.
1640 + Kernel updates for dat_strerror.
1641 + Cleaned up the transport layer/adapter interface to use DAPL
1642 types rather than transport types.
1643 + Fixed ring buffer reallocation.
1644 + Removed old test/udapl/dapltest directory.
1645 + Fixed DAT_IA_HANDLE translation (from pointer to int and
1646 vice versa) on 64-bit platforms.
1647
1648 DAP BETA 2.06 RELEASE NOTES
1649
1650 We are not planning any further releases of the Beta series,
1651 which are based on the 1.1 version of the spec. There may be
1652 further releases for bug fixes, but we anticipate the DAPL
1653 community to move to the new 1.2 version of the spec and the
1654 changes mandated in the reference implementation.
1655
1656 The biggest item in this release is the first inclusion of the
1657 OpenIB Gen 1 provider, an item generating a lot of interest in
1658 the IB community. This implementation has graciously been
1659 provided by the Mellanox team. The kdapl implementation is in
1660 progress, and we imagine work will soon begin on Gen 2.
1661
1662 There are also a handful of bug fixes available, as well as a long
1663 awaited update to the endpoint design document.
1664
1665 NEW SINCE Beta 2.05
1666
1667 - OpenIB gen 1 provider support has been added
1668 - Added dapls_evd_post_generic_event(), routine to post generic
1669 event types as requested by some providers. Also cleaned up
1670 error reporting.
1671 - Updated the endpoint design document in the doc/ directory.
1672
1673 * BUG FIXES
1674
1675 + Cleaned up memory leak on close by freeing the HCA structure;
1676 + Removed bogus #defs for rdtsc calls on IA64.
1677 + Changed daptest thread types to use internal types for
1678 portability & correctness
1679 + Various 64 bit enhancements & updates
1680 + Fixes to conformance test that were defining CONN_QUAL twice
1681 and using it in different ways
1682 + Cleaned up private data handling in ep_connect & provider
1683 support: we now avoid extra copy in connect code; reduced
1684 stack requirements by using private_data structure in the EP;
1685 removed provider variable.
1686 + Fixed problem in the dat conformance test where cno_wait would
1687 attempt to dereference a timer value and SEGV.
1688 + Removed old vestiges of depricated POLLING_COMPLETIONS
1689 conditionals.
1690
1691 DAPL BETA 2.05 RELEASE NOTES
1692
1693 This was to be a very minor release, the primary change was
1694 going to be the new wording of the DAT license as contained in
1695 the header for all source files. But the interest and
1696 development occurring in DAPL provided some extra bug fixes, and
1697 some new functionality that has been requested for a while.
1698
1699 First, you may notice that every single source file was
1700 changed. If you read the release notes from DAPL BETA 2.04, you
1701 were warned this would happen. There was a legal issue with the
1702 wording in the header, the end result was that every source file
1703 was required to change the word 'either of' to 'both'. We've
1704 been putting this change off as long as possible, but we wanted
1705 to do it in a clean drop before we start working on DAT 1.2
1706 changes in the reference implementation, just to keep things
1707 reasonably sane.
1708
1709 kdapltest has enabled three of the subtests supported by
1710 dapltest. The Performance test in particular has been very
1711 useful to dapltest in getting minima and maxima. The Limit test
1712 pushes the limits by allocating the maximum number of specific
1713 resources. And the FFT tests are also available.
1714
1715 Most vendors have supported shared memory regions for a while,
1716 several of which have asked the reference implementation team to
1717 provide a common implementation. Shared memory registration has
1718 been tested on ibapi, and compiled into vapi. Both InfiniBand
1719 providers have the restriction that a memory region must be
1720 created before it can be shared; not all RDMA APIs are this way,
1721 several allow you to declare a memory region shared when it is
1722 registered. Hence, details of the implementation are hidden in
1723 the provider layer, rather than forcing other APIs to do
1724 something strange.
1725
1726 This release also contains some changes that will allow dapl to
1727 work on Opteron processors, as well as some preliminary support
1728 for Power PC architecture. These features are not well tested
1729 and may be incomplete at this time.
1730
1731 Finally, we have been asked several times over the course of the
1732 project for a canonical interface between the common and
1733 provider layers. This release includes a dummy provider to meet
1734 that need. Anyone should be able to download the release and do
1735 a:
1736 make VERBS=DUMMY
1737
1738 And have a cleanly compiled dapl library. This will be useful
1739 both to those porting new transport providers, as well as those
1740 going to new machines.
1741
1742 The DUMMY provider has been compiled on both Linux and Windows
1743 machines.
1744
1745
1746 NEW SINCE Beta 2.4
1747 - kdapltest enhancements:
1748 * Limit subtests now work
1749 * Performance subtests now work.
1750 * FFT tests now work.
1751
1752 - The VAPI headers have been refreshed by Mellanox
1753
1754 - Initial Opteron and PPC support.
1755
1756 - Atomic data types now have consistent treatment, allowing us to
1757 use native data types other than integers. The Linux kdapl
1758 uses atomic_t, allowing dapl to use the kernel macros and
1759 eliminate the assembly code in dapl_osd.h
1760
1761 - The license language was updated per the direction of the
1762 DAT Collaborative. This two word change affected the header
1763 of every file in the tree.
1764
1765 - SHARED memory regions are now supported.
1766
1767 - Initial support for the TOPSPIN provider.
1768
1769 - Added a dummy provider, essentially the NULL provider. It's
1770 purpose is to aid in porting and to clarify exactly what is
1771 expected in a provider implementation.
1772
1773 - Removed memory allocation from the DTO path for VAPI
1774
1775 - cq_resize will now allow the CQ to be resized smaller. Not all
1776 providers support this, but it's a provider problem, not a
1777 limitation of the common code.
1778
1779 * BUG FIXES
1780
1781 + Removed spurious lock in dapl_evd_connection_callb.c that
1782 would have caused a deadlock.
1783 + The Async EVD was getting torn down too early, potentially
1784 causing lost errors. Has been moved later in the teardown
1785 process.
1786 + kDAPL replaced mem_map_reserve() with newer SetPageReserved()
1787 for better Linux integration.
1788 + kdapltest no longer allocate large print buffers on the stack,
1789 is more careful to ensure buffers don't overflow.
1790 + Put dapl_os_dbg_print() under DAPL_DBG conditional, it is
1791 supposed to go away in a production build.
1792 + dapltest protocol version has been bumped to reflect the
1793 change in the Service ID.
1794 + Corrected several instances of routines that did not adhere
1795 to the DAT 1.1 error code scheme.
1796 + Cleaned up vapi ib_reject_connection to pass DAT types rather
1797 than provider specific types. Also cleaned up naming interface
1798 declarations and their use in vapi_cm.c; fixed incorrect
1799 #ifdef for naming.
1800 + Initialize missing uDAPL provider attr, pz_support.
1801 + Changes for better layering: first, moved
1802 dapl_lmr_convert_privileges to the provider layer as memory
1803 permissions are clearly transport specific and are not always
1804 defined in an integer bitfield; removed common routines for
1805 lmr and rmr. Second, move init and release setup/teardown
1806 routines into adapter_util.h, which defined the provider
1807 interface.
1808 + Cleaned up the HCA name cruft that allowed different types
1809 of names such as strings or ints to be dealt with in common
1810 code; but all names are presented by the dat_registry as
1811 strings, so pushed conversions down to the provider
1812 level. Greatly simplifies names.
1813 + Changed deprecated true/false to DAT_TRUE/DAT_FALSE.
1814 + Removed old IB_HCA_NAME type in favor of char *.
1815 + Fixed race condition in kdapltest's use of dat_evd_dequeue.
1816 + Changed cast for SERVER_PORT_NUMBER to DAT_CONN_QUAL as it
1817 should be.
1818 + Small code reorg to put the CNO into the EVD when it is
1819 allocated, which simplifies things.
1820 + Removed gratuitous ib_hca_port_t and ib_send_op_type_t types,
1821 replaced with standard int.
1822 + Pass a pointer to cqe debug routine, not a structure. Some
1823 clean up of data types.
1824 + kdapl threads now invoke reparent_to_init() on exit to allow
1825 threads to get cleaned up.
1826
1827
1828
1829 DAPL BETA 2.04 RELEASE NOTES
1830
1831 The big changes for this release involve a more strict adherence
1832 to the original dapl architecture. Originally, only InfiniBand
1833 providers were available, so allowing various data types and
1834 event codes to show through into common code wasn't a big deal.
1835
1836 But today, there are an increasing number of providers available
1837 on a number of transports. Requiring an IP iWarp provider to
1838 match up to InfiniBand events is silly, for example.
1839
1840 Restructuring the code allows more flexibility in providing an
1841 implementation.
1842
1843 There are also a large number of bug fixes available in this
1844 release, particularly in kdapl related code.
1845
1846 Be warned that the next release will change every file in the
1847 tree as we move to the newly approved DAT license. This is a
1848 small change, but all files are affected.
1849
1850 Future releases will also support to the soon to be ratified DAT
1851 1.2 specification.
1852
1853 This release has benefited from many bug reports and fixes from
1854 a number of individuals and companies. On behalf of the DAPL
1855 community, thank you!
1856
1857
1858 NEW SINCE Beta 2.3
1859
1860 - Made several changes to be more rigorous on the layering
1861 design of dapl. The intent is to make it easier for non
1862 InfiniBand transports to use dapl. These changes include:
1863
1864 * Revamped the ib_hca_open/close code to use an hca_ptr
1865 rather than an ib_handle, giving the transport layer more
1866 flexibility in assigning transport handles and resources.
1867
1868 * Removed the CQD calls, they are specific to the IBM API;
1869 folded this functionality into the provider open/close calls.
1870
1871 * Moved VAPI, IBAPI transport specific items into a transport
1872 structure placed inside of the HCA structure. Also updated
1873 routines using these fields to use the new location. Cleaned
1874 up provider knobs that have been exposed for too long.
1875
1876 * Changed a number of provider routines to use DAPL structure
1877 pointers rather than exposing provider handles & values. Moved
1878 provider specific items out of common code, including provider
1879 data types (e.g. ib_uint32_t).
1880
1881 * Pushed provider completion codes and type back into the
1882 provider layer. We no longer use EVD or CM completion types at
1883 the common layer, instead we obtain the appropriate DAT type
1884 from the provider and process only DAT types.
1885
1886 * Change private_data handling such that we can now accommodate
1887 variable length private data.
1888
1889 - Remove DAT 1.0 cruft from the DAT header files.
1890
1891 - Better spec compliance in headers and various routines.
1892
1893 - Major updates to the VAPI implementation from
1894 Mellanox. Includes initial kdapl implementation
1895
1896 - Move kdapl platform specific support for hash routines into
1897 OSD file.
1898
1899 - Cleanups to make the code more readable, including comments
1900 and certain variable and structure names.
1901
1902 - Fixed CM_BUSTED code so that it works again: very useful for
1903 new dapl ports where infrastructure is lacking. Also made
1904 some fixes for IBHOSTS_NAMING conditional code.
1905
1906 - Added DAPL_MERGE_CM_DTO as a compile time switch to support
1907 EVD stream merging of CM and DTO events. Default is off.
1908
1909 - 'Quit' test ported to kdapltest
1910
1911 - uDAPL now builds on Linux 2.6 platform (SuSE 9.1).
1912
1913 - kDAPL now builds for a larger range of Linux kernels, but
1914 still lacks 2.6 support.
1915
1916 - Added shared memory ID to LMR structure. Shared memory is
1917 still not fully supported in the reference implementation, but
1918 the common code will appear soon.
1919
1920 * Bug fixes
1921 - Various Makefiles fixed to use the correct dat registry
1922 library in its new location (as of Beta 2.03)
1923 - Simple reorg of dat headers files to be consistent with
1924 the spec.
1925 - fixed bug in vapi_dto.h recv macro where we could have an
1926 uninitialized pointer.
1927 - Simple fix in dat_dr.c to initialize a variable early in the
1928 routine before errors occur.
1929 - Removed private data pointers from a CONNECTED event, as
1930 there should be no private data here.
1931 - dat_strerror no longer returns an uninitialized pointer if
1932 the error code is not recognized.
1933 - dat_dup_connect() will reject 0 timeout values, per the
1934 spec.
1935 - Removed unused internal_hca_names parameter from
1936 ib_enum_hcas() interface.
1937 - Use a temporary DAT_EVENT for kdapl up-calls rather than
1938 making assumptions about the current event queue.
1939 - Relocated some platform dependent code to an OSD file.
1940 - Eliminated several #ifdefs in .c files.
1941 - Inserted a missing unlock() on an error path.
1942 - Added bounds checking on size of private data to make sure
1943 we don't overrun the buffer
1944 - Fixed a kdapltest problem that caused a machine to panic if
1945 the user hit ^C
1946 - kdapltest now uses spin locks more appropriate for their
1947 context, e.g. spin_lock_bh or spin_lock_irq. Under a
1948 conditional.
1949 - Fixed kdapltest loops that drain EVDs so they don't go into
1950 endless loops.
1951 - Fixed bug in dapl_llist_add_entry link list code.
1952 - Better error reporting from provider code.
1953 - Handle case of user trying to reap DTO completions on an
1954 EP that has been freed.
1955 - No longer hold lock when ep_free() calls into provider layer
1956 - Fixed cr_accept() to not have an extra copy of
1957 private_data.
1958 - Verify private_data pointers before using them, avoid
1959 panic.
1960 - Fixed memory leak in kdapltest where print buffers were not
1961 getting reclaimed.
1962
1963
1964
1965 DAPL BETA 2.03 RELEASE NOTES
1966
1967 There are some prominent features in this release:
1968 1) dapltest/kdapltest. The dapltest test program has been
1969 rearchitected such that a kernel version is now available
1970 to test with kdapl. The most obvious change is a new
1971 directory structure that more closely matches other core
1972 dapl software. But there are a large number of changes
1973 throughout the source files to accommodate both the
1974 differences in udapl/kdapl interfaces, but also more mundane
1975 things such as printing.
1976
1977 The new dapltest is in the tree at ./test/dapltest, while the
1978 old remains at ./test/udapl/dapltest. For this release, we
1979 have maintained both versions. In a future release, perhaps
1980 the next release, the old dapltest directory will be
1981 removed. Ongoing development will only occur in the new tree.
1982
1983 2) DAT 1.1 compliance. The DAT Collaborative has been busy
1984 finalizing the 1.1 revision of the spec. The header files
1985 have been reviewed and posted on the DAT Collaborative web
1986 site, they are now in full compliance.
1987
1988 The reference implementation has been at a 1.1 level for a
1989 while. The current implementation has some features that will
1990 be part of the 1.2 DAT specification, but only in places
1991 where full compatibility can be maintained.
1992
1993 3) The DAT Registry has undergone some positive changes for
1994 robustness and support of more platforms. It now has the
1995 ability to support several identical provider names
1996 simultaneously, which enables the same dat.conf file to
1997 support multiple platforms. The registry will open each
1998 library and return when successful. For example, a dat.conf
1999 file may contain multiple provider names for ex0a, each
2000 pointing to a different library that may represent different
2001 platforms or vendors. This simplifies distribution into
2002 different environments by enabling the use of common
2003 dat.conf files.
2004
2005 In addition, there are a large number of bug fixes throughout
2006 the code. Bug reports and fixes have come from a number of
2007 companies.
2008
2009 Also note that the Release notes are cleaned up, no longer
2010 containing the complete text of previous releases.
2011
2012 * EVDs no longer support DTO and CONNECTION event types on the
2013 same EVD. NOTE: The problem is maintaining the event ordering
2014 between two channels such that no DTO completes before a
2015 connection is received; and no DTO completes after a
2016 disconnect is received. For 90% of the cases this can be made
2017 to work, but the remaining 10% will cause serious performance
2018 degradation to get right.
2019
2020 NEW SINCE Beta 2.2
2021
2022 * DAT 1.1 spec compliance. This includes some new types, error
2023 codes, and moving structures around in the header files,
2024 among other things. Note the Class bits of dat_error.h have
2025 returned to a #define (from an enum) to cover the broadest
2026 range of platforms.
2027
2028 * Several additions for robustness, including handle and
2029 pointer checking, better argument checking, state
2030 verification, etc. Better recovery from error conditions,
2031 and some assert()s have been replaced with 'if' statements to
2032 handle the error.
2033
2034 * EVDs now maintain the actual queue length, rather than the
2035 requested amount. Both the DAT spec and IB (and other
2036 transports) allow the underlying implementation to provide
2037 more CQ entries than requested.
2038
2039 Requests for the same number of entries contained by an EVD
2040 return immediate success.
2041
2042 * kDAPL enhancements:
2043 - module parameters & OS support calls updated to work with
2044 more recent Linux kernels.
2045 - kDAPL build options changes to match the Linux kernel, vastly
2046 reducing the size and making it more robust.
2047 - kDAPL unload now works properly
2048 - kDAPL takes a reference on the provider driver when it
2049 obtains a verbs vector, to prevent an accidental unload
2050 - Cleaned out all of the uDAPL cruft from the linux/osd files.
2051
2052 * New dapltest (see above).
2053
2054 * Added a new I/O trace facility, enabling a developer to debug
2055 all I/O that are in progress or recently completed. Default
2056 is OFF in the build.
2057
2058 * 0 timeout connections now refused, per the spec.
2059
2060 * Moved the remaining uDAPL specific files from the common/
2061 directory to udapl/. Also removed udapl files from the kdapl
2062 build.
2063
2064 * Bug fixes
2065 - Better error reporting from provider layer
2066 - Fixed race condition on reference counts for posting DTO
2067 ops.
2068 - Use DAT_COMPLETION_SUPPRESS_FLAG to suppress successful
2069 completion of dapl_rmr_bind (instead of
2070 DAT_COMPLEITON_UNSIGNALLED, which is for non-notification
2071 completion).
2072 - Verify psp_flags value per the spec
2073 - Bug in psp_create_any() checking psp_flags fixed
2074 - Fixed type of flags in ib_disconnect from
2075 DAT_COMPLETION_FLAGS to DAT_CLOSE_FLAGS
2076 - Removed hard coded check for ASYNC_EVD. Placed all EVD
2077 prevention in evd_stream_merging_supported array, and
2078 prevent ASYNC_EVD from being created by an app.
2079 - ep_free() fixed to comply with the spec
2080 - Replaced various printfs with dbg_log statements
2081 - Fixed kDAPL interaction with the Linux kernel
2082 - Corrected phy_register protottype
2083 - Corrected kDAPL wait/wakeup synchronization
2084 - Fixed kDAPL evd_kcreate() such that it no longer depends
2085 on uDAPL only code.
2086 - dapl_provider.h had wrong guard #def: changed DAT_PROVIDER_H
2087 to DAPL_PROVIDER_H
2088 - removed extra (and bogus) call to dapls_ib_completion_notify()
2089 in evd_kcreate.c
2090 - Inserted missing error code assignment in
2091 dapls_rbuf_realloc()
2092 - When a CONNECTED event arrives, make sure we are ready for
2093 it, else something bad may have happened to the EP and we
2094 just return; this replaces an explicit check for a single
2095 error condition, replacing it with the general check for the
2096 state capable of dealing with the request.
2097 - Better context pointer verification. Removed locks around
2098 call to ib_disconnect on an error path, which would result
2099 in a deadlock. Added code for BROKEN events.
2100 - Brought the vapi code more up to date: added conditional
2101 compile switches, removed obsolete __ActivePort, deal
2102 with 0 length DTO
2103 - Several dapltest fixes to bring the code up to the 1.1
2104 specification.
2105 - Fixed mismatched dalp_os_dbg_print() #else dapl_Dbg_Print();
2106 the latter was replaced with the former.
2107 - ep_state_subtype() now includes UNCONNECTED.
2108 - Added some missing ibapi error codes.
2109
2110
2111
2112 NEW SINCE Beta 2.1
2113
2114 * Changes for Erratta and 1.1 Spec
2115 - Removed DAT_NAME_NOT_FOUND, per DAT erratta
2116 - EVD's with DTO and CONNECTION flags set no longer valid.
2117 - Removed DAT_IS_SUCCESS macro
2118 - Moved provider attribute structures from vendor files to udat.h
2119 and kdat.h
2120 - kdapl UPCALL_OBJECT now passed by reference
2121
2122 * Completed dat_strerr return strings
2123
2124 * Now support interrupted system calls
2125
2126 * dapltest now used dat_strerror for error reporting.
2127
2128 * Large number of files were formatted to meet project standard,
2129 very cosmetic changes but improves readability and
2130 maintainability. Also cleaned up a number of comments during
2131 this effort.
2132
2133 * dat_registry and RPM file changes (contributed by Steffen Persvold):
2134 - Renamed the RPM name of the registry to be dat-registry
2135 (renamed the .spec file too, some cvs add/remove needed)
2136 - Added the ability to create RPMs as normal user (using
2137 temporal paths), works on SuSE, Fedora, and RedHat.
2138 - 'make rpm' now works even if you didn't build first.
2139 - Changed to using the GNU __attribute__((constructor)) and
2140 __attribute__((destructor)) on the dat_init functions, dat_init
2141 and dat_fini. The old -init and -fini options to LD makes
2142 applications crash on some platforms (Fedora for example).
2143 - Added support for 64 bit platforms.
2144 - Added code to allow multiple provider names in the registry,
2145 primarily to support ia32 and ia64 libraries simultaneously.
2146 Provider names are now kept in a list, the first successful
2147 library open will be the provider.
2148
2149 * Added initial infrastructure for DAPL_DCNTR, a feature that
2150 will aid in debug and tuning of a dapl implementation. Partial
2151 implementation only at this point.
2152
2153 * Bug fixes
2154 - Prevent debug messages from crashing dapl in EVD completions by
2155 verifying the error code to ensure data is valid.
2156 - Verify CNO before using it to clean up in evd_free()
2157 - CNO timeouts now return correct error codes, per the spec.
2158 - cr_accept now complies with the spec concerning connection
2159 requests that go away before the accept is invoked.
2160 - Verify valid EVD before posting connection evens on active side
2161 of a connection. EP locking also corrected.
2162 - Clean up of dapltest Makefile, no longer need to declare
2163 DAT_THREADSAFE
2164 - Fixed check of EP states to see if we need to disconnect an
2165 IA is closed.
2166 - ep_free() code reworked such that we can properly close a
2167 connection pending EP.
2168 - Changed disconnect processing to comply with the spec: user will
2169 see a BROKEN event, not DISCONNECTED.
2170 - If we get a DTO error, issue a disconnect to let the CM and
2171 the user know the EP state changed to disconnect; checked IBA
2172 spec to make sure we disconnect on correct error codes.
2173 - ep_disconnect now properly deals with abrupt disconnects on the
2174 active side of a connection.
2175 - PSP now created in the correct state for psp_create_any(), making
2176 it usable.
2177 - dapl_evd_resize() now returns correct status, instead of always
2178 DAT_NOT_IMPLEMENTED.
2179 - dapl_evd_modify_cno() does better error checking before invoking
2180 the provider layer, avoiding bugs.
2181 - Simple change to allow dapl_evd_modify_cno() to set the CNO to
2182 NULL, per the spec.
2183 - Added required locking around call to dapl_sp_remove_cr.
2184
2185 - Fixed problems related to dapl_ep_free: the new
2186 disconnect(abrupt) allows us to do a more immediate teardown of
2187 connections, removing the need for the MAGIC_EP_EXIT magic
2188 number/state, which has been removed. Mmuch cleanup of paths,
2189 and made more robust.
2190 - Made changes to meet the spec, uDAPL 1.1 6.3.2.3: CNO is
2191 triggered if there are waiters when the last EVD is removed
2192 or when the IA is freed.
2193 - Added code to deal with the provider synchronously telling us
2194 a connection is unreachable, and generate the appropriate
2195 event.
2196 - Changed timer routine type from unsigned long to uintptr_t
2197 to better fit with machine architectures.
2198 - ep.param data now initialized in ep_create, not ep_alloc.
2199 - Or Gerlitz provided updates to Mellanox files for evd_resize,
2200 fw attributes, many others. Also implemented changes for correct
2201 sizes on REP side of a connection request.
2202
2203
2204
2205 NEW SINCE Beta 2.0
2206
2207 * dat_echo now DAT 1.1 compliant. Various small enhancements.
2208
2209 * Revamped atomic_inc/dec to be void, the return value was never
2210 used. This allows kdapl to use Linux kernel equivalents, and
2211 is a small performance advantage.
2212
2213 * kDAPL: dapl_evd_modify_upcall implemented and tested.
2214
2215 * kDAPL: physical memory registration implemented and tested.
2216
2217 * uDAPL now builds cleanly for non-debug versions.
2218
2219 * Default RDMA credits increased to 8.
2220
2221 * Default ACK_TIMEOUT now a reasonable value (2 sec vs old 2
2222 months).
2223
2224 * Cleaned up dat_error.h, now 1.1 compliant in comments.
2225
2226 * evd_resize initial implementation. Untested.
2227
2228 * Bug fixes
2229 - __KDAPL__ is defined in kdat_config.h, so apps don't need
2230 to define it.
2231 - Changed include file ordering in kdat.h to put kdat_config.h
2232 first.
2233 - resolved connection/tear-down race on the client side.
2234 - kDAPL timeouts now scaled properly; fixed 3 orders of
2235 magnitude difference.
2236 - kDAPL EVD callbacks now get invoked for all completions; old
2237 code would drop them in heavy utilization.
2238 - Fixed error path in kDAPL evd creation, so we no longer
2239 leak CNOs.
2240 - create_psp_any returns correct error code if it can't create
2241 a connection qualifier.
2242 - lock fix in ibapi disconnect code.
2243 - kDAPL INFINITE waits now work properly (non connection
2244 waits)
2245 - kDAPL driver unload now works properly
2246 - dapl_lmr_[k]create now returns 1.1 error codes
2247 - ibapi routines now return DAT 1.1 error codes
2248
2249
2250
2251 NEW SINCE Beta 1.10
2252
2253 * kDAPL is now part of the DAPL distribution. See the release
2254 notes above.
2255
2256 The kDAPL 1.1 spec is now contained in the doc/ subdirectory.
2257
2258 * Several files have been moved around as part of the kDAPL
2259 checkin. Some files that were previously in udapl/ are now
2260 in common/, some in common are now in udapl/. The goal was
2261 to make sure files are properly located and make sense for
2262 the build.
2263
2264 * Source code formatting changes for consistency.
2265
2266 * Bug fixes
2267 - dapl_evd_create() was comparing the wrong bit combinations,
2268 allowing bogus EVDs to be created.
2269 - Removed code that swallowed zero length I/O requests, which
2270 are allowed by the spec and are useful to applications.
2271 - Locking in dapli_get_sp_ep was asymmetric; fixed it so the
2272 routine will take and release the lock. Cosmetic change.
2273 - dapl_get_consuemr_context() will now verify the pointer
2274 argument 'context' is not NULL.
2275
2276
2277 OBTAIN THE CODE
2278
2279 To obtain the tree for your local machine you can check it
2280 out of the source repository using CVS tools. CVS is common
2281 on Unix systems and available as freeware on Windows machines.
2282 The command to anonymously obtain the source code from
2283 Source Forge (with no password) is:
2284
2285 cvs -d:pserver:[email protected]:/cvsroot/dapl login
2286 cvs -z3 -d:pserver:[email protected]:/cvsroot/dapl co .
2287
2288 When prompted for a password, simply press the Enter key.
2289
2290 Source Forge also contains explicit directions on how to become
2291 a developer, as well as how to use different CVS commands. You may
2292 also browse the source code using the URL:
2293
2294 http://svn.sourceforge.net/viewvc/dapl/trunk/
2295
2296 SYSTEM REQUIREMENTS
2297
2298 This project has been implemented on Red Hat Linux 7.3, SuSE
2299 SLES 8, 9, and 10, Windows 2000, RHEL 3.0, 4.0 and 5.0 and a few
2300 other Linux distrubutions. The structure of the code is designed
2301 to allow other operating systems to easily be adapted.
2302
2303 The DAPL team has used Mellanox Tavor based InfiniBand HCAs for
2304 development, and continues with this platform. Our HCAs use the
2305 IB verbs API submitted by IBM. Mellanox has contributed an
2306 adapter layer using their VAPI verbs API. Either platform is
2307 available to any group considering DAPL work. The structure of
2308 the uDAPL source allows other provider API sets to be easily
2309 integrated.
2310
2311 The development team uses any one of three topologies: two HCAs
2312 in a single machine; a single HCA in each of two machines; and
2313 most commonly, a switch. Machines connected to a switch may have
2314 more than one HCA.
2315
2316 The DAPL Plugfest revealed that switches and HCAs available from
2317 most vendors will interoperate with little trouble, given the
2318 most recent releases of software. The dapl reference team makes
2319 no recommendation on HCA or switch vendors.
2320
2321 Explicit machine configurations are available upon request.
2322
2323 IN THE TREE
2324
2325 The DAPL tree contains source code for the uDAPL and kDAPL
2326 implementations, and also includes tests and documentation.
2327
2328 Included documentation has the base level API of the
2329 providers: OpenFabrics, IBM Access, and Mellanox Verbs API. Also
2330 included are a growing number of DAPL design documents which
2331 lead the reader through specific DAPL subsystems. More
2332 design documents are in progress and will appear in the tree in
2333 the near future.
2334
2335 A small number of test applications and a unit test framework
2336 are also included. dapltest is the primary testing application
2337 used by the DAPL team, it is capable of simulating a variety of
2338 loads and exercises a large number of interfaces. Full
2339 documentation is included for each of the tests.
2340
2341 Recently, the dapl conformance test has been added to the source
2342 repository. The test provides coverage of the most common
2343 interfaces, doing both positive and negative testing. Vendors
2344 providing DAPL implementation are strongly encouraged to run
2345 this set of tests.
2346
2347 MAKEFILE NOTES
2348
2349 There are a number #ifdef's in the code that were necessary
2350 during early development. They are disappearing as we
2351 have time to take advantage of features and work available from
2352 newer releases of provider software. These #ifdefs are not
2353 documented as the intent is to remove them as soon as possible.
2354
2355 CONTRIBUTIONS
2356
2357 As is common to Source Forge projects, there are a small number
2358 of developers directly associated with the source tree and having
2359 privileges to change the tree. Requested updates, changes, bug
2360 fixes, enhancements, or contributions should be sent to
2361 James Lentini at [email protected] for review. We welcome your
2362 contributions and expect the quality of the project will
2363 improve thanks to your help.
2364
2365 The core DAPL team is:
2366
2367 James Lentini
2368 Arlin Davis
2369 Steve Sears
2370
2371 ... with contributions from a number of excellent engineers in
2372 various companies contributing to the open source effort.
2373
2374
2375 ONGOING WORK
2376
2377 Not all of the DAPL spec is implemented at this time.
2378 Functionality such as shared memory will probably not be
2379 implemented by the reference implementation (there is a write up
2380 on this in the doc/ area), and there are yet various cases where
2381 work remains to be done. And of course, not all of the
2382 implemented functionality has been tested yet. The DAPL team
2383 continues to develop and test the tree with the intent of
2384 completing the specification and delivering a robust and useful
2385 implementation.
2386
2387
2388 The DAPL Team
2389