Debian Codegraph @

doc/uDAPL_release_notes.txt

raw
1 README/Release Notes
2 OFED 4.8 DAPL Release 2.1.10
3 Dec 2016
4
5 User space libraries/utilities for Direct Access Transport (DAT) v2.0. DAT is
6 a transport-independent, platform-independent Application Programming
7 Interface that supports RDMA (remote direct memory access) devices.
8 Note: v1.2 is no longer supported and will not be included with OFED releases
9
10 MIC support is provided with the new MCM provider and MPXYD service, since dapl-2.1.0.
11 MCM requires the Intel(R) MPSS 3.x (YOCTO) release for Linux to be installed on your system.
12 MPSS 3.x for Linux can be downloaded from: http://software.intel.com/mic-developer
13
14 For latest documentation and packages: //www.openfabrics.org/downloads/dapl/
15
16 =================
17 1.0 Release Notes
18 =================
19 dapl-2.1.10 changes include bug fixes, CCL Proxy post_send optimizations, dtest/dtest_suite improvements
20
21 dapl-2.1.9-2 changes include bug fixes, MFO fixes, GID query attributes
22 dapl-2.1.9-1 changes include dtestcm improvement, bug fixes for MIC, Fix SCM interop issue with MTU
23
24 dapl-2.1.8 changes include dtest improvement, adding Intel OPA support, and MTU negotiation
25 - Add dtest -D option for data check/validation via pingpong test
26 - Add dtest -q option for open/query only option
27 - Add support for new hfi (Intel OPA) driver, all providers
28 CCL Proxy can support hfi in MFO mode via mcm provider
29 - Active MTU is now set as default QP MTU, CM mechanism added to negotiate
30 and fallback to smaller size of endpoints. Is backward compatible with
31 older providers running defaults of 2K and can be adjusted with
32 pre-existing DAPL_IB_MTU environment setting.
33 - Bug fixes, including CCL Data corruption (scif_writeto ordering requirements).
34 - Provide new test/dtest/dtest_suite.sh script for automated host to mic test/validation.
35
36 dapl-2.1.7 changes include dtest improvement and CCL Proxy P2P inline support:
37
38 - Add dtest -W option for rdma write pinpong test,
39 new options with -W include -a (all data sizes) -i (incremental size)
40 - CCL Proxy small message latency improvment with Proxy2Proxy inline support
41 for message sizes < 96 bytes, reduces MPI pingpong single byte latency
42 for MFO devices by 27%.
43
44 dapl-2.1.6 changes include MIC support for full offload mode
45
46 - Add support for Truescale qib devices with no CCL Direct verbs support on MIC.
47 - Enhancement for inside the box transfers without IB adapter via ibscif.
48 - Add DAPL_NETWORK_NODES, DAPL_NETWORK_PPN environment variables.
49
50 dapl-2.1.5 changes include improvements for large scale UD communication management:
51
52 - AH caching, reduced memory footprint (grows as needed)
53 - Port space increased to 24 bits
54 - Hash table for port space, CM object management
55 - Optimized CM wire protocol for fast index lookup
56
57 Tested on 1200n 28ppn cluster, AlltoAll Intel MPI, UD mode.
58 Both static and dynamic modes, over 500m UD QP connections.
59
60 ==========
61 2.0 BUILD:
62 ==========
63
64 # NON_DEBUG build/install example for x86_64, OFED targets
65 ./configure --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include"
66 make install
67
68 # DEBUG build/install example for x86_64, using OFED targets
69 ./configure --enable-debug --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include"
70 make install
71
72 # COUNTERS build/install example for x86_64, using OFED targets
73 ./configure --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include -DDAPL_COUNTERS"
74 make install
75
76 =========================================================
77 3.0 Provider descriptions and CM results (cma, scm, ucm):
78 =========================================================
79
80 1. CMA - uses OFA rdma_cm to setup QP's. IPoIB, ARP, and SA queries required.
81
82 Provider name: ofa-v2-ib0
83 PROs: OFA rdma_cm has the most testing across many applications.
84 Supports both iWARP and IB.
85
86 CONs: Serialization of conn processing with kernel based CM service
87 Requires IPoIB ARP for name resolution, storms
88 Requires SA for path record queries for IB fabrics.
89 Conn Request private data limited to 52 bytes.
90
91 Settings for larger clusters (512+ cores):
92
93 setenv DAPL_CM_ROUTE_TIMEOUT_MS 20000
94 setenv DAPL_CM_ARP_TIMEOUT_MS 10000
95
96 2. SCM - uses sockets to exchange QP information. IPoIB, ARP, and SA queries NOT required.
97
98 Provider name (connectx): ofa-v2-mlx4_0-1
99 PROs: Each rank has own instance of socket cm. More private data with requests.
100 Doesn't require path-record lookup.
101
102 CONs: Socket resources grow with scale-out, serialization of
103 connections with kernel based tcp sockets,
104 Competes for MPI socket resources/port space and other TCP applications.
105 Sockets remain in TIMEWAIT state for minutes after closure.
106 Requires ARP for name resolution.
107 Doesn't support iWARP devices.
108
109 Settings for larger clusters (512+ cores):
110
111 setenv DAPL_ACK_RETRY 7 /* IB RC Ack retry count */
112 setenv DAPL_ACK_TIMER 20 /* IB RC Ack retry timer */
113
114 3. UCM - use's IB UD QP to exchange QP info. Sockets, ARP, IPoIB, and SA queries NOT required.
115
116 Provider name (connectx): ofa-v2-mlx4_0-1u
117 PROs: Each rank has own instance of CM in user process
118 Resources fixed per rank regardless of scale-out size
119 No serialization of user or kernel resources establishing connections,
120 Simple 3-way msg handsake, CM messages fit in inline data for lowest message latency,
121 Supports alternate paths
122 No address resolution required.
123 No path resolution required.
124
125 CONs: New provider with limited testing, a little tougher to debug.
126 Doesn't support iWARP
127
128 Settings for larger clusters (512+ cores):
129
130 setenv DAPL_UCM_REP_TIME 10000 /* REQUEST timer, waiting for REPLY in millisecs */
131 setenv DAPL_UCM_RTU_TIME 10000 /* REPLY timer, waiting for RTU in millisecs */
132 setenv DAPL_UCM_CQ_SIZE 2000 /* CM completion queue */
133 setenv DAPL_UCM_QP_SIZE 2000 /* CM message queue */
134 setenv DAPL_UCM_RETRY 7 /* REQUEST and REPLY retries */
135 setenv DAPL_ACK_RETRY 7 /* IB RC Ack retry count */
136 setenv DAPL_ACK_TIMER 20 /* IB RC Ack retry timer */
137
138 CM Performance: CPS profile for cma, scm, and ucm v2 uDAPL providers:
139 -----------------------------------------------------------------------
140 Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz (IVT)
141 Mellanox MLX4 IB FDR, no switch.
142
143 dtestcm (server/client):
144
145 cma: Connections: 313.10 usec, CPS 3193.83 Total 0.31 secs, poll_cnt=6300, Num=1000
146 scm: Connections: 167.65 usec, CPS 5964.92 Total 0.17 secs, poll_cnt=2394, Num=1000
147 ucm: Connections: 71.85 usec, CPS 13918.06 Total 0.07 secs, poll_cnt=2360, Num=1000
148
149 dapl_cm_bw: MPI uDAPL/CM profiling application (all-to-all connections, all ranks)
150
151 CMA
152 2 Connect times (10): Total 0.0049 per 0.0005 CPS=2051.38
153 4 Connect times (40): Total 0.0151 per 0.0004 CPS=2650.16
154 8 Connect times (240): Total 0.0548 per 0.0002 CPS=4380.59
155 16 Connect times (1120): Total 4.0356 per 0.0036 CPS=277.53
156 32 Connect times (4800): Total 4.4704 per 0.0009 CPS=1073.72
157
158 SCM
159 2 Connect times (10): Total 0.0029 per 0.0003 CPS=3441.31
160 4 Connect times (40): Total 0.0060 per 0.0002 CPS=6635.97
161 8 Connect times (240): Total 0.0194 per 0.0001 CPS=12383.47
162 16 Connect times (1120): Total 0.0649 per 0.0001 CPS=17246.93
163 32 Connect times (4800): Total 1.0193 per 0.0002 CPS=4708.95
164
165 UCM
166 2 Connect times (10): Total 0.0014 per 0.0001 CPS=6993.91
167 4 Connect times (40): Total 0.0045 per 0.0001 CPS=8837.87
168 8 Connect times (240): Total 0.0155 per 0.0001 CPS=15477.13
169 16 Connect times (1120): Total 0.0630 per 0.0001 CPS=17765.12
170 32 Connect times (4800): Total 0.2632 per 0.0001 CPS=18236.54
171
172 ===================================================================================================
173 4.0 BKM for installing new DAPL library on your cluster without any impact on existing OFED install:
174 ====================================================================================================
175
176 Note: example for user /home/user1, (assumes /home/user1 is exported) and MLX4 adapter, port 1
177
178 Download latest 2.1.x package: http://www.openfabrics.org/downloads/dapl/dapl-2.1.x.tar.gz
179
180 untar in /home/user1
181 cd /home/user1/dapl-2.1.x
182 ./configure LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include"
183 make
184
185 Create /home/user1/dat.conf with following 3 lines. (entries with path to new libraries):
186
187 ofa-v2-mlx4_0-1u u2.0 nonthreadsafe default /home/user1/dapl-2.1.x/dapl/udapl/.libs/libdaploucm.so.2 dapl.2.0 "mlx4_0 1" ""
188 ofa-v2-mlx4_0-1m u2.0 nonthreadsafe default /home/user1/dapl-2.1.x/dapl/udapl/.libs/libdaplomcm.so.2 dapl.2.0 "mlx4_0 1" ""
189 ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default /home/user1/dapl-2.1.x/dapl/udapl/.libs/libdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
190 ofa-v2-ib0 u2.0 nonthreadsafe default /home/user1/dapl-2.1.x/dapl/udapl/.libs/libdaplcma.so.1 dapl.2.0 "ib0 0" ""
191
192 Run uDAPL application or Intel MPI that uses uDAPL, with (assuming mlx4_0 adapters) following:
193
194 setenv DAT_OVERRIDE=/home/user1/dat.conf
195 setenv LD_LIBRARY_PATH=/home/user1/dapl-2.1.5/dapl/udapl/.libs:$LD_LIBRARY_PATH
196
197 If running Intel MPI and uDAPL IB UD cm, set the following (recommended):
198
199 setenv I_MPI_DAPL_PROVIDER=ofa-v2-mlx4_0-1u
200
201 If running Intel MPI and uDAPL IB mcm with MIC, set the following:
202
203 setenv I_MPI_DAPL_PROVIDER=ofa-v2-mlx4_0-1m
204
205 If running Intel MPI and uDAPL socket cm, set the following:
206
207 setenv I_MPI_DAPL_PROVIDER=ofa-v2-mlx4_0-1
208
209
210 If running Intel MPI and uDAPL rdma_cm, set the following:
211
212 setenv I_MPI_DAPL_PROVIDER=ofa-v2-ib0
213
214
215 ============================================================
216 5.0 MCM Provider, MPXYD Daemon (CCL-proxy) Build and Install
217 ============================================================
218
219 MCM is a new uDAPL provider that is an extension to standard DAT 2.0 libraries. The purpose of this service
220 is to proxy RDMA writes from the MIC to the HOST to improve large IO performance. The provider will support
221 MIC to MIC, HOST to HOST, and MIC to HOST environments. The mcm client will NOT use MPXYD when running on the host.
222 It requires a new MPXYD daemon service when clients are running on a MIC KNC adapter. This package installs all the
223 host side libraries and daemon service. The MIC libraries must be built and moved over to MIC adapter. This verion
224 is currently included with MPSS and all libraries and services will be installed by default.
225
226 Current release package: dapl-2.1.9.tar.gz
227
228 * Sample host build from source package (ofed must be installed)
229
230 ./configure --enable-mcm --prefix=/usr --libdir=/usr/lib64 --sysconfdir=/etc
231 make
232 make install
233
234 * Sample host rpmbuild/update from release tarball, /root:
235
236 rpmbuild -ta dapl-2.1.9.tar.gz
237 rpm -U /root/rpmbuild/RPMS/x86_64/dapl*
238
239 * Sample MIC build from source package for MPSS 3.x KNC (MPSS must be installed)
240 * Assume /opt is nfs mounted across cluster
241
242 source /opt/mpss/3.x/environment-setup-k1om-mpss-linux
243 ./configure --enable-mcm --prefix /opt/dapl/mic --host=x86_64-k1om-linux
244 make
245 make install
246
247 copy /opt/dapl/mic/* files out to all MIC cards
248
249 * Cluster deployment
250
251 (1) Build once on the head or on one of the nodes (with MPSS) as described in the above steps.
252
253 (2) HOST: Install dapl libraries and mpxyd service, "rpm -U" all dapl RPM files on host nodes:
254
255 (3) MIC: Setup dapl overlay for new package (/opt/intel/dapl):
256
257 Create /etc/mpss/conf.d/dapl.conf with following entry:
258
259 Overlay Filelist /opt/dapl /opt/dapl/dapl.filelist on
260
261 Create /opt/dapl/dapl.filelist with following entries:
262
263 file /etc/dat.conf mic/etc/dat.conf 755 0 0
264 file /usr/bin/dtest mic/bin/dtest 755 0 0
265 file /usr/bin/dtestx mic/bin/dtestx 755 0 0
266 file /usr/bin/dtestcm mic/bin/dtestcm 755 0 0
267 file /usr/bin/dapltest mic/bin/dapltest 755 0 0
268 file /usr/lib64/libdat.so.2.0.0 mic/lib/libdat.so.2.0.0 755 0 0
269 file /usr/lib64/libdaplofa.so.2.0.0 mic/lib/libdaplofa.so.2.0.0 755 0 0
270 file /usr/lib64/libdaplomcm.so.2.0.0 mic/lib/libdaplomcm.so.2.0.0 755 0 0
271 file /usr/lib64/libdaploscm.so.2.0.0 mic/lib/libdaploscm.so.2.0.0 755 0 0
272 file /usr/lib64/libdaploucm.so.2.0.0 mic/lib/libdaploucm.so.2.0.0 755 0 0
273
274 slink /usr/lib64/libdat.so libdat.so.2.0.0 777 0 0
275 slink /usr/lib64/libdat.so.2 libdat.so.2.0.0 777 0 0
276 slink /usr/lib64/libdaplofa.so libdaplofa.so.2.0.0 777 0 0
277 slink /usr/lib64/libdaplofa.so.2 libdaplofa.so.2.0.0 777 0 0
278 slink /usr/lib64/libdaplomcm.so libdaplomcm.so.2.0.0 777 0 0
279 slink /usr/lib64/libdaplomcm.so.2 libdaplomcm.so.2.0.0 777 0 0
280 slink /usr/lib64/libdaploscm.so libdaploscm.so.2.0.0 777 0 0
281 slink /usr/lib64/libdaploscm.so.2 libdaploscm.so.2.0.0 777 0 0
282 slink /usr/lib64/libdaploucm.so libdaploucm.so.2.0.0 777 0 0
283 slink /usr/lib64/libdaploucm.so.2 libdaploucm.so.2.0.0 777 0 0
284
285 Reboot or restart MPSS and ofed-mic services
286
287 Check for dapl overlay
288 micctrl --config
289
290 * Setup for non-root CCL Proxy testing, MPXYD running as process with different service port from your /home directory:
291
292 Using build instructions above, change prefix as follow and "make install":
293
294 Build MIC:
295 --prefix=/home/username/ccl-proxy-mic
296
297 Build host:
298 --prefix=/home/username/ccl-proxy-host
299
300 edit /home/username/ccl-proxy-host/etc/mpxyd.conf and change the following entries:
301
302 log_file /var/log/mpxyd.log to log_file /tmp/username/mpxyd.log
303 lock_file /var/log/mpxyd.pid to lock_file /tmp/username/mpxyd.log
304 scif_port_id 68 to scif_port_id 1068
305
306 start the mpxyd process on each node
307
308 ssh node1-hostname /home/username/ccl-proxy-host/sbin/mpxyd -P -O /home/username/ccl-proxy-host/etc/mpxyd.conf&
309
310 Note: override default port id using following environment variable:
311
312 export DAPL_MCM_PORT_ID=1068
313
314 * Notes
315
316 (1) Modify "/etc/mpxyd.conf" to change the settings for the proxy. Especially, try different values
317 of "buffer_segment_size" for performance tuning. Use a smaller value for "buffer_pool_mb"
318 to reduce the memory foorprint of mpxyd. Use a larger value for "scif_listen_qlen" to run
319 more MPI ranks per card. Also modify mcm_affinity_base to the desired CPU_id to insure
320 socket to adapter affinity. Best performance when HCA, MIC, and CPU are on same socket.
321 Default settings are on CPU socket 0.
322
323 (2) By default, only writes originated from MIC is proxied. However, it is also possible to proxy
324 host-originated writes (e.g. for debugging purpose). To do this, set the environment variable
325 "DAPL_MCM_ALWAYS_PROXY=1". This variable applies to the provider, not the proxy.
326
327 (3) Use the MCM provider with Intel MPI 5.1 or greater for best out of box experiences with MIC.
328
329 Recommended settings:
330
331 export I_MPI_MIC=1
332 export I_MPI_DEBUG=2
333 export I_MPI_FALLBACK=0
334
335 (4) Server Platform Notes:
336
337 For optimal CCL Proxy performance, installed on Intel® Xeon® Processor E5-26xx v2 and v3 platforms,
338 please update to the latest platform BIOS and set the following Processor, Power, and Memory settings.
339
340 https://downloadcenter.intel.com/default.aspx
341
342 [BIOS::Advanced::Processor Configuration]
343 Intel(R) QPI Frequency Select=Auto Max
344 Intel(R) Turbo Boost Technology=Enabled
345 Enhanced Intel SpeedStep(R) Tech=Enabled
346 Processor C3=Disabled
347 Processor C6=Enabled
348 Intel(R) Virtualization Technology=Disabled
349 Intel(R) VT for Directed I/O=Disabled
350 Direct Cache Access (DCA)=Enabled
351 Extended ATR=0x01
352
353 [BIOS::Advanced::Power & Performance]
354 CPU Power and Performance Policy=Performance
355
356 [BIOS::Advanced::Memory Configuration::Memory RAS and Performance Configuration]
357 Select Memory RAS Configuration=Maximum Performance
358 NUMA Optimized=Enabled
359
360 ====================================================================
361 6.0 Mellanox Fabric Collective Accelerator (FCA) build for Intel MPI
362 ====================================================================
363
364 1) Download latest package: http://www.openfabrics.org/downloads/dapl/dapl-2.x.x.tar.gz
365
366 2) Build/Install with collectives configured for fca type:
367
368 ./configure --enable-coll-type=fca --disable-mcm --prefix=/usr
369 LDFLAGS=-L/opt/mellanox/fca/lib
370 CPPFLAGS=-I/opt/mellanox/fca/include
371 make install
372
373 If an uDAPL provider with collective support is installed as an additional package,
374 while another uDAPL provider also exists in a system, LD_LIBRARY_PATH environment
375 variable should be updated to include the path to the newly installed DAPL package.
376
377 3) Set following MPI variables:
378
379 export FCA_MGR_HOME=/opt/mellanox/fca
380 export FCA_HOME=/opt/mellanox/fca
381 export I_MPI_DAPL_COLLECTIVES=all
382 export I_MPI_FABRICS=dapl
383 export I_MPI_FALLBACK_DEVICE=off
384
385 =============================
386 7.0 Environment Variables
387 =============================
388
389 - IB UD options using UCM provider, large scale settings (Xeon)
390
391 export DAPL_NETWORK_NODES= /* set to active nodes on network for CM */
392 export DAPL_NETWORK_PPN= /* set to active processes per node for CM */
393
394 /* The following will be adjusted by provider based on NODES, PPN */
395 export DAPL_UCM_REP_TIME=8000 /* REQUEST timer, waiting on REPLY, msecs, default = 800 */
396 export DAPL_UCM_RTU_TIME=8000 /* REPLY timer, waiting for RTU in msecs, default=400 */
397 export DAPL_UCM_RETRY=7 /* REQUEST & REPLY retries, default = 7 */
398 export DAPL_UCM_QP_SIZE=4000 /* CM req/reply work queue size, default = 500 entries */
399 export DAPL_UCM_CQ_SIZE=4000 /* CM req/reply completion queue size, default = 500 entries */
400 export DAPL_UCM_TX_BURST=100 /* CM signal rate on send messages */
401 export DAPL_UCM_ENTRY_BITS=11 /* default = 11-bit, 2KB entries, allocation blocks */;
402 export DAPL_UCM_ARRAY_BITS=18 /* default = 18 bit, 256KB total */
403
404 - IB RC options using SCM provider
405
406 export DAPL_SCM_NETDEV=ib0 /* default is first non-loopback netdev */
407
408 - Other IB settings for all providers:
409
410 export DAPL_MAX_INLINE=64 /* IB RC inline optimization, best small msg latency, def=64 */
411 export DAPL_ACK_RETRY=7 /* IB RC Ack retry count, default 7 */
412 export DAPL_ACK_TIMER=20 /* IB RC Ack retry timer, 5 bits, 4.096us*2^ack_timer. 16== 268ms, 20==4.2s */
413 export DAPL_IB_MTU=2048 /* IB MTU size, default = 2048 */
414 export DAPL_RNR_TIMER=12 /* 5 bits, 12 =.64ms, 28 =163ms, 31 =491ms */
415 export DAPL_RNR_RETRY=7 /* 3 bits, 7 == infinite */
416 export DAPL_IB_PKEY= 0 /* override IB partition key, default is pkey index 0 */
417 export DAPL_IB_SL=0 /* override IB Sevice level, default = 0 */
418
419 - Other options:
420 export DAPL_WR_MAX=500 /* used to reduce max qp depth on all IB providers, default = dev attributes */
421
422 Debug logging and Counter settings ( --enable-counters)
423
424 export DAPL_DBG_SYS_MEM=10 /* threshold for low sys memory warning, def = 10 percent */
425 export DAPL_DBG_TYPE=0x0000003 /* set log, monitor, and error checking, default = warnings and errors */
426
427 DAPL_DBG_TYPE bit settings as follow:
428
429 DAPL_DBG_TYPE_ERR = 0x0001,
430 DAPL_DBG_TYPE_WARN = 0x0002,
431 DAPL_DBG_TYPE_EVD = 0x0004,
432 DAPL_DBG_TYPE_CM = 0x0008,
433 DAPL_DBG_TYPE_EP = 0x0010,
434 DAPL_DBG_TYPE_UTIL = 0x0020,
435 DAPL_DBG_TYPE_CALLBACK = 0x0040,
436 DAPL_DBG_TYPE_DTO_COMP_ERR = 0x0080,
437 DAPL_DBG_TYPE_API = 0x0100,
438 DAPL_DBG_TYPE_RTN = 0x0200,
439 DAPL_DBG_TYPE_EXCEPTION = 0x0400,
440 DAPL_DBG_TYPE_SRQ = 0x0800,
441 DAPL_DBG_TYPE_CNTR = 0x1000,
442 DAPL_DBG_TYPE_CM_LIST = 0x2000,
443 DAPL_DBG_TYPE_THREAD = 0x4000,
444 DAPL_DBG_TYPE_CM_EST = 0x8000,
445 DAPL_DBG_TYPE_CM_WARN = 0x10000,
446 DAPL_DBG_TYPE_EXTENSION = 0x20000,
447 DAPL_DBG_TYPE_CM_STATS = 0x40000,
448 DAPL_DBG_TYPE_CM_ERRS = 0x80000, /* print any cm errors on device close */
449 DAPL_DBG_TYPE_LINK_ERRS = 0x100000, /* print any link errors on device close */
450 DAPL_DBG_TYPE_LINK_WARN = 0x200000, /* print any link warning on device close */
451 DAPL_DBG_TYPE_DIAG_ERRS = 0x400000, /* print any diag_counter errors on dev close */
452 DAPL_DBG_TYPE_SYS_WARN = 0x800000, /* print low mem warning during alloc, reg_mem */
453 DAPL_DBG_TYPE_VER = 0x1000000, /* print dapl ver and build date during dev open */
454
455 =============================
456 9.0 SAMPLE uDAPL APPLICATION:
457 =============================
458
459 There are 2 sample programs, with manpages, provided with this package.
460
461 (dapl/test/dtest/)
462
463 NAME
464 dtest - simple uDAPL send/receive and RDMA test
465
466 SYNOPSIS
467 dtest [-P provider] [-b buf size] [-B burst count][-v] [-c] [-p] [-d] [-s]
468
469 dtest [-P provider] [-b buf size] [-B burst count][-v] [-c] [-p] [-d] [-h HOSTNAME]
470
471 DESCRIPTION
472 dtest is a simple test used to exercise and verify the uDAPL interfaces. At least two instantia-
473 tions of the test must be run. One acts as the server and the other the client. The server side of
474 the test, once invoked listens for connection requests, until timing out or killed. Upon receipt
475 of a cd connection request, the connection is established, the server and client sides exchange
476 information necessary to perform RDMA writes and reads.
477
478 OPTIONS
479 -P=PROVIDER
480 use PROVIDER to specify uDAPL interface using /etc/dat.conf (default OpenIB-cma)
481
482 -b=BUFFER_SIZE
483 use buffer size BUFFER_SIZE for RDMA(default 64)
484
485 -B=BURST_COUNT
486 use busrt count BURST_COUNT for interations (default 10)
487
488 -v, verbose output(default off)
489
490 -c, use consumer notification events (default off)
491
492 -p, use polling (default wait for event)
493
494 -d, delay in seconds before close (default off)
495
496 -s, run as server (default - run as server)
497
498 -h=HOSTNAME
499 use HOSTNAME to specify server hostname or IP address (default - none)
500
501 EXAMPLES
502 dtest -P OpenIB-cma -v -s
503 Starts a server process with debug verbosity using provider OpenIB-cma.
504
505 dtest -P OpenIB-cma -h server1-ib0
506
507 Starts a client process, using OpenIB-cma provider to connect to hostname server1-ib0.
508
509 SEE ALSO
510 dapltest(1)
511
512 AUTHORS
513 Arlin Davis
514 <[email protected]>
515
516 BUGS
517
518 /dapl/test/dapltest/
519
520 NAME
521 dapltest - test for the Direct Access Programming Library (DAPL)
522
523 DESCRIPTION
524 Dapltest is a set of tests developed to exercise, characterize, and verify the DAPL interfaces
525 during development and porting. At least two instantiations of the test must be run. One acts as
526 the server, fielding requests and spawning server-side test threads as needed. Other client invo-
527 cations connect to the server and issue test requests. The server side of the test, once invoked,
528 listens continuously for client connection requests, until quit or killed. Upon receipt of a con-
529 nection request, the connection is established, the server and client sides swap version numbers
530 to verify that they are able to communicate, and the client sends the test request to the server.
531 If the version numbers match, and the test request is well-formed, the server spawns the threads
532 needed to run the test before awaiting further connections.
533
534 USAGE
535 dapltest [ -f script_file_name ] [ -T S|Q|T|P|L ] [ -D device_name ] [ -d ] [ -R HT|LL|EC|PM|BE ]
536
537 With no arguments, dapltest runs as a server using default values, and loops accepting requests
538 from clients.
539
540 The -f option allows all arguments to be placed in a file, to ease test automation.
541
542 The following arguments are common to all tests:
543
544 [ -T S|Q|T|P|L ]
545 Test function to be performed:
546
547 S - server loop
548
549 Q - quit, client requests that server wait for any outstanding tests to complete, then
550 clean up and exit
551
552 T - transaction test, transfers data between client and server
553
554 P - performance test, times DTO operations
555
556 L - limit test, exhausts various resources, runs in client w/o server interaction
557 Default: S
558
559 [ -D device_name ]
560 Specifies the interface adapter name as documented in the /etc/dat.conf static configura-
561 tion file. This name corresponds to the provider library to open. Default: none
562
563 [ -d ] Enables extra debug verbosity, primarily tracing of the various DAPL operations as they
564 progress. Repeating this parameter increases debug spew. Errors encountered result in the
565 test spewing some explanatory text and stopping; this flag provides more detail about what
566 lead up to the error. Default: zero
567
568 [ -R BE ]
569 Indicate the quality of service (QoS) desired. Choices are:
570
571 HT - high throughput
572
573 LL - low latency
574
575 EC - economy (neither HT nor LL)
576
577 PM - premium
578
579 BE - best effort Default: BE
580
581 Usage - Quit test client
582
583 dapltest [Common_Args] [ -s server_name ]
584
585 Quit testing (-T Q) connects to the server to ask it to clean up and
586 exit (after it waits for any outstanding test runs to complete).
587 In addition to being more polite than simply killing the server,
588 this test exercises the DAPL object teardown code paths.
589 There is only one argument other than those supported by all tests:
590
591 -s server_name Specifies the name of the server interface.
592 No default.
593
594 Usage - Transaction test client
595
596 dapltest [Common_Args] [ -s server_name ]
597 [ -t threads ] [ -w endpoints ] [ -i iterations ] [ -Q ]
598 [ -V ] [ -P ] OPclient OPserver [ op3,
599
600 Transaction testing (-T T) transfers a variable amount of data between
601 client and server. The data transfer can be described as a sequence of
602 individual operations; that entire sequence is transferred ’iterations’
603 times by each thread over all of its endpoint(s).
604
605 The following parameters determine the behavior of the transaction test:
606
607 -s server_name Specifies the name or IP address of the server interface.
608 No default.
609
610 [ -t threads ] Specify the number of threads to be used.
611 Default: 1
612
613 [ -w endpoints ] Specify the number of connected endpoints per thread.
614 Default: 1
615
616 [ -i iterations ] Specify the number of times the entire sequence
617 of data transfers will be made over each endpoint.
618 Default: 1000
619
620 [ -Q ] Funnel completion events into a CNO.
621 Default: use EVDs
622
623 [ -V ] Validate the data being transferred.
624 Default: ignore the data
625
626 [ -P ] Turn on DTO completion polling
627 Default: off
628
629 OP1 OP2 [ OP3, ... ]
630 A single transaction (OPx) consists of:
631
632 server|client Indicates who initiates the
633 data transfer.
634
635 SR|RR|RW Indicates the type of transfer:
636 SR send/recv
637 RR RDMA read
638 RW RDMA write
639 Defaults: none
640
641 [ seg_size [ num_segs ] ]
642 :
643
644 Indicates the amount and format
645 of the data to be transferred.
646 Default: 4096 1
647 (i.e., 1 4KB buffer)
648
649 [ -f ] For SR transfers only, indicates
650 that a client’s send transfer
651 completion should be reaped when
652 the next recv completion is reaped.
653 Sends and receives must be paired
654 (one client, one server, and in that
655 order) for this option to be used.
656 Restrictions:
657
658 Due to the flow control algorithm used by the transaction test, there
659 must be at least one SR OP for both the client and the server.
660
661 Requesting data validation (-V) causes the test to automatically append
662 three OPs to those specified. These additional operations provide
663 synchronization points during each iteration, at which all user-specified
664 transaction buffers are checked. These three appended operations satisfy
665 the "one SR in each direction" requirement.
666
667 The transaction OP list is printed out if -d is supplied.
668
669 Usage - Performance test client
670
671 dapltest [Common_Args] -s server_name [ -m p|b ]
672 [ -i iterations ] [ -p pipeline ] OP
673
674 Performance testing (-T P) times the transfer of an operation.
675 The operation is posted ’iterations’ times.
676
677 The following parameters determine the behavior of the transaction test:
678
679 -s server_name Specifies the name or IP address of the server interface.
680 No default.
681
682 -m b|p Used to choose either blocking (b) or polling (p)
683 Default: blocking (b)
684 [ -i iterations ] Specify the number of times the entire sequence
685 of data transfers will be made over each endpoint.
686 Default: 1000
687
688 [ -p pipeline ] Specify the pipline length, valid arguments are in
689 the range [0,MAX_SEND_DTOS]. If a value greater than
690 MAX_SEND_DTOS is requested the value will be
691 adjusted down to MAX_SEND_DTOS.
692 Default: MAX_SEND_DTOS
693
694 OP Specifies the operation as follow:
695
696 RR|RW Indicates the type of transfer:
697 RR RDMA read
698 RW RDMA write
699 Defaults: none
700
701 [ seg_size [ num_segs ] ]
702 Indicates the amount and format
703 of the data to be transferred.
704 Default: 4096 1
705 (i.e., 1 4KB buffer)
706 Usage - Limit test client
707
708 Limit testing (-T L) neither requires nor connects to any server
709 instance. The client runs one or more tests which attempt to
710 exhaust various resources to determine DAPL limits and exercise
711 DAPL error paths. If no arguments are given, all tests are run.
712
713 Limit testing creates the sequence of DAT objects needed to
714 move data back and forth, attempting to find the limits supported
715 for the DAPL object requested. For example, if the LMR creation
716 limit is being examined, the test will create a set of
717 {IA, PZ, CNO, EVD, EP} before trying to run dat_lmr_create() to
718 failure using that set of DAPL objects. The ’width’ parameter
719 can be used to control how many of these parallel DAPL object
720 sets are created before beating upon the requested constructor.
721 Use of -m limits the number of dat_*_create() calls that will
722 be attempted, which can be helpful if the DAPL in use supports
723 essentailly unlimited numbers of some objects.
724 The limit test arguments are:
725
726 [ -m maximum ] Specify the maximum number of dapl_*_create()
727 attempts.
728 Default: run to object creation failure
729
730 [ -w width ] Specify the number of DAPL object sets to
731 create while initializing.
732 Default: 1
733
734 [ limit_ia ] Attempt to exhaust dat_ia_open()
735
736 [ limit_pz ] Attempt to exhaust dat_pz_create()
737
738 [ limit_cno ] Attempt to exhaust dat_cno_create()
739
740 [ limit_evd ] Attempt to exhaust dat_evd_create()
741
742 [ limit_ep ] Attempt to exhaust dat_ep_create()
743
744 [ limit_rsp ] Attempt to exhaust dat_rsp_create()
745
746 [ limit_psp ] Attempt to exhaust dat_psp_create()
747
748 [ limit_lmr ] Attempt to exhaust dat_lmr_create(4KB)
749
750 [ limit_rpost ] Attempt to exhaust dat_ep_post_recv(4KB)
751
752 [ limit_size_lmr ] Probe maximum size dat_lmr_create()
753
754 Default: run all tests
755 EXAMPLES
756 dapltest -T S -d -D OpenIB-cma
757
758 Starts a server process with debug verbosity.
759
760 dapltest -T T -d -s host1-ib0 -D OpenIB-cma -i 100 client SR 4096 2 server SR 4096 2
761
762 Runs a transaction test, with both sides
763 sending one buffer with two 4KB segments,
764 one hundred times.
765
766 dapltest -T P -d -s host1-ib0 -D OpenIB-cma -i 100 SR 4096 2
767
768 Runs a performance test, with the client
769 sending one buffer with two 4KB segments,
770 one hundred times.
771
772 dapltest -T Q -s host1-ib0 -D OpenIB-cma
773
774 Asks the server to clean up and exit.
775
776 dapltest -T L -D OpenIB-cma -d -w 16 -m 1000
777
778 Runs all of the limit tests, setting up
779 16 complete sets of DAPL objects, and
780 creating at most a thousand instances
781 when trying to exhaust resources.
782
783 dapltest -T T -V -d -t 2 -w 4 -i 55555 -s linux3 -D OpenIB-cma client RW 4096 1 server RW 2048 4
784 client SR 1024 4 server SR 4096 2 client SR 1024 3 -f server SR 2048 1 -f
785
786 Runs a more complicated transaction test,
787 with two thread using four EPs each,
788 sending a more complicated buffer pattern
789 for a larger number of iterations,
790 validating the data received.
791
792 =============================
793 9.0 Summary of Fixes/Changes:
794 =============================
795
796 Release 2.1.7 (OFED 3.18-1 GA)
797 dtest: add -a -i options, all data sizes, incremental size
798 dapl: Fix segfault while freeing qp
799 mpxyd: add P2P inline support for data size <= 96 bytes
800 dtest: change rdma_write_ping_pong so client is always last receiver
801 ucm: add DAPL_NETWORK_PROCESS_NUM option for total ranks
802 ucm: fca create group incorrectly using IB addr instead of socket address.
803 ucm: fca_comm_destroy called with NULL
804 dtest: add -W option for rdma write pinpong, similiar to ib_write_lat
805 docs: update release notes for collective build
806 mpxyd: reduce log level for rcv message flush
807 dapltest: dapltest with no argument not working in ppc64 arch
808
809 Release 2.1.6 (OFED 3.18-1)
810 ucm: add cluster size environments to adjust CM timers
811 mpxyd: proxy_in data transfers can improperly start before RTU received
812 mcm: forward open/query for MFO devices in query only mode
813 mpxyd: byte swap incorrect on WRC wr_len
814 dtest: remove ERR message from flush QP function
815 dapltest: Quit command with "-n port" number will core dump
816 config: update dat.conf for MFO qib devices, 2 adapters/ports
817 mpxyd: add MFO support on proxy side
818 mcm: add MFO proxy commands, device, and CM support
819 mcm: add MFO support to openib_common code base
820 mcm: add full offload (MFO) mode to provider to support qib on MIC
821 dtest: pre-allocated buffer too small for RMR, DTO ops timeout
822 mpxyd: fix buffer initialization when no-inline support is active
823 mpxyd: reduce log level on qp_flush to CM level
824 mcm: intra-node proxy missing LID setup on rejects
825 mcm: add intra-node support via ibscif device and mcm provider
826 mcm: provide MIC address info with proxy device open
827 mcm: add device info to non-debug log
828 common: add DAPL_DTO_TYPE_EXTENSION_IMM for rdma_write_imm DTO type checking
829 mpxyd: fix up some of the PI logging
830 dtest: modify rdma_write_with_msg to support uni-direction streaming
831 mcm,mpxyd: fix dreq processing to defer QP flush when proxy WRs still pending
832 mpxyd: update byte_len and comp_cnt for PO to remote HST communications
833 mcm: bug fixes for non-inline devices
834 mcm: return CM_rej with CM_req_in errors
835 mpxyd,mcm: RDMA write with immed data not signaled on request side
836 mcm: add WC opcode and wc_flags in debug log message
837 mpxyd: set options bug fix for mcm_ib_inline
838 Update release notes with latest CM times
839
840 Release 2.1.5 (OFED 3.18 RC3)
841 update release notes, readme
842 dat.conf: update comments regarding versions
843 dtest: add logging of provider private data size with -v
844 scm: remove use of msg.resv field for process id logging
845 cma: report correct CM req private data size on query
846 mpxyd: memset ib_wr structure before post_send on WC and WR requests
847 mcm: add HST side provider support for device without inline data capability
848 ucm: CM changes for UD extended port space and indexer
849 ucm: add device support for new port space hash table
850 ucm: allocate/free AH hash table for UD endpoint types
851 ucm: check for AH caching when destroying via UD extension
852 ucm: optimizations for large scale UD communication management
853 mpxyd: use wr opcode instead of wc opcode to support logging on error cases
854 mcm: HST->MXS mode, using RDMA_WRITE_WITH_IMM, fails with dtest -w
855 dapl: aarch64 support for linux
856 dapltest: add scripts to dist, set default device to IPoIB
857 mpxyd: add wc_flags to proxy work completions
858
859 Release 2.1.4 (OFED 3.18 RC1)
860 mpxyd: fix typo in configuration file
861 cma: RR attributes moved to common ib_cm struct
862 mpxyd: tx thread incorrectly sleeps with negative pi_rw_cnt value
863 dat.conf: add entries for True Scale qib device
864 mpxyd: add support for devices without inline data support
865 ucm: long disconnect times with many-to-one applications
866 openib: add inline data support check during device open
867 cleanup ib/cm attribute management across openib providers
868 dapltest: fix -Werror=format-security issue with printf
869 Release 2.1.3 (targeting OFED 3.18)
870 dapl: mpxyd service changes to support multi-thread single-core option
871 dapl: add rdma_write_imm and write only option to dtest
872 ucm: add time wait override capability for CM services
873 common: dapl_ep_free must serialize CM object destroy
874 dtestx: allow scale up to 1000 EP's
875 ucm: RTU not retransmitted in TIMEWAIT state
876 mpxyd: increase max open files for service
877 mpxyd: DTO completion ERR: status 12, op RDMA_WRITE running MPI alltoall test
878 mcm: HST->MXS mode incorrectly signals multiple fragments per WR
879 mcm: add segmentation to HST->MXS mode for improved performance
880 mpxyd: set global seg_sz to 128KB for proxy data service
881 openib: add port_num to provider named attributes
882 mcm: provide CPU family/model attribute on both host and mic sides
883 dtestx: update IB extension example test with new v2.0.9 features
884 dtest: add dtestsrq for SRQ example and provider testing
885 common: add srq support for openib verbs providers
886 openib: add IB UD cm_free/ah_free extension support in UCM provider
887 openib: add new TIMEWAIT state for CM
888 extension: add IB UD extensions to reduce provider CM and AH memory footprint
889 mpxyd/mcm: add provider specific attribute DAT_IB_PROXY_VERSION
890 mpxyd: log warning if running in COMPAT mode
891 add provider and proxy support for GUID across platform
892 common: return appropriate handles with affiliated EP and EVD async events
893
894 Release 2.1.2 (OFED 3.12-1)
895 mpxyd: add global routing support for proxy connections
896 mcm: only call mix_get_attr if running on MIC
897 openib: modify check for link_layer to handle unspecified
898 dapl: add support for the s390x platform
899 dtest server exchange connection info with client
900 mpxyd: 2 MICs in same numa_node will overlap CPU affinity, don't reset base
901 mcm: implement proxy mix_prov_attr function, add fields CPU model and family
902 mpxyd: tx thread may not be signaled on small segment writes
903
904 Release 2.1.1 (OFED 3.12-1 RC1)
905 common: add provider name to log messages
906 mpxyd: log warning message if numa_node invalid include debuginfo with build
907 build: include debuginfo with build
908 mpxyd: tx thread doesn't sleep during no pending IO state
909 mpxyd: change MIC cpu_mask to per numa node instead of adapter
910 mpxyd: set to MXS mode if device numa_node is invalid (-1)
911 mpxyd: MXS based alltoall benchmark hangs or returns post_send timeout
912 mpxyd: add IO profile capabilities to help debug alltoall stall cases
913 mpxyd: retry stalled inline post_send, init m_idx only when signaled
914
915 Release 2.1.0 (OFED 3.12-1, MIC support added)
916 build: add missing NEWS file
917 update autogen.sh
918 add MCM provider and MPXYD service to build
919 mpxyd: service startup script and configuration file
920 add readme for MCM provider and MPXYD service
921 update Copyright dates
922 add new MIC RDMA proxy service daemon (MPXYD)
923 add new dapl MIC provider (MCM) to support MIC RDMA proxy services
924 MCM: new MIC provider and proxy service definitions
925 cleanup build warnings
926 common: add CQ,QP,MR abstractions for new MIC provider and data proxy service
927 openib: cleanup, use inet_ntop for GIDs, remove some logs, destroy pipes on release
928 common: new dapls_evd_cqe_to_event call, cqe to event
929 common: init ring_buffer, assign hd/tl pos in range
930 allow log level changes during device open
931 ucm: fix cm rbuf setup, include grh pad on initialization
932 ucm: remove duplicate async_event code, use common async event call
933 new lightweight open_query/close_query IB extension for fast attribute query
934 dtestcm: add more detailed debug during disconnect phase
935 cma: long delays when opening cma provider with no IPoIB configured
936 common: new debug levels for low system memory, IA stats, and package info
937 build: remove library check for mverbs with --enable-fca
938 IB extension: segfault in create collective group with non-vector type IA handle"
939 build: change configure help to correctly state collective default=none
940
941 Release 2.0.42 fixes (OFED 3.12 GA)
942 dapltest: increase DTO evd size to prevent CQ overflow on limit_rpost test
943 dapltest: RSP limit test fails. Creation of reserved SP moves EP state to DAT_EP_STATE_RESERVED in error cases.
944 dapl: fix string bug in dapls_dto_op_str
945
946 Release 2.0.41 fixes (OFED 3.12 RC1)
947 dapltest: change server port, from 45278 to 62000, out of registered IANA range
948 dat: lower log level on load errors of provider library
949 dat: dat_ia_open should close provider after failure
950 dapltest: set default limit max to 1000
951 openib: add new provider specific attributes
952 dapltest: update scripts for regression testing purposes
953 dapltest: Add final send/recv "sync" for transaction tests.
954
955 Release 2.0.40 fixes (OFED 3.12)
956 dist: ib collective extension include files missing
957 dapltest: the quit command is missing changes for -n option
958 dat.conf: remove v1, add Mellanox Connect-IB and Intel Xeon Phi MIC
959 NULL undefined on Fedora, incorrectly using kernel stddef.h
960
961 Release 2.0.39 fixes (OFED 3.5-2 GA)
962 dapltest: fix endian swap issue with performance test
963 scm: getifaddrs modfications for better out of the box experience
964 ucm, scm: UD mode triggers list_head assert with large scale alltoall test
965
966 Release 2.0.38
967 dapltest: add -n parameter to override default server port number (45278)
968 ucm,scm: UD mode creates many CR objects per EP that needs cleaned up
969 cma: add DAPL_CM_TOS environment variable to enable passing a TOS to the RDMA CM
970
971 Release 2.0.37
972 common: add support for ia name during dat_ia_query
973 common: dapl_os_atomic_inc/dec() not working as expected on ppc64 machines.
974 dapltest: ppc64 endian issue with exchanged mem handle and address
975
976 Release 2.0.36
977 scm: increase ACK timeout to 20 for a default value to match other providers.
978 common: allow qp modify in init state
979 common: check for valid states during ep posting
980 dat.conf: keep list of providers in order for backward compatibility
981 ucm: record and silently drop a duplicate reject CM message
982 windows: new version of getlocalipaddr not portable
983 dapltest: DFLT_QLEN is defined in multiple tests
984
985 Release 2.0.35
986 config/build: remove post/postun hacking used to modify dat.conf
987 config: clean up help option displays with ext-type options
988 windows: Provide auto-detect between RoCE and Infiniband for Windows.
989 ucm: update UD cm provider to support new CM stat and error counters
990 scm: update socket cm provider to support new CM stat and error counters
991 commom: add cm, link, and diag event counters in IB extended builds
992 scm: use ioctl SIOCIFCONF to get complete list of configured netdev interfaces
993 ucm: UD send failures at scale, ucm_send ERR: get_smsg(hd=149,tl=150)
994 scm: fix retry count on connection pending timeout
995 ucm: cleanup debug message, ntohl on p_size is incorrect
996 cma, scm, ucm: allow EP (QP) creation without EVD (CQ)
997 common: add DAPL_DBG_TYPE_CM_STATS (0x40000) to debug log options
998 common: dapls_ep_flush_cq will segfault when no CQ is attached to EP
999 common: ep_create should allow max_request_iov attribute setting of zero
1000 common: add check for NULL handle on ext calls, SRQ free, and helper functions
1001 common: add missing sub-types to dat_strerror()
1002 common: extended CR event processing missing rejects on errors
1003 ucm: incorrectly sends user reject during CR callback errors
1004 common: change dbg level on CR callback if not listening on SP
1005 scm: incorrectly sends user reject during CR callback errors
1006 dat: add check for NULL handle on IA calls
1007 cma,scm,ucm: extra reference on EP, with RSP, causes dat_ep_free() to hang
1008 common: RSP service points incorrectly freed during CR callback
1009 common: clean up dat_rsp_create log message
1010 common: cleanup debug message on EVD overflows
1011 scm: return correct event error code when remote host refuses requests
1012 dapltest: server CR EVD is too small for multi-client configurations.
1013 Common: CR EVD overflow causes segfault.
1014
1015 Release 2.0.34
1016 scm: change debug message level for listen/bind errors
1017 common: increase default IB ack timer from 16 to 20
1018 common: remote ia address null pointer creates seg fault
1019 common: posting events on full queue returns wrong error code
1020 common: dat_ep_modify seg faults with null ep_param ptr
1021 common: dat_evd_free seg faults with resized software EVD
1022 common: remove assert for incorrect events during cm_request
1023 dat: dat_cno_query with NULL cno_handle causes segmentation fault
1024 scm: dat_psp_create returns wrong error code on bind/listen failure
1025 scm: socket connect request count is reset improperly on retry
1026 scm: when hostname has loopback addr assigned, default to eth0 instead of failing
1027 scm: add port number to error log during hca_open failures
1028 common: query calls return incorrect IA handle to consumer
1029 common: srq create asserts with !dapl_llist_is_empty(head) failed
1030
1031 Release 2.0.33
1032 scm,ucm: fix compatibility issues and set minimum protocol support
1033 build: link librdmacm dependency to ib_acm usage for ucm and scm providers
1034 build: add selective enable/disable-xxx build switch for each provider
1035 build: add extended header files to EXTRA_DIST and fix missing backslash
1036 build: set IB extended coll-type to none by default
1037 common: change errno mapping of EINVAL to DAT_INVALID_PARAMETER
1038 build: add IB collective and FCA provider to dapl build package as an option
1039 common: add new dapls_evd_post_event_ext call for extended events
1040 ucm: add support for IB collective providers
1041 scm: add support for IB collective providers
1042 cma: add support for IB collective providers
1043 common: add supported collective types in named attributes for query
1044 common: add collective call mappings via standard dapli_post_ext()
1045 common: new debug bitmask definition for extension logging
1046 common: new IB collective provider for Mellanox Fabric Collective Agent
1047 dat: add definitions for MPI offloaded collectives in IB transport extensions
1048 common: cleanup debug messages when building with ibacm feature
1049
1050 Release 2.0.32 fixes (OFED 1.5.3 GA):
1051
1052 cma: reduce output log level in disconnect from WARN to CM_WARN
1053 ucm: delay freeing of active side UD cm object in case RTU is dropped
1054 ucm: cm object needs to be on work queue before req sent on wire
1055 ucm,scm: remove use of usec_sleep delays and use events for disc and destroy
1056 common: reduce default max inline data size because of performance anomaly
1057 common: dapls_evd_dto_wait() dbg message should print status and not errno
1058 ucm, scm: exchange max_qp_rd_atom and limit outstanding requests
1059 scm: retry socket connect on ECONNREFUSED under heavy load
1060 common: qp modify RTR using wrong ep attribute parameter for dest_rd_atomic
1061
1062 Release 2.0.31 fixes (OFED 1.5.3 RC1):
1063
1064 common: clean up build warning for unused variable event_ptr
1065 scm, ucm: set RAI_NOROUTE flag with rdma_getaddrinfo() call to avoid blocking.
1066 cma: definition for dapl_sp_remove_ep() is missing in cm.c
1067 libdat: static provider entries created for local SR database not freed
1068 libdat: memory leak in static registration during parsing
1069 common: increase default IB inline send threshold to 400
1070 common cq: a mixup of errno and the -1 return from poll in dapls_wait_comp_channel
1071 ucm: release UD cm objects after AH is exchanged to avoid duplicate request drops
1072 ucm: decrease timeout retry count for disconnect requests
1073 ucm: hold lock when sending cm_msgs to sync timer start with packet send
1074 ucm: add debugging to include process id for better scale up debug aids
1075 cma: disconnect can block for excessive times waiting for rdma_cm DREP timeout
1076 ucm: configure the recv channel FD to non-blocking
1077 windows: Missing librdmacm include path for build
1078 debug build: only timestamp if sending to stdout to avoid performance hit
1079 common: print out errors on free build and not just debug builds
1080 cma: fix debug build issue
1081 scm, ucm: MPI spawn test on oversubcribed server taking excessive time to complete
1082 common: add high resolution time stamps and thread id to sdtout debug logs
1083 common: modify debug in dat_evd_dequeue to reduce noise, only output on non-empty
1084 cma: rdma_destroy_id called twice during device open bind error
1085 common: dat_evd_dequeue (poll_cq) fails with invalid parameter after EP (qp) free
1086 ucm: allow configuration of CM burst (signal) threshold on posting
1087 cma: fix debug build
1088 windows: debug version of windows does not build.
1089 Allow DAPL out of band connection models to use ibacm to obtain path record data.
1090 ucm: add missing map file for UCM provider
1091 ibal: delay QP transition during disconnect phase
1092 Revert "ibal: delay QP transition during disconnect phase"
1093 ibal: delay QP transition during disconnect phase
1094 common: restructure EVD processing to handle EP destruction phase
1095 ibal: sync QP destruction and device close
1096 ucm: remove unnecessary debug warning in async callback
1097