| 1 |
README/Release Notes |
| 2 |
OFED 3.18 DAPL Release 2.1.6 |
| 3 |
August 2015 |
| 4 |
|
| 5 |
User space libraries/utilities for Direct Access Transport (DAT) v2.0. DAT is |
| 6 |
a transport-independent, platform-independent Application Programming |
| 7 |
Interface that supports RDMA (remote direct memory access) devices. |
| 8 |
Note: v1.2 is no longer supported and will not be included with OFED releases |
| 9 |
|
| 10 |
MIC support is provided with the new MCM provider and MPXYD service, since dapl-2.1.0. |
| 11 |
MCM requires the Intel(R) MPSS 3.x (YOCTO) release for Linux to be installed on your system. |
| 12 |
MPSS 3.x for Linux can be downloaded from: http://software.intel.com/mic-developer |
| 13 |
|
| 14 |
For latest documentation and packages: //www.openfabrics.org/downloads/dapl/ |
| 15 |
|
| 16 |
================= |
| 17 |
1.0 Release Notes |
| 18 |
================= |
| 19 |
|
| 20 |
dapl-2.1.5 changes include improvements for large scale UD communication management: |
| 21 |
|
| 22 |
- AH caching, reduced memory footprint (grows as needed) |
| 23 |
- Port space increased to 24 bits |
| 24 |
- Hash table for port space, CM object management |
| 25 |
- Optimized CM wire protocol for fast index lookup |
| 26 |
|
| 27 |
Tested on 1200n 28ppn cluster, AlltoAll Intel MPI, UD mode. |
| 28 |
Both static and dynamic modes, over 500m UD QP connections. |
| 29 |
|
| 30 |
dapl-2.1.6 changes include MIC support for full offload mode |
| 31 |
|
| 32 |
- Add support for Truescale qib devices with no CCL Direct verbs support on MIC. |
| 33 |
- Enhancement for inside the box transfers without IB adapter via ibscif. |
| 34 |
- Add DAPL_NETWORK_NODES, DAPL_NETWORK_PPN environment variables. |
| 35 |
|
| 36 |
========== |
| 37 |
2.0 BUILD: |
| 38 |
========== |
| 39 |
|
| 40 |
# NON_DEBUG build/install example for x86_64, OFED targets |
| 41 |
./configure --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include" |
| 42 |
make install |
| 43 |
|
| 44 |
# DEBUG build/install example for x86_64, using OFED targets |
| 45 |
./configure --enable-debug --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include" |
| 46 |
make install |
| 47 |
|
| 48 |
# COUNTERS build/install example for x86_64, using OFED targets |
| 49 |
./configure --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include -DDAPL_COUNTERS" |
| 50 |
make install |
| 51 |
|
| 52 |
========================================================= |
| 53 |
3.0 Provider descriptions and CM results (cma, scm, ucm): |
| 54 |
========================================================= |
| 55 |
|
| 56 |
1. CMA - uses OFA rdma_cm to setup QP's. IPoIB, ARP, and SA queries required. |
| 57 |
|
| 58 |
Provider name: ofa-v2-ib0 |
| 59 |
PROs: OFA rdma_cm has the most testing across many applications. |
| 60 |
Supports both iWARP and IB. |
| 61 |
|
| 62 |
CONs: Serialization of conn processing with kernel based CM service |
| 63 |
Requires IPoIB ARP for name resolution, storms |
| 64 |
Requires SA for path record queries for IB fabrics. |
| 65 |
Conn Request private data limited to 52 bytes. |
| 66 |
|
| 67 |
Settings for larger clusters (512+ cores): |
| 68 |
|
| 69 |
setenv DAPL_CM_ROUTE_TIMEOUT_MS 20000 |
| 70 |
setenv DAPL_CM_ARP_TIMEOUT_MS 10000 |
| 71 |
|
| 72 |
2. SCM - uses sockets to exchange QP information. IPoIB, ARP, and SA queries NOT required. |
| 73 |
|
| 74 |
Provider name (connectx): ofa-v2-mlx4_0-1 |
| 75 |
PROs: Each rank has own instance of socket cm. More private data with requests. |
| 76 |
Doesn't require path-record lookup. |
| 77 |
|
| 78 |
CONs: Socket resources grow with scale-out, serialization of |
| 79 |
connections with kernel based tcp sockets, |
| 80 |
Competes for MPI socket resources/port space and other TCP applications. |
| 81 |
Sockets remain in TIMEWAIT state for minutes after closure. |
| 82 |
Requires ARP for name resolution. |
| 83 |
Doesn't support iWARP devices. |
| 84 |
|
| 85 |
Settings for larger clusters (512+ cores): |
| 86 |
|
| 87 |
setenv DAPL_ACK_RETRY 7 /* IB RC Ack retry count */ |
| 88 |
setenv DAPL_ACK_TIMER 20 /* IB RC Ack retry timer */ |
| 89 |
|
| 90 |
3. UCM - use's IB UD QP to exchange QP info. Sockets, ARP, IPoIB, and SA queries NOT required. |
| 91 |
|
| 92 |
Provider name (connectx): ofa-v2-mlx4_0-1u |
| 93 |
PROs: Each rank has own instance of CM in user process |
| 94 |
Resources fixed per rank regardless of scale-out size |
| 95 |
No serialization of user or kernel resources establishing connections, |
| 96 |
Simple 3-way msg handsake, CM messages fit in inline data for lowest message latency, |
| 97 |
Supports alternate paths |
| 98 |
No address resolution required. |
| 99 |
No path resolution required. |
| 100 |
|
| 101 |
CONs: New provider with limited testing, a little tougher to debug. |
| 102 |
Doesn't support iWARP |
| 103 |
|
| 104 |
Settings for larger clusters (512+ cores): |
| 105 |
|
| 106 |
setenv DAPL_UCM_REP_TIME 2000 /* REQUEST timer, waiting for REPLY in millisecs */ |
| 107 |
setenv DAPL_UCM_RTU_TIME 2000 /* REPLY timer, waiting for RTU in millisecs */ |
| 108 |
setenv DAPL_UCM_CQ_SIZE 2000 /* CM completion queue */ |
| 109 |
setenv DAPL_UCM_QP_SIZE 2000 /* CM message queue */ |
| 110 |
setenv DAPL_UCM_RETRY 7 /* REQUEST and REPLY retries */ |
| 111 |
setenv DAPL_ACK_RETRY 7 /* IB RC Ack retry count */ |
| 112 |
setenv DAPL_ACK_TIMER 20 /* IB RC Ack retry timer */ |
| 113 |
|
| 114 |
CM Performance: CPS profile for cma, scm, and ucm v2 uDAPL providers: |
| 115 |
----------------------------------------------------------------------- |
| 116 |
Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz (IVT) |
| 117 |
Mellanox MLX4 IB FDR, no switch. |
| 118 |
|
| 119 |
dtestcm (server/client): |
| 120 |
|
| 121 |
cma: Connections: 313.10 usec, CPS 3193.83 Total 0.31 secs, poll_cnt=6300, Num=1000 |
| 122 |
scm: Connections: 167.65 usec, CPS 5964.92 Total 0.17 secs, poll_cnt=2394, Num=1000 |
| 123 |
ucm: Connections: 71.85 usec, CPS 13918.06 Total 0.07 secs, poll_cnt=2360, Num=1000 |
| 124 |
|
| 125 |
dapl_cm_bw: MPI uDAPL/CM profiling application (all-to-all connections, all ranks) |
| 126 |
|
| 127 |
CMA |
| 128 |
2 Connect times (10): Total 0.0049 per 0.0005 CPS=2051.38 |
| 129 |
4 Connect times (40): Total 0.0151 per 0.0004 CPS=2650.16 |
| 130 |
8 Connect times (240): Total 0.0548 per 0.0002 CPS=4380.59 |
| 131 |
16 Connect times (1120): Total 4.0356 per 0.0036 CPS=277.53 |
| 132 |
32 Connect times (4800): Total 4.4704 per 0.0009 CPS=1073.72 |
| 133 |
|
| 134 |
SCM |
| 135 |
2 Connect times (10): Total 0.0029 per 0.0003 CPS=3441.31 |
| 136 |
4 Connect times (40): Total 0.0060 per 0.0002 CPS=6635.97 |
| 137 |
8 Connect times (240): Total 0.0194 per 0.0001 CPS=12383.47 |
| 138 |
16 Connect times (1120): Total 0.0649 per 0.0001 CPS=17246.93 |
| 139 |
32 Connect times (4800): Total 1.0193 per 0.0002 CPS=4708.95 |
| 140 |
|
| 141 |
UCM |
| 142 |
2 Connect times (10): Total 0.0014 per 0.0001 CPS=6993.91 |
| 143 |
4 Connect times (40): Total 0.0045 per 0.0001 CPS=8837.87 |
| 144 |
8 Connect times (240): Total 0.0155 per 0.0001 CPS=15477.13 |
| 145 |
16 Connect times (1120): Total 0.0630 per 0.0001 CPS=17765.12 |
| 146 |
32 Connect times (4800): Total 0.2632 per 0.0001 CPS=18236.54 |
| 147 |
|
| 148 |
=================================================================================================== |
| 149 |
4.0 BKM for installing new DAPL library on your cluster without any impact on existing OFED install: |
| 150 |
==================================================================================================== |
| 151 |
|
| 152 |
Note: example for user /home/user1, (assumes /home/user1 is exported) and MLX4 adapter, port 1 |
| 153 |
|
| 154 |
Download latest 2.1.x package: http://www.openfabrics.org/downloads/dapl/dapl-2.1.6.tar.gz |
| 155 |
|
| 156 |
untar in /home/user1 |
| 157 |
cd /home/user1/dapl-2.1.6 |
| 158 |
./configure LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include" |
| 159 |
make |
| 160 |
|
| 161 |
Create /home/user1/dat.conf with following 3 lines. (entries with path to new libraries): |
| 162 |
|
| 163 |
ofa-v2-mlx4_0-1u u2.0 nonthreadsafe default /home/user1/dapl-2.1.5/dapl/udapl/.libs/libdaploucm.so.2 dapl.2.0 "mlx4_0 1" "" |
| 164 |
ofa-v2-mlx4_0-1m u2.0 nonthreadsafe default /home/user1/dapl-2.1.5/dapl/udapl/.libs/libdaplomcm.so.2 dapl.2.0 "mlx4_0 1" "" |
| 165 |
ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default /home/user1/dapl-2.1.5/dapl/udapl/.libs/libdaploscm.so.2 dapl.2.0 "mlx4_0 1" "" |
| 166 |
ofa-v2-ib0 u2.0 nonthreadsafe default /home/user1/dapl-2.1.5/dapl/udapl/.libs/libdaplcma.so.1 dapl.2.0 "ib0 0" "" |
| 167 |
|
| 168 |
Run uDAPL application or Intel MPI that uses uDAPL, with (assuming mlx4_0 adapters) following: |
| 169 |
|
| 170 |
setenv DAT_OVERRIDE=/home/user1/dat.conf |
| 171 |
setenv LD_LIBRARY_PATH=/home/user1/dapl-2.1.5/dapl/udapl/.libs:$LD_LIBRARY_PATH |
| 172 |
|
| 173 |
If running Intel MPI and uDAPL IB UD cm, set the following (recommended): |
| 174 |
|
| 175 |
setenv I_MPI_DAPL_PROVIDER=ofa-v2-mlx4_0-1u |
| 176 |
|
| 177 |
If running Intel MPI and uDAPL IB mcm with MIC, set the following: |
| 178 |
|
| 179 |
setenv I_MPI_DAPL_PROVIDER=ofa-v2-mlx4_0-1m |
| 180 |
|
| 181 |
If running Intel MPI and uDAPL socket cm, set the following: |
| 182 |
|
| 183 |
setenv I_MPI_DAPL_PROVIDER=ofa-v2-mlx4_0-1 |
| 184 |
|
| 185 |
|
| 186 |
If running Intel MPI and uDAPL rdma_cm, set the following: |
| 187 |
|
| 188 |
setenv I_MPI_DAPL_PROVIDER=ofa-v2-ib0 |
| 189 |
|
| 190 |
|
| 191 |
============================================================ |
| 192 |
5.0 MCM Provider, MPXYD Daemon (CCL-proxy) Build and Install |
| 193 |
============================================================ |
| 194 |
|
| 195 |
MCM is a new uDAPL provider that is an extension to standard DAT 2.0 libraries. The purpose of this service |
| 196 |
is to proxy RDMA writes from the MIC to the HOST to improve large IO performance. The provider will support |
| 197 |
MIC to MIC, HOST to HOST, and MIC to HOST environments. The mcm client will NOT use MPXYD when running on the host. |
| 198 |
It requires a new MPXYD daemon service when clients are running on a MIC KNC adapter. This package installs all the |
| 199 |
host side libraries and daemon service. The MIC libraries must be built and moved over to MIC adapter. This verion |
| 200 |
is currently included with MPSS and all libraries and services will be installed by default. |
| 201 |
|
| 202 |
Current release package: dapl-2.1.6.tar.gz |
| 203 |
|
| 204 |
* Sample host build from source package (ofed must be installed) |
| 205 |
|
| 206 |
./configure --enable-mcm --prefix=/usr --libdir=/usr/lib64 --sysconfdir=/etc |
| 207 |
make |
| 208 |
make install |
| 209 |
|
| 210 |
* Sample host rpmbuild/update from release tarball, /root: |
| 211 |
|
| 212 |
rpmbuild -ta dapl-2.1.6.tar.gz |
| 213 |
rpm -U /root/rpmbuild/RPMS/x86_64/dapl* |
| 214 |
|
| 215 |
* Sample MIC build from source package for MPSS 3.x (MPSS must be installed) |
| 216 |
* Assume /opt is nfs mounted across cluster |
| 217 |
|
| 218 |
source /opt/mpss/3.x/environment-setup-k1om-mpss-linux |
| 219 |
./configure --enable-mcm --prefix /opt/dapl/mic --host=x86_64-k1om-linux |
| 220 |
make |
| 221 |
make install |
| 222 |
|
| 223 |
copy /opt/dapl/mic/* files out to all MIC cards |
| 224 |
|
| 225 |
* Cluster deployment |
| 226 |
|
| 227 |
(1) Build once on the head or on one of the nodes (with MPSS) as described in the above steps. |
| 228 |
|
| 229 |
(2) HOST: Install dapl libraries and mpxyd service, "rpm -U" all dapl RPM files on host nodes: |
| 230 |
|
| 231 |
(3) MIC: Setup dapl overlay for new package (/opt/intel/dapl): |
| 232 |
|
| 233 |
Create /etc/mpss/conf.d/dapl.conf with following entry: |
| 234 |
|
| 235 |
Overlay Filelist /opt/dapl /opt/dapl/dapl.filelist on |
| 236 |
|
| 237 |
Create /opt/dapl/dapl.filelist with following entries: |
| 238 |
|
| 239 |
file /etc/dat.conf mic/etc/dat.conf 755 0 0 |
| 240 |
file /usr/bin/dtest mic/bin/dtest 755 0 0 |
| 241 |
file /usr/bin/dtestx mic/bin/dtestx 755 0 0 |
| 242 |
file /usr/bin/dtestcm mic/bin/dtestcm 755 0 0 |
| 243 |
file /usr/bin/dapltest mic/bin/dapltest 755 0 0 |
| 244 |
file /usr/lib64/libdat.so.2.0.0 mic/lib/libdat.so.2.0.0 755 0 0 |
| 245 |
file /usr/lib64/libdaplofa.so.2.0.0 mic/lib/libdaplofa.so.2.0.0 755 0 0 |
| 246 |
file /usr/lib64/libdaplomcm.so.2.0.0 mic/lib/libdaplomcm.so.2.0.0 755 0 0 |
| 247 |
file /usr/lib64/libdaploscm.so.2.0.0 mic/lib/libdaploscm.so.2.0.0 755 0 0 |
| 248 |
file /usr/lib64/libdaploucm.so.2.0.0 mic/lib/libdaploucm.so.2.0.0 755 0 0 |
| 249 |
|
| 250 |
slink /usr/lib64/libdat.so libdat.so.2.0.0 777 0 0 |
| 251 |
slink /usr/lib64/libdat.so.2 libdat.so.2.0.0 777 0 0 |
| 252 |
slink /usr/lib64/libdaplofa.so libdaplofa.so.2.0.0 777 0 0 |
| 253 |
slink /usr/lib64/libdaplofa.so.2 libdaplofa.so.2.0.0 777 0 0 |
| 254 |
slink /usr/lib64/libdaplomcm.so libdaplomcm.so.2.0.0 777 0 0 |
| 255 |
slink /usr/lib64/libdaplomcm.so.2 libdaplomcm.so.2.0.0 777 0 0 |
| 256 |
slink /usr/lib64/libdaploscm.so libdaploscm.so.2.0.0 777 0 0 |
| 257 |
slink /usr/lib64/libdaploscm.so.2 libdaploscm.so.2.0.0 777 0 0 |
| 258 |
slink /usr/lib64/libdaploucm.so libdaploucm.so.2.0.0 777 0 0 |
| 259 |
slink /usr/lib64/libdaploucm.so.2 libdaploucm.so.2.0.0 777 0 0 |
| 260 |
|
| 261 |
Reboot or restart MPSS and ofed-mic services |
| 262 |
|
| 263 |
Check for dapl overlay |
| 264 |
micctrl --config |
| 265 |
|
| 266 |
* Setup for non-root CCL Proxy testing, MPXYD running as process with different service port from your /home directory: |
| 267 |
|
| 268 |
Using build instructions above, change prefix as follow and "make install": |
| 269 |
|
| 270 |
Build MIC: |
| 271 |
--prefix=/home/username/ccl-proxy-mic |
| 272 |
|
| 273 |
Build host: |
| 274 |
--prefix=/home/username/ccl-proxy-host |
| 275 |
|
| 276 |
edit /home/username/ccl-proxy-host/etc/mpxyd.conf and change the following entries: |
| 277 |
|
| 278 |
log_file /var/log/mpxyd.log to log_file /tmp/username/mpxyd.log |
| 279 |
lock_file /var/log/mpxyd.pid to lock_file /tmp/username/mpxyd.log |
| 280 |
scif_port_id 68 to scif_port_id 1068 |
| 281 |
|
| 282 |
start the mpxyd process on each node |
| 283 |
|
| 284 |
ssh node1-hostname /home/username/ccl-proxy-host/sbin/mpxyd -P -O /home/username/ccl-proxy-host/etc/mpxyd.conf& |
| 285 |
|
| 286 |
Note: override default port id using following environment variable: |
| 287 |
|
| 288 |
export DAPL_MCM_PORT_ID=1068 |
| 289 |
|
| 290 |
* Notes |
| 291 |
|
| 292 |
(1) Modify "/etc/mpxyd.conf" to change the settings for the proxy. Especially, try different values |
| 293 |
of "buffer_segment_size" for performance tuning. Use a smaller value for "buffer_pool_mb" |
| 294 |
to reduce the memory foorprint of mpxyd. Use a larger value for "scif_listen_qlen" to run |
| 295 |
more MPI ranks per card. Also modify mcm_affinity_base to the desired CPU_id to insure |
| 296 |
socket to adapter affinity. Best performance when HCA, MIC, and CPU are on same socket. |
| 297 |
Default settings are on CPU socket 0. |
| 298 |
|
| 299 |
(2) By default, only writes originated from MIC is proxied. However, it is also possible to proxy |
| 300 |
host-originated writes (e.g. for debugging purpose). To do this, set the environment variable |
| 301 |
"DAPL_MCM_ALWAYS_PROXY=1". This variable applies to the provider, not the proxy. |
| 302 |
|
| 303 |
* Use the MCM provider with Intel MPI 5.1 or greater for best out of box experiences with MIC. |
| 304 |
|
| 305 |
(1) Recommended settings: |
| 306 |
|
| 307 |
export I_MPI_MIC=1 |
| 308 |
export I_MPI_DEBUG=2 |
| 309 |
export I_MPI_FALLBACK=0 |
| 310 |
|
| 311 |
============================= |
| 312 |
6.0 Environment Variables |
| 313 |
============================= |
| 314 |
|
| 315 |
- IB UD options using UCM provider, large scale settings (Xeon) |
| 316 |
|
| 317 |
export DAPL_NETWORK_NODES= /* set to active nodes on network for CM */ |
| 318 |
export DAPL_NETWORK_PPN= /* set to active processes per node for CM */ |
| 319 |
|
| 320 |
/* The following will be adjusted by provider based on NODES, PPN */ |
| 321 |
export DAPL_UCM_REP_TIME=8000 /* REQUEST timer, waiting on REPLY, msecs, default = 800 */ |
| 322 |
export DAPL_UCM_RTU_TIME=8000 /* REPLY timer, waiting for RTU in msecs, default=400 */ |
| 323 |
export DAPL_UCM_RETRY=7 /* REQUEST & REPLY retries, default = 7 */ |
| 324 |
export DAPL_UCM_QP_SIZE=4000 /* CM req/reply work queue size, default = 500 entries */ |
| 325 |
export DAPL_UCM_CQ_SIZE=4000 /* CM req/reply completion queue size, default = 500 entries */ |
| 326 |
export DAPL_UCM_TX_BURST=100 /* CM signal rate on send messages */ |
| 327 |
export DAPL_UCM_ENTRY_BITS=11 /* default = 11-bit, 2KB entries, allocation blocks */; |
| 328 |
export DAPL_UCM_ARRAY_BITS=18 /* default = 18 bit, 256KB total */ |
| 329 |
|
| 330 |
- IB RC options using SCM provider |
| 331 |
|
| 332 |
export DAPL_SCM_NETDEV=ib0 /* default is first non-loopback netdev */ |
| 333 |
|
| 334 |
- Other IB settings for all providers: |
| 335 |
|
| 336 |
export DAPL_MAX_INLINE=64 /* IB RC inline optimization, best small msg latency, def=64 */ |
| 337 |
export DAPL_ACK_RETRY=7 /* IB RC Ack retry count, default 7 */ |
| 338 |
export DAPL_ACK_TIMER=20 /* IB RC Ack retry timer, 5 bits, 4.096us*2^ack_timer. 16== 268ms, 20==4.2s */ |
| 339 |
export DAPL_IB_MTU=2048 /* IB MTU size, default = 2048 */ |
| 340 |
export DAPL_RNR_TIMER=12 /* 5 bits, 12 =.64ms, 28 =163ms, 31 =491ms */ |
| 341 |
export DAPL_RNR_RETRY=7 /* 3 bits, 7 == infinite */ |
| 342 |
export DAPL_IB_PKEY= 0 /* override IB partition key, default is pkey index 0 */ |
| 343 |
export DAPL_IB_SL=0 /* override IB Sevice level, default = 0 */ |
| 344 |
|
| 345 |
- Other options: |
| 346 |
export DAPL_WR_MAX=500 /* used to reduce max qp depth on all IB providers, default = dev attributes */ |
| 347 |
|
| 348 |
Debug logging and Counter settings ( --enable-counters) |
| 349 |
|
| 350 |
export DAPL_DBG_SYS_MEM=10 /* threshold for low sys memory warning, def = 10 percent */ |
| 351 |
export DAPL_DBG_TYPE=0x0000003 /* set log, monitor, and error checking, default = warnings and errors */ |
| 352 |
|
| 353 |
DAPL_DBG_TYPE bit settings as follow: |
| 354 |
|
| 355 |
DAPL_DBG_TYPE_ERR = 0x0001, |
| 356 |
DAPL_DBG_TYPE_WARN = 0x0002, |
| 357 |
DAPL_DBG_TYPE_EVD = 0x0004, |
| 358 |
DAPL_DBG_TYPE_CM = 0x0008, |
| 359 |
DAPL_DBG_TYPE_EP = 0x0010, |
| 360 |
DAPL_DBG_TYPE_UTIL = 0x0020, |
| 361 |
DAPL_DBG_TYPE_CALLBACK = 0x0040, |
| 362 |
DAPL_DBG_TYPE_DTO_COMP_ERR = 0x0080, |
| 363 |
DAPL_DBG_TYPE_API = 0x0100, |
| 364 |
DAPL_DBG_TYPE_RTN = 0x0200, |
| 365 |
DAPL_DBG_TYPE_EXCEPTION = 0x0400, |
| 366 |
DAPL_DBG_TYPE_SRQ = 0x0800, |
| 367 |
DAPL_DBG_TYPE_CNTR = 0x1000, |
| 368 |
DAPL_DBG_TYPE_CM_LIST = 0x2000, |
| 369 |
DAPL_DBG_TYPE_THREAD = 0x4000, |
| 370 |
DAPL_DBG_TYPE_CM_EST = 0x8000, |
| 371 |
DAPL_DBG_TYPE_CM_WARN = 0x10000, |
| 372 |
DAPL_DBG_TYPE_EXTENSION = 0x20000, |
| 373 |
DAPL_DBG_TYPE_CM_STATS = 0x40000, |
| 374 |
DAPL_DBG_TYPE_CM_ERRS = 0x80000, /* print any cm errors on device close */ |
| 375 |
DAPL_DBG_TYPE_LINK_ERRS = 0x100000, /* print any link errors on device close */ |
| 376 |
DAPL_DBG_TYPE_LINK_WARN = 0x200000, /* print any link warning on device close */ |
| 377 |
DAPL_DBG_TYPE_DIAG_ERRS = 0x400000, /* print any diag_counter errors on dev close */ |
| 378 |
DAPL_DBG_TYPE_SYS_WARN = 0x800000, /* print low mem warning during alloc, reg_mem */ |
| 379 |
DAPL_DBG_TYPE_VER = 0x1000000, /* print dapl ver and build date during dev open */ |
| 380 |
|
| 381 |
============================= |
| 382 |
7.0 SAMPLE uDAPL APPLICATION: |
| 383 |
============================= |
| 384 |
|
| 385 |
There are 2 sample programs, with manpages, provided with this package. |
| 386 |
|
| 387 |
(dapl/test/dtest/) |
| 388 |
|
| 389 |
NAME |
| 390 |
dtest - simple uDAPL send/receive and RDMA test |
| 391 |
|
| 392 |
SYNOPSIS |
| 393 |
dtest [-P provider] [-b buf size] [-B burst count][-v] [-c] [-p] [-d] [-s] |
| 394 |
|
| 395 |
dtest [-P provider] [-b buf size] [-B burst count][-v] [-c] [-p] [-d] [-h HOSTNAME] |
| 396 |
|
| 397 |
DESCRIPTION |
| 398 |
dtest is a simple test used to exercise and verify the uDAPL interfaces. At least two instantia- |
| 399 |
tions of the test must be run. One acts as the server and the other the client. The server side of |
| 400 |
the test, once invoked listens for connection requests, until timing out or killed. Upon receipt |
| 401 |
of a cd connection request, the connection is established, the server and client sides exchange |
| 402 |
information necessary to perform RDMA writes and reads. |
| 403 |
|
| 404 |
OPTIONS |
| 405 |
-P=PROVIDER |
| 406 |
use PROVIDER to specify uDAPL interface using /etc/dat.conf (default OpenIB-cma) |
| 407 |
|
| 408 |
-b=BUFFER_SIZE |
| 409 |
use buffer size BUFFER_SIZE for RDMA(default 64) |
| 410 |
|
| 411 |
-B=BURST_COUNT |
| 412 |
use busrt count BURST_COUNT for interations (default 10) |
| 413 |
|
| 414 |
-v, verbose output(default off) |
| 415 |
|
| 416 |
-c, use consumer notification events (default off) |
| 417 |
|
| 418 |
-p, use polling (default wait for event) |
| 419 |
|
| 420 |
-d, delay in seconds before close (default off) |
| 421 |
|
| 422 |
-s, run as server (default - run as server) |
| 423 |
|
| 424 |
-h=HOSTNAME |
| 425 |
use HOSTNAME to specify server hostname or IP address (default - none) |
| 426 |
|
| 427 |
EXAMPLES |
| 428 |
dtest -P OpenIB-cma -v -s |
| 429 |
Starts a server process with debug verbosity using provider OpenIB-cma. |
| 430 |
|
| 431 |
dtest -P OpenIB-cma -h server1-ib0 |
| 432 |
|
| 433 |
Starts a client process, using OpenIB-cma provider to connect to hostname server1-ib0. |
| 434 |
|
| 435 |
SEE ALSO |
| 436 |
dapltest(1) |
| 437 |
|
| 438 |
AUTHORS |
| 439 |
Arlin Davis |
| 440 |
<[email protected]> |
| 441 |
|
| 442 |
BUGS |
| 443 |
|
| 444 |
/dapl/test/dapltest/ |
| 445 |
|
| 446 |
NAME |
| 447 |
dapltest - test for the Direct Access Programming Library (DAPL) |
| 448 |
|
| 449 |
DESCRIPTION |
| 450 |
Dapltest is a set of tests developed to exercise, characterize, and verify the DAPL interfaces |
| 451 |
during development and porting. At least two instantiations of the test must be run. One acts as |
| 452 |
the server, fielding requests and spawning server-side test threads as needed. Other client invo- |
| 453 |
cations connect to the server and issue test requests. The server side of the test, once invoked, |
| 454 |
listens continuously for client connection requests, until quit or killed. Upon receipt of a con- |
| 455 |
nection request, the connection is established, the server and client sides swap version numbers |
| 456 |
to verify that they are able to communicate, and the client sends the test request to the server. |
| 457 |
If the version numbers match, and the test request is well-formed, the server spawns the threads |
| 458 |
needed to run the test before awaiting further connections. |
| 459 |
|
| 460 |
USAGE |
| 461 |
dapltest [ -f script_file_name ] [ -T S|Q|T|P|L ] [ -D device_name ] [ -d ] [ -R HT|LL|EC|PM|BE ] |
| 462 |
|
| 463 |
With no arguments, dapltest runs as a server using default values, and loops accepting requests |
| 464 |
from clients. |
| 465 |
|
| 466 |
The -f option allows all arguments to be placed in a file, to ease test automation. |
| 467 |
|
| 468 |
The following arguments are common to all tests: |
| 469 |
|
| 470 |
[ -T S|Q|T|P|L ] |
| 471 |
Test function to be performed: |
| 472 |
|
| 473 |
S - server loop |
| 474 |
|
| 475 |
Q - quit, client requests that server wait for any outstanding tests to complete, then |
| 476 |
clean up and exit |
| 477 |
|
| 478 |
T - transaction test, transfers data between client and server |
| 479 |
|
| 480 |
P - performance test, times DTO operations |
| 481 |
|
| 482 |
L - limit test, exhausts various resources, runs in client w/o server interaction |
| 483 |
Default: S |
| 484 |
|
| 485 |
[ -D device_name ] |
| 486 |
Specifies the interface adapter name as documented in the /etc/dat.conf static configura- |
| 487 |
tion file. This name corresponds to the provider library to open. Default: none |
| 488 |
|
| 489 |
[ -d ] Enables extra debug verbosity, primarily tracing of the various DAPL operations as they |
| 490 |
progress. Repeating this parameter increases debug spew. Errors encountered result in the |
| 491 |
test spewing some explanatory text and stopping; this flag provides more detail about what |
| 492 |
lead up to the error. Default: zero |
| 493 |
|
| 494 |
[ -R BE ] |
| 495 |
Indicate the quality of service (QoS) desired. Choices are: |
| 496 |
|
| 497 |
HT - high throughput |
| 498 |
|
| 499 |
LL - low latency |
| 500 |
|
| 501 |
EC - economy (neither HT nor LL) |
| 502 |
|
| 503 |
PM - premium |
| 504 |
|
| 505 |
BE - best effort Default: BE |
| 506 |
|
| 507 |
Usage - Quit test client |
| 508 |
|
| 509 |
dapltest [Common_Args] [ -s server_name ] |
| 510 |
|
| 511 |
Quit testing (-T Q) connects to the server to ask it to clean up and |
| 512 |
exit (after it waits for any outstanding test runs to complete). |
| 513 |
In addition to being more polite than simply killing the server, |
| 514 |
this test exercises the DAPL object teardown code paths. |
| 515 |
There is only one argument other than those supported by all tests: |
| 516 |
|
| 517 |
-s server_name Specifies the name of the server interface. |
| 518 |
No default. |
| 519 |
|
| 520 |
Usage - Transaction test client |
| 521 |
|
| 522 |
dapltest [Common_Args] [ -s server_name ] |
| 523 |
[ -t threads ] [ -w endpoints ] [ -i iterations ] [ -Q ] |
| 524 |
[ -V ] [ -P ] OPclient OPserver [ op3, |
| 525 |
|
| 526 |
Transaction testing (-T T) transfers a variable amount of data between |
| 527 |
client and server. The data transfer can be described as a sequence of |
| 528 |
individual operations; that entire sequence is transferred ’iterations’ |
| 529 |
times by each thread over all of its endpoint(s). |
| 530 |
|
| 531 |
The following parameters determine the behavior of the transaction test: |
| 532 |
|
| 533 |
-s server_name Specifies the name or IP address of the server interface. |
| 534 |
No default. |
| 535 |
|
| 536 |
[ -t threads ] Specify the number of threads to be used. |
| 537 |
Default: 1 |
| 538 |
|
| 539 |
[ -w endpoints ] Specify the number of connected endpoints per thread. |
| 540 |
Default: 1 |
| 541 |
|
| 542 |
[ -i iterations ] Specify the number of times the entire sequence |
| 543 |
of data transfers will be made over each endpoint. |
| 544 |
Default: 1000 |
| 545 |
|
| 546 |
[ -Q ] Funnel completion events into a CNO. |
| 547 |
Default: use EVDs |
| 548 |
|
| 549 |
[ -V ] Validate the data being transferred. |
| 550 |
Default: ignore the data |
| 551 |
|
| 552 |
[ -P ] Turn on DTO completion polling |
| 553 |
Default: off |
| 554 |
|
| 555 |
OP1 OP2 [ OP3, ... ] |
| 556 |
A single transaction (OPx) consists of: |
| 557 |
|
| 558 |
server|client Indicates who initiates the |
| 559 |
data transfer. |
| 560 |
|
| 561 |
SR|RR|RW Indicates the type of transfer: |
| 562 |
SR send/recv |
| 563 |
RR RDMA read |
| 564 |
RW RDMA write |
| 565 |
Defaults: none |
| 566 |
|
| 567 |
[ seg_size [ num_segs ] ] |
| 568 |
: |
| 569 |
|
| 570 |
Indicates the amount and format |
| 571 |
of the data to be transferred. |
| 572 |
Default: 4096 1 |
| 573 |
(i.e., 1 4KB buffer) |
| 574 |
|
| 575 |
[ -f ] For SR transfers only, indicates |
| 576 |
that a client’s send transfer |
| 577 |
completion should be reaped when |
| 578 |
the next recv completion is reaped. |
| 579 |
Sends and receives must be paired |
| 580 |
(one client, one server, and in that |
| 581 |
order) for this option to be used. |
| 582 |
Restrictions: |
| 583 |
|
| 584 |
Due to the flow control algorithm used by the transaction test, there |
| 585 |
must be at least one SR OP for both the client and the server. |
| 586 |
|
| 587 |
Requesting data validation (-V) causes the test to automatically append |
| 588 |
three OPs to those specified. These additional operations provide |
| 589 |
synchronization points during each iteration, at which all user-specified |
| 590 |
transaction buffers are checked. These three appended operations satisfy |
| 591 |
the "one SR in each direction" requirement. |
| 592 |
|
| 593 |
The transaction OP list is printed out if -d is supplied. |
| 594 |
|
| 595 |
Usage - Performance test client |
| 596 |
|
| 597 |
dapltest [Common_Args] -s server_name [ -m p|b ] |
| 598 |
[ -i iterations ] [ -p pipeline ] OP |
| 599 |
|
| 600 |
Performance testing (-T P) times the transfer of an operation. |
| 601 |
The operation is posted ’iterations’ times. |
| 602 |
|
| 603 |
The following parameters determine the behavior of the transaction test: |
| 604 |
|
| 605 |
-s server_name Specifies the name or IP address of the server interface. |
| 606 |
No default. |
| 607 |
|
| 608 |
-m b|p Used to choose either blocking (b) or polling (p) |
| 609 |
Default: blocking (b) |
| 610 |
[ -i iterations ] Specify the number of times the entire sequence |
| 611 |
of data transfers will be made over each endpoint. |
| 612 |
Default: 1000 |
| 613 |
|
| 614 |
[ -p pipeline ] Specify the pipline length, valid arguments are in |
| 615 |
the range [0,MAX_SEND_DTOS]. If a value greater than |
| 616 |
MAX_SEND_DTOS is requested the value will be |
| 617 |
adjusted down to MAX_SEND_DTOS. |
| 618 |
Default: MAX_SEND_DTOS |
| 619 |
|
| 620 |
OP Specifies the operation as follow: |
| 621 |
|
| 622 |
RR|RW Indicates the type of transfer: |
| 623 |
RR RDMA read |
| 624 |
RW RDMA write |
| 625 |
Defaults: none |
| 626 |
|
| 627 |
[ seg_size [ num_segs ] ] |
| 628 |
Indicates the amount and format |
| 629 |
of the data to be transferred. |
| 630 |
Default: 4096 1 |
| 631 |
(i.e., 1 4KB buffer) |
| 632 |
Usage - Limit test client |
| 633 |
|
| 634 |
Limit testing (-T L) neither requires nor connects to any server |
| 635 |
instance. The client runs one or more tests which attempt to |
| 636 |
exhaust various resources to determine DAPL limits and exercise |
| 637 |
DAPL error paths. If no arguments are given, all tests are run. |
| 638 |
|
| 639 |
Limit testing creates the sequence of DAT objects needed to |
| 640 |
move data back and forth, attempting to find the limits supported |
| 641 |
for the DAPL object requested. For example, if the LMR creation |
| 642 |
limit is being examined, the test will create a set of |
| 643 |
{IA, PZ, CNO, EVD, EP} before trying to run dat_lmr_create() to |
| 644 |
failure using that set of DAPL objects. The ’width’ parameter |
| 645 |
can be used to control how many of these parallel DAPL object |
| 646 |
sets are created before beating upon the requested constructor. |
| 647 |
Use of -m limits the number of dat_*_create() calls that will |
| 648 |
be attempted, which can be helpful if the DAPL in use supports |
| 649 |
essentailly unlimited numbers of some objects. |
| 650 |
The limit test arguments are: |
| 651 |
|
| 652 |
[ -m maximum ] Specify the maximum number of dapl_*_create() |
| 653 |
attempts. |
| 654 |
Default: run to object creation failure |
| 655 |
|
| 656 |
[ -w width ] Specify the number of DAPL object sets to |
| 657 |
create while initializing. |
| 658 |
Default: 1 |
| 659 |
|
| 660 |
[ limit_ia ] Attempt to exhaust dat_ia_open() |
| 661 |
|
| 662 |
[ limit_pz ] Attempt to exhaust dat_pz_create() |
| 663 |
|
| 664 |
[ limit_cno ] Attempt to exhaust dat_cno_create() |
| 665 |
|
| 666 |
[ limit_evd ] Attempt to exhaust dat_evd_create() |
| 667 |
|
| 668 |
[ limit_ep ] Attempt to exhaust dat_ep_create() |
| 669 |
|
| 670 |
[ limit_rsp ] Attempt to exhaust dat_rsp_create() |
| 671 |
|
| 672 |
[ limit_psp ] Attempt to exhaust dat_psp_create() |
| 673 |
|
| 674 |
[ limit_lmr ] Attempt to exhaust dat_lmr_create(4KB) |
| 675 |
|
| 676 |
[ limit_rpost ] Attempt to exhaust dat_ep_post_recv(4KB) |
| 677 |
|
| 678 |
[ limit_size_lmr ] Probe maximum size dat_lmr_create() |
| 679 |
|
| 680 |
Default: run all tests |
| 681 |
EXAMPLES |
| 682 |
dapltest -T S -d -D OpenIB-cma |
| 683 |
|
| 684 |
Starts a server process with debug verbosity. |
| 685 |
|
| 686 |
dapltest -T T -d -s host1-ib0 -D OpenIB-cma -i 100 client SR 4096 2 server SR 4096 2 |
| 687 |
|
| 688 |
Runs a transaction test, with both sides |
| 689 |
sending one buffer with two 4KB segments, |
| 690 |
one hundred times. |
| 691 |
|
| 692 |
dapltest -T P -d -s host1-ib0 -D OpenIB-cma -i 100 SR 4096 2 |
| 693 |
|
| 694 |
Runs a performance test, with the client |
| 695 |
sending one buffer with two 4KB segments, |
| 696 |
one hundred times. |
| 697 |
|
| 698 |
dapltest -T Q -s host1-ib0 -D OpenIB-cma |
| 699 |
|
| 700 |
Asks the server to clean up and exit. |
| 701 |
|
| 702 |
dapltest -T L -D OpenIB-cma -d -w 16 -m 1000 |
| 703 |
|
| 704 |
Runs all of the limit tests, setting up |
| 705 |
16 complete sets of DAPL objects, and |
| 706 |
creating at most a thousand instances |
| 707 |
when trying to exhaust resources. |
| 708 |
|
| 709 |
dapltest -T T -V -d -t 2 -w 4 -i 55555 -s linux3 -D OpenIB-cma client RW 4096 1 server RW 2048 4 |
| 710 |
client SR 1024 4 server SR 4096 2 client SR 1024 3 -f server SR 2048 1 -f |
| 711 |
|
| 712 |
Runs a more complicated transaction test, |
| 713 |
with two thread using four EPs each, |
| 714 |
sending a more complicated buffer pattern |
| 715 |
for a larger number of iterations, |
| 716 |
validating the data received. |
| 717 |
|
| 718 |
============================= |
| 719 |
8.0 Summary of Fixes/Changes: |
| 720 |
============================= |
| 721 |
|
| 722 |
Release 2.1.6 (OFED 3.18-1) |
| 723 |
ucm: add cluster size environments to adjust CM timers |
| 724 |
mpxyd: proxy_in data transfers can improperly start before RTU received |
| 725 |
mcm: forward open/query for MFO devices in query only mode |
| 726 |
mpxyd: byte swap incorrect on WRC wr_len |
| 727 |
dtest: remove ERR message from flush QP function |
| 728 |
dapltest: Quit command with "-n port" number will core dump |
| 729 |
config: update dat.conf for MFO qib devices, 2 adapters/ports |
| 730 |
mpxyd: add MFO support on proxy side |
| 731 |
mcm: add MFO proxy commands, device, and CM support |
| 732 |
mcm: add MFO support to openib_common code base |
| 733 |
mcm: add full offload (MFO) mode to provider to support qib on MIC |
| 734 |
dtest: pre-allocated buffer too small for RMR, DTO ops timeout |
| 735 |
mpxyd: fix buffer initialization when no-inline support is active |
| 736 |
mpxyd: reduce log level on qp_flush to CM level |
| 737 |
mcm: intra-node proxy missing LID setup on rejects |
| 738 |
mcm: add intra-node support via ibscif device and mcm provider |
| 739 |
mcm: provide MIC address info with proxy device open |
| 740 |
mcm: add device info to non-debug log |
| 741 |
common: add DAPL_DTO_TYPE_EXTENSION_IMM for rdma_write_imm DTO type checking |
| 742 |
mpxyd: fix up some of the PI logging |
| 743 |
dtest: modify rdma_write_with_msg to support uni-direction streaming |
| 744 |
mcm,mpxyd: fix dreq processing to defer QP flush when proxy WRs still pending |
| 745 |
mpxyd: update byte_len and comp_cnt for PO to remote HST communications |
| 746 |
mcm: bug fixes for non-inline devices |
| 747 |
mcm: return CM_rej with CM_req_in errors |
| 748 |
mpxyd,mcm: RDMA write with immed data not signaled on request side |
| 749 |
mcm: add WC opcode and wc_flags in debug log message |
| 750 |
mpxyd: set options bug fix for mcm_ib_inline |
| 751 |
Update release notes with latest CM times |
| 752 |
|
| 753 |
Release 2.1.5 (OFED 3.18 RC3) |
| 754 |
update release notes, readme |
| 755 |
dat.conf: update comments regarding versions |
| 756 |
dtest: add logging of provider private data size with -v |
| 757 |
scm: remove use of msg.resv field for process id logging |
| 758 |
cma: report correct CM req private data size on query |
| 759 |
mpxyd: memset ib_wr structure before post_send on WC and WR requests |
| 760 |
mcm: add HST side provider support for device without inline data capability |
| 761 |
ucm: CM changes for UD extended port space and indexer |
| 762 |
ucm: add device support for new port space hash table |
| 763 |
ucm: allocate/free AH hash table for UD endpoint types |
| 764 |
ucm: check for AH caching when destroying via UD extension |
| 765 |
ucm: optimizations for large scale UD communication management |
| 766 |
mpxyd: use wr opcode instead of wc opcode to support logging on error cases |
| 767 |
mcm: HST->MXS mode, using RDMA_WRITE_WITH_IMM, fails with dtest -w |
| 768 |
dapl: aarch64 support for linux |
| 769 |
dapltest: add scripts to dist, set default device to IPoIB |
| 770 |
mpxyd: add wc_flags to proxy work completions |
| 771 |
|
| 772 |
Release 2.1.4 (OFED 3.18 RC1) |
| 773 |
mpxyd: fix typo in configuration file |
| 774 |
cma: RR attributes moved to common ib_cm struct |
| 775 |
mpxyd: tx thread incorrectly sleeps with negative pi_rw_cnt value |
| 776 |
dat.conf: add entries for True Scale qib device |
| 777 |
mpxyd: add support for devices without inline data support |
| 778 |
ucm: long disconnect times with many-to-one applications |
| 779 |
openib: add inline data support check during device open |
| 780 |
cleanup ib/cm attribute management across openib providers |
| 781 |
dapltest: fix -Werror=format-security issue with printf |
| 782 |
Release 2.1.3 (targeting OFED 3.18) |
| 783 |
dapl: mpxyd service changes to support multi-thread single-core option |
| 784 |
dapl: add rdma_write_imm and write only option to dtest |
| 785 |
ucm: add time wait override capability for CM services |
| 786 |
common: dapl_ep_free must serialize CM object destroy |
| 787 |
dtestx: allow scale up to 1000 EP's |
| 788 |
ucm: RTU not retransmitted in TIMEWAIT state |
| 789 |
mpxyd: increase max open files for service |
| 790 |
mpxyd: DTO completion ERR: status 12, op RDMA_WRITE running MPI alltoall test |
| 791 |
mcm: HST->MXS mode incorrectly signals multiple fragments per WR |
| 792 |
mcm: add segmentation to HST->MXS mode for improved performance |
| 793 |
mpxyd: set global seg_sz to 128KB for proxy data service |
| 794 |
openib: add port_num to provider named attributes |
| 795 |
mcm: provide CPU family/model attribute on both host and mic sides |
| 796 |
dtestx: update IB extension example test with new v2.0.9 features |
| 797 |
dtest: add dtestsrq for SRQ example and provider testing |
| 798 |
common: add srq support for openib verbs providers |
| 799 |
openib: add IB UD cm_free/ah_free extension support in UCM provider |
| 800 |
openib: add new TIMEWAIT state for CM |
| 801 |
extension: add IB UD extensions to reduce provider CM and AH memory footprint |
| 802 |
mpxyd/mcm: add provider specific attribute DAT_IB_PROXY_VERSION |
| 803 |
mpxyd: log warning if running in COMPAT mode |
| 804 |
add provider and proxy support for GUID across platform |
| 805 |
common: return appropriate handles with affiliated EP and EVD async events |
| 806 |
|
| 807 |
Release 2.1.2 (OFED 3.12-1) |
| 808 |
mpxyd: add global routing support for proxy connections |
| 809 |
mcm: only call mix_get_attr if running on MIC |
| 810 |
openib: modify check for link_layer to handle unspecified |
| 811 |
dapl: add support for the s390x platform |
| 812 |
dtest server exchange connection info with client |
| 813 |
mpxyd: 2 MICs in same numa_node will overlap CPU affinity, don't reset base |
| 814 |
mcm: implement proxy mix_prov_attr function, add fields CPU model and family |
| 815 |
mpxyd: tx thread may not be signaled on small segment writes |
| 816 |
|
| 817 |
Release 2.1.1 (OFED 3.12-1 RC1) |
| 818 |
common: add provider name to log messages |
| 819 |
mpxyd: log warning message if numa_node invalid include debuginfo with build |
| 820 |
build: include debuginfo with build |
| 821 |
mpxyd: tx thread doesn't sleep during no pending IO state |
| 822 |
mpxyd: change MIC cpu_mask to per numa node instead of adapter |
| 823 |
mpxyd: set to MXS mode if device numa_node is invalid (-1) |
| 824 |
mpxyd: MXS based alltoall benchmark hangs or returns post_send timeout |
| 825 |
mpxyd: add IO profile capabilities to help debug alltoall stall cases |
| 826 |
mpxyd: retry stalled inline post_send, init m_idx only when signaled |
| 827 |
|
| 828 |
Release 2.1.0 (OFED 3.12-1, MIC support added) |
| 829 |
build: add missing NEWS file |
| 830 |
update autogen.sh |
| 831 |
add MCM provider and MPXYD service to build |
| 832 |
mpxyd: service startup script and configuration file |
| 833 |
add readme for MCM provider and MPXYD service |
| 834 |
update Copyright dates |
| 835 |
add new MIC RDMA proxy service daemon (MPXYD) |
| 836 |
add new dapl MIC provider (MCM) to support MIC RDMA proxy services |
| 837 |
MCM: new MIC provider and proxy service definitions |
| 838 |
cleanup build warnings |
| 839 |
common: add CQ,QP,MR abstractions for new MIC provider and data proxy service |
| 840 |
openib: cleanup, use inet_ntop for GIDs, remove some logs, destroy pipes on release |
| 841 |
common: new dapls_evd_cqe_to_event call, cqe to event |
| 842 |
common: init ring_buffer, assign hd/tl pos in range |
| 843 |
allow log level changes during device open |
| 844 |
ucm: fix cm rbuf setup, include grh pad on initialization |
| 845 |
ucm: remove duplicate async_event code, use common async event call |
| 846 |
new lightweight open_query/close_query IB extension for fast attribute query |
| 847 |
dtestcm: add more detailed debug during disconnect phase |
| 848 |
cma: long delays when opening cma provider with no IPoIB configured |
| 849 |
common: new debug levels for low system memory, IA stats, and package info |
| 850 |
build: remove library check for mverbs with --enable-fca |
| 851 |
IB extension: segfault in create collective group with non-vector type IA handle" |
| 852 |
build: change configure help to correctly state collective default=none |
| 853 |
|
| 854 |
Release 2.0.42 fixes (OFED 3.12 GA) |
| 855 |
dapltest: increase DTO evd size to prevent CQ overflow on limit_rpost test |
| 856 |
dapltest: RSP limit test fails. Creation of reserved SP moves EP state to DAT_EP_STATE_RESERVED in error cases. |
| 857 |
dapl: fix string bug in dapls_dto_op_str |
| 858 |
|
| 859 |
Release 2.0.41 fixes (OFED 3.12 RC1) |
| 860 |
dapltest: change server port, from 45278 to 62000, out of registered IANA range |
| 861 |
dat: lower log level on load errors of provider library |
| 862 |
dat: dat_ia_open should close provider after failure |
| 863 |
dapltest: set default limit max to 1000 |
| 864 |
openib: add new provider specific attributes |
| 865 |
dapltest: update scripts for regression testing purposes |
| 866 |
dapltest: Add final send/recv "sync" for transaction tests. |
| 867 |
|
| 868 |
Release 2.0.40 fixes (OFED 3.12) |
| 869 |
dist: ib collective extension include files missing |
| 870 |
dapltest: the quit command is missing changes for -n option |
| 871 |
dat.conf: remove v1, add Mellanox Connect-IB and Intel Xeon Phi MIC |
| 872 |
NULL undefined on Fedora, incorrectly using kernel stddef.h |
| 873 |
|
| 874 |
Release 2.0.39 fixes (OFED 3.5-2 GA) |
| 875 |
dapltest: fix endian swap issue with performance test |
| 876 |
scm: getifaddrs modfications for better out of the box experience |
| 877 |
ucm, scm: UD mode triggers list_head assert with large scale alltoall test |
| 878 |
|
| 879 |
Release 2.0.38 |
| 880 |
dapltest: add -n parameter to override default server port number (45278) |
| 881 |
ucm,scm: UD mode creates many CR objects per EP that needs cleaned up |
| 882 |
cma: add DAPL_CM_TOS environment variable to enable passing a TOS to the RDMA CM |
| 883 |
|
| 884 |
Release 2.0.37 |
| 885 |
common: add support for ia name during dat_ia_query |
| 886 |
common: dapl_os_atomic_inc/dec() not working as expected on ppc64 machines. |
| 887 |
dapltest: ppc64 endian issue with exchanged mem handle and address |
| 888 |
|
| 889 |
Release 2.0.36 |
| 890 |
scm: increase ACK timeout to 20 for a default value to match other providers. |
| 891 |
common: allow qp modify in init state |
| 892 |
common: check for valid states during ep posting |
| 893 |
dat.conf: keep list of providers in order for backward compatibility |
| 894 |
ucm: record and silently drop a duplicate reject CM message |
| 895 |
windows: new version of getlocalipaddr not portable |
| 896 |
dapltest: DFLT_QLEN is defined in multiple tests |
| 897 |
|
| 898 |
Release 2.0.35 |
| 899 |
config/build: remove post/postun hacking used to modify dat.conf |
| 900 |
config: clean up help option displays with ext-type options |
| 901 |
windows: Provide auto-detect between RoCE and Infiniband for Windows. |
| 902 |
ucm: update UD cm provider to support new CM stat and error counters |
| 903 |
scm: update socket cm provider to support new CM stat and error counters |
| 904 |
commom: add cm, link, and diag event counters in IB extended builds |
| 905 |
scm: use ioctl SIOCIFCONF to get complete list of configured netdev interfaces |
| 906 |
ucm: UD send failures at scale, ucm_send ERR: get_smsg(hd=149,tl=150) |
| 907 |
scm: fix retry count on connection pending timeout |
| 908 |
ucm: cleanup debug message, ntohl on p_size is incorrect |
| 909 |
cma, scm, ucm: allow EP (QP) creation without EVD (CQ) |
| 910 |
common: add DAPL_DBG_TYPE_CM_STATS (0x40000) to debug log options |
| 911 |
common: dapls_ep_flush_cq will segfault when no CQ is attached to EP |
| 912 |
common: ep_create should allow max_request_iov attribute setting of zero |
| 913 |
common: add check for NULL handle on ext calls, SRQ free, and helper functions |
| 914 |
common: add missing sub-types to dat_strerror() |
| 915 |
common: extended CR event processing missing rejects on errors |
| 916 |
ucm: incorrectly sends user reject during CR callback errors |
| 917 |
common: change dbg level on CR callback if not listening on SP |
| 918 |
scm: incorrectly sends user reject during CR callback errors |
| 919 |
dat: add check for NULL handle on IA calls |
| 920 |
cma,scm,ucm: extra reference on EP, with RSP, causes dat_ep_free() to hang |
| 921 |
common: RSP service points incorrectly freed during CR callback |
| 922 |
common: clean up dat_rsp_create log message |
| 923 |
common: cleanup debug message on EVD overflows |
| 924 |
scm: return correct event error code when remote host refuses requests |
| 925 |
dapltest: server CR EVD is too small for multi-client configurations. |
| 926 |
Common: CR EVD overflow causes segfault. |
| 927 |
|
| 928 |
Release 2.0.34 |
| 929 |
scm: change debug message level for listen/bind errors |
| 930 |
common: increase default IB ack timer from 16 to 20 |
| 931 |
common: remote ia address null pointer creates seg fault |
| 932 |
common: posting events on full queue returns wrong error code |
| 933 |
common: dat_ep_modify seg faults with null ep_param ptr |
| 934 |
common: dat_evd_free seg faults with resized software EVD |
| 935 |
common: remove assert for incorrect events during cm_request |
| 936 |
dat: dat_cno_query with NULL cno_handle causes segmentation fault |
| 937 |
scm: dat_psp_create returns wrong error code on bind/listen failure |
| 938 |
scm: socket connect request count is reset improperly on retry |
| 939 |
scm: when hostname has loopback addr assigned, default to eth0 instead of failing |
| 940 |
scm: add port number to error log during hca_open failures |
| 941 |
common: query calls return incorrect IA handle to consumer |
| 942 |
common: srq create asserts with !dapl_llist_is_empty(head) failed |
| 943 |
|
| 944 |
Release 2.0.33 |
| 945 |
scm,ucm: fix compatibility issues and set minimum protocol support |
| 946 |
build: link librdmacm dependency to ib_acm usage for ucm and scm providers |
| 947 |
build: add selective enable/disable-xxx build switch for each provider |
| 948 |
build: add extended header files to EXTRA_DIST and fix missing backslash |
| 949 |
build: set IB extended coll-type to none by default |
| 950 |
common: change errno mapping of EINVAL to DAT_INVALID_PARAMETER |
| 951 |
build: add IB collective and FCA provider to dapl build package as an option |
| 952 |
common: add new dapls_evd_post_event_ext call for extended events |
| 953 |
ucm: add support for IB collective providers |
| 954 |
scm: add support for IB collective providers |
| 955 |
cma: add support for IB collective providers |
| 956 |
common: add supported collective types in named attributes for query |
| 957 |
common: add collective call mappings via standard dapli_post_ext() |
| 958 |
common: new debug bitmask definition for extension logging |
| 959 |
common: new IB collective provider for Mellanox Fabric Collective Agent |
| 960 |
dat: add definitions for MPI offloaded collectives in IB transport extensions |
| 961 |
common: cleanup debug messages when building with ibacm feature |
| 962 |
|
| 963 |
Release 2.0.32 fixes (OFED 1.5.3 GA): |
| 964 |
|
| 965 |
cma: reduce output log level in disconnect from WARN to CM_WARN |
| 966 |
ucm: delay freeing of active side UD cm object in case RTU is dropped |
| 967 |
ucm: cm object needs to be on work queue before req sent on wire |
| 968 |
ucm,scm: remove use of usec_sleep delays and use events for disc and destroy |
| 969 |
common: reduce default max inline data size because of performance anomaly |
| 970 |
common: dapls_evd_dto_wait() dbg message should print status and not errno |
| 971 |
ucm, scm: exchange max_qp_rd_atom and limit outstanding requests |
| 972 |
scm: retry socket connect on ECONNREFUSED under heavy load |
| 973 |
common: qp modify RTR using wrong ep attribute parameter for dest_rd_atomic |
| 974 |
|
| 975 |
Release 2.0.31 fixes (OFED 1.5.3 RC1): |
| 976 |
|
| 977 |
common: clean up build warning for unused variable event_ptr |
| 978 |
scm, ucm: set RAI_NOROUTE flag with rdma_getaddrinfo() call to avoid blocking. |
| 979 |
cma: definition for dapl_sp_remove_ep() is missing in cm.c |
| 980 |
libdat: static provider entries created for local SR database not freed |
| 981 |
libdat: memory leak in static registration during parsing |
| 982 |
common: increase default IB inline send threshold to 400 |
| 983 |
common cq: a mixup of errno and the -1 return from poll in dapls_wait_comp_channel |
| 984 |
ucm: release UD cm objects after AH is exchanged to avoid duplicate request drops |
| 985 |
ucm: decrease timeout retry count for disconnect requests |
| 986 |
ucm: hold lock when sending cm_msgs to sync timer start with packet send |
| 987 |
ucm: add debugging to include process id for better scale up debug aids |
| 988 |
cma: disconnect can block for excessive times waiting for rdma_cm DREP timeout |
| 989 |
ucm: configure the recv channel FD to non-blocking |
| 990 |
windows: Missing librdmacm include path for build |
| 991 |
debug build: only timestamp if sending to stdout to avoid performance hit |
| 992 |
common: print out errors on free build and not just debug builds |
| 993 |
cma: fix debug build issue |
| 994 |
scm, ucm: MPI spawn test on oversubcribed server taking excessive time to complete |
| 995 |
common: add high resolution time stamps and thread id to sdtout debug logs |
| 996 |
common: modify debug in dat_evd_dequeue to reduce noise, only output on non-empty |
| 997 |
cma: rdma_destroy_id called twice during device open bind error |
| 998 |
common: dat_evd_dequeue (poll_cq) fails with invalid parameter after EP (qp) free |
| 999 |
ucm: allow configuration of CM burst (signal) threshold on posting |
| 1000 |
cma: fix debug build |
| 1001 |
windows: debug version of windows does not build. |
| 1002 |
Allow DAPL out of band connection models to use ibacm to obtain path record data. |
| 1003 |
ucm: add missing map file for UCM provider |
| 1004 |
ibal: delay QP transition during disconnect phase |
| 1005 |
Revert "ibal: delay QP transition during disconnect phase" |
| 1006 |
ibal: delay QP transition during disconnect phase |
| 1007 |
common: restructure EVD processing to handle EP destruction phase |
| 1008 |
ibal: sync QP destruction and device close |
| 1009 |
ucm: remove unnecessary debug warning in async callback |
| 1010 |
|
| 1011 |
v1.2 Package: |
| 1012 |
|
| 1013 |
Release 1.2.19 fixes (OFED 1.5.2 GA): |
| 1014 |
|
| 1015 |
common, cma: disconnect and cleanup CR linkings after DTO error on EP |
| 1016 |
common: race conditions with DTO error, disconnect and dapl_reset_ep |
| 1017 |
common: add new dapl_os_sleep_usec() function |
| 1018 |
configure: need a false conditional for verbs attr.link_layer member check |
| 1019 |
config: add conditional check for new verbs port_attr.link_layer |
| 1020 |
cma, scm: new provider entries for Mellanox RDMA over Ethernet device for uDAPL v1.2 |
| 1021 |
cma: memory leak of verbs CQ and completion channels created during dat_ia_open |
| 1022 |
cma: memory leak of FD's (pipe) created during dat_evd_create |
| 1023 |
|
| 1024 |
|
| 1025 |
--- HISTORY ----------- |
| 1026 |
|
| 1027 |
OFED 1.5.1 RELEASE NOTES |
| 1028 |
uDAPL v1 (1.2.16-1) and v2 (2.0.27-1) |
| 1029 |
|
| 1030 |
---------------- |
| 1031 |
|
| 1032 |
* New Features (v2 only) - UCM provider with IB UD based CM per process. |
| 1033 |
More scalable then rdma_cm (cma) or socket cm (scm). |
| 1034 |
---------------- |
| 1035 |
|
| 1036 |
* Bug Fixes |
| 1037 |
|
| 1038 |
V2.0 Package |
| 1039 |
|
| 1040 |
Release 2.0.27 |
| 1041 |
windows: add scm makefile |
| 1042 |
windows does not require rdma_cma_abi.h, move the include from common code |
| 1043 |
windows patch to fix IB_INVALID_HANDLE name collision |
| 1044 |
scm: dat_ep_connect fails on 32bit servers |
| 1045 |
undefined symbol: dapls_print_cm_list |
| 1046 |
cleanup CM object lock before freeing CM object memory |
| 1047 |
destroy verbs completion channels created via ia_open or ep_create. |
| 1048 |
package: update Copyright file and include the 3 license files in distribution |
| 1049 |
common: when copying private_data out of rdma_cm events, use the |
| 1050 |
cma: fix referencing freed address |
| 1051 |
dapl: move close device after async thread is done |
| 1052 |
|
| 1053 |
Release 2.0.26 |
| 1054 |
openib_common: add check for both gid and global routing in RTR |
| 1055 |
openib_common: remote memory read privilege set multi times |
| 1056 |
ucm, scm: DAPL_GLOBAL_ROUTING enabled causes segv |
| 1057 |
|
| 1058 |
Release 2.0.25 |
| 1059 |
winof scm: initialize opt for NODELAY setsockopt |
| 1060 |
winof cma: windows definition for EADDRNOTAVAIL missing |
| 1061 |
scm: client side setsockopt NODELAY fails if data arrives before setting |
| 1062 |
cma: setup_listener Cannot assign requested address |
| 1063 |
common: seg fault in dapl_evd_wait with multi-thread application using CNO's. |
| 1064 |
ucm: inbound DREQ/DREP handshake should transition QP. |
| 1065 |
winof: Remove duplicate include of comp_channel.cpp from cm.c as it is |
| 1066 |
included in opensm_ucb/device.c. |
| 1067 |
|
| 1068 |
Release 2.0.24 |
| 1069 |
winof: Utilize WinOF version of inet_ntop() for Windows OSes which do not |
| 1070 |
support inet_ntop(). |
| 1071 |
ucm: windows build issue with new CQ completion channel |
| 1072 |
winof: add ucm provider to windows build |
| 1073 |
winof: add missing build files for ibal, scm |
| 1074 |
scm: connection peer resets under heavy load, incorrect event on error |
| 1075 |
ucm: increase default reply and rtu timeout values. |
| 1076 |
ucm: change some debug message levels and add check for valid UD REPLY during retries. |
| 1077 |
ucm: increase timers during subsequent retries |
| 1078 |
ucm, scm: address handles need destroyed when freeing Endpoints with UD QP's. |
| 1079 |
openib_common: ignore pd free errors, clear pd_handle and return. |
| 1080 |
ucm: using UD type QP's, ucm reports wrong reject event when user rejects AH resolution request. |
| 1081 |
ucm, scm, cma: Fix CNO support on DTO type EVD's |
| 1082 |
ucm: fix lock init bug in ucm_cm_find |
| 1083 |
ucm: fix build problem with latest windows ucm changes |
| 1084 |
ucm: The HCA should not be closed until all resources have been released. |
| 1085 |
ucm: Fix build warning when compiling on 32-bit systems. |
| 1086 |
ucm: Trying to deregister the same memory region twice leads to an |
| 1087 |
dat: reduce debug message level when parsing for location of dat.conf |
| 1088 |
ucm: update ucm provider for windows environment |
| 1089 |
ucm: add timer/retry CM logic to the ucm provider |
| 1090 |
|
| 1091 |
Release 2.0.23 |
| 1092 |
cma: cannot reuse the cm_id and qp for new connection, must reallocate a new one. |
| 1093 |
scm, cma: update DAPL cm protocol revision with latest address/port changes |
| 1094 |
ucm: modify IB address format to align better with sockaddr_in6 |
| 1095 |
Add definition for getpid similar to that used by the other dtest apps. |
| 1096 |
WinOF provides a common implementation of gettimeofday that should |
| 1097 |
The completion manager was updated to provide an abstraction that |
| 1098 |
dtestcm: remove IB verb definitions |
| 1099 |
dtest, dtestx: remove IB verb definitions |
| 1100 |
scm: tighten up socket options to insure similiar behavior on Windows and Linux. |
| 1101 |
cma: improve serialization of destroy and event processing |
| 1102 |
scm: improve serialization of destroy and state changes |
| 1103 |
common: no cleanup/release code for timer thread |
| 1104 |
scm, cma: dapli_thread doesn't always get teminated on library close. |
| 1105 |
ucm: tighten up locking with CM processing, state changes |
| 1106 |
ucm: For UD type QP's, return CR p_data with CONN_EST event on passive side. |
| 1107 |
ucm: cleanup extra cr/lf |
| 1108 |
ucm: fix issues with UD QP's. |
| 1109 |
winof: Convert windows version of dapl and dat libaries to use private heaps. |
| 1110 |
dtest, dtestx: modifications for UD QP testing with ucm provider. |
| 1111 |
scm, ucm: UD QP support was broken when porting to common openib code base. |
| 1112 |
cma: cleanup warning with unused local variable, ret, in disconnect |
| 1113 |
cma: remove debug message after rdma_disconnect failure |
| 1114 |
scm: socket errno check needs O/S dependent wrapper |
| 1115 |
dapltest: update script files for WinOF |
| 1116 |
cma: conditional check for new rdma_cm definition. |
| 1117 |
|
| 1118 |
Release 2.0.22 |
| 1119 |
dapltest: add mdep processor yield and use with dapltest |
| 1120 |
ucm: Add new provider using a DAPL based IB-UD cm mechanism for MPI implementations. |
| 1121 |
|
| 1122 |
Release 2.0.21 |
| 1123 |
scm: Fix disconnect. QP's need to move to ERROR state in |
| 1124 |
modify dtest.c to cleanup CNO wait code and consolidate into |
| 1125 |
CNO events, once triggered will not be returned during the cno wait. |
| 1126 |
CNO support broken in both CMA and SCM providers. |
| 1127 |
common osd: include winsock2.h for IPv6 definitions. |
| 1128 |
common osd: include w2tcpip.h for sockaddr_in6 definitions. |
| 1129 |
DAPL introduced the concept of directly waiting on the CQ for |
| 1130 |
dapltest: Implement a malloc() threshold for the completion reaping. |
| 1131 |
scm: handle connected state when freeing CM objects |
| 1132 |
scm, dtest: changes for winof gettimeofday and FD_SETSIZE settings. |
| 1133 |
scm: set TCP_NODELAY sockopt on the server side for sends. |
| 1134 |
remove obsolete files in dapl/udapl source tree |
| 1135 |
dtestcm: add UD type QP option to test |
| 1136 |
scm: destroy QP called before disconnect |
| 1137 |
cma: add support for rdma_cm TIME_WAIT event. |
| 1138 |
scm: remove old udapl_scm code replaced by openib_scm. |
| 1139 |
winof: fix issues after consolidating cma, scm code base. |
| 1140 |
cma: lock held when exiting as a result of a rdma_create_event_channel failure. |
| 1141 |
windows: all dlist functions have been moved to the header file. |
| 1142 |
dtestcm windows: add build infrastructure for new dtestcm test suite |
| 1143 |
openib_common: reorganize code base to share common mem, cq, qp, dto functions |
| 1144 |
scm: fixes and optimizations for connection scaling |
| 1145 |
scm: double the default fd_set_size |
| 1146 |
scm: EP reference in CR should be cleared during ep_destroy |
| 1147 |
dtestx: fix conn establishment event checking |
| 1148 |
dtestcm: new test to measure dapl connection rates. |
| 1149 |
|
| 1150 |
Release 2.0.20 |
| 1151 |
common,scm: add debug capabilities to print in-process CM lists |
| 1152 |
scm: disconnect EP before cleaning up orphaned CR's during dat_ep_free |
| 1153 |
dapltest: windows scripts updated |
| 1154 |
scm: private data is not handled properly via CR rejects. |
| 1155 |
scm: cleanup orphaned UD CR's when destroying the EP |
| 1156 |
scm: provider specific query for default UD MTU is wrong. |
| 1157 |
scm: update CM code to shutdown before closing socket |
| 1158 |
dapltest: windows script dt-cli.bat updated |
| 1159 |
dapl/windows cma provider: add support for network devices based on index |
| 1160 |
openib: remove 1st gen provider, replaced with openib_cma and openib_scm |
| 1161 |
dapltest: update windows script files |
| 1162 |
dapltest: windows batch files in sripts directory |
| 1163 |
windows_osd/linux_osd: new dapl_os_gettid macro to return thread id |
| 1164 |
windows: missing build files for common and udapl sub-directories |
| 1165 |
windows: add build files for openib_scm, remove /Wp64 build option. |
| 1166 |
scm: multi-hca CM processing broken. Need cr thread wakeup mechanism per HCA. |
| 1167 |
dtest: add connection timers on client side |
| 1168 |
linux_osd: use pthread_self instead of getpid for debug messages |
| 1169 |
windows ibal-scm: dapl/dirs file needs updated to remove ibal-scm |
| 1170 |
|
| 1171 |
v1.2 Package: |
| 1172 |
|
| 1173 |
Release 1.2.16 |
| 1174 |
package: update Copyright file and include the 3 license files in distribution |
| 1175 |
cma: max sge incorrectly decremented during ibv_device_query |
| 1176 |
|
| 1177 |
Release 1.2.15 |
| 1178 |
dtest, dapltest: conflict with dapl-2 utils package, change to dapl1, dapltest1 |
| 1179 |
scm: fix compiler warning, unused variable |
| 1180 |
|
| 1181 |
---------------- |
| 1182 |
|
| 1183 |
* BKM for running new DAPL library on your cluster without any impact on existing OFED installation: |
| 1184 |
|
| 1185 |
Note: example for user /home/user1, (assumes /home/user1 is exported) and MLX4 adapter, port 1 |
| 1186 |
|
| 1187 |
Download latest 2.x package: http://www.openfabrics.org/downloads/dapl/dapl-2.0.25.tar.gz |
| 1188 |
|
| 1189 |
untar in /home/user1 |
| 1190 |
cd /home/user1/dapl-2.0.25 |
| 1191 |
./configure && make (build on node with OFED 1.3 or higher installed, dependency on verb/rdma_cm libraries) |
| 1192 |
|
| 1193 |
create /home/user1/dat.conf with following 3 lines. (entries with path to new libraries): |
| 1194 |
|
| 1195 |
ofa-v2-ib0 u2.0 nonthreadsafe default /home/user1/dapl-2.0.19/dapl/udapl/.libs/libdaplcma.so.1 dapl.2.0 "ib0 0" "" |
| 1196 |
ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default /home/user1/dapl-2.0.19/dapl/udapl/.libs/libdaploscm.so.2 dapl.2.0 "mlx4_0 1" "" |
| 1197 |
ofa-v2-mlx4_0-1u u2.0 nonthreadsafe default /home/user1/dapl-2.0.19/dapl/udapl/.libs/libdaploucm.so.2 dapl.2.0 "mlx4_0 1" "" |
| 1198 |
|
| 1199 |
Run uDAPL application or an MPI that uses uDAPL, with (assuming MLX4 connectx adapters) following: |
| 1200 |
|
| 1201 |
setenv DAT_OVERRIDE=/home/user1/dat.conf |
| 1202 |
|
| 1203 |
If running Intel MPI and uDAPL socket cm, set the following: |
| 1204 |
|
| 1205 |
setenv I_MPI_DEVICE=rdssm:ofa-v2-mlx4_0-1 |
| 1206 |
|
| 1207 |
or if running Intel MPI and uDAPL IB UD cm, set the following: |
| 1208 |
|
| 1209 |
setenv I_MPI_DEVICE=rdssm:ofa-v2-mlx4_0-1u |
| 1210 |
|
| 1211 |
or if running Intel MPI and uDAPL rdma_cm, set the following: |
| 1212 |
|
| 1213 |
setenv I_MPI_DEVICE=rdssm:ofa-v2-ib0 |
| 1214 |
|
| 1215 |
------------------------- |
| 1216 |
|
| 1217 |
OFED 1.4.1 RELEASE NOTES |
| 1218 |
|
| 1219 |
NEW SINCE OFED 1.4 - new versions of uDAPL v1 (1.2.14-1) and v2 (2.0.19-1) |
| 1220 |
|
| 1221 |
* New Features - optional counters, must be configured/built with -DDAPL_COUNTERS |
| 1222 |
|
| 1223 |
* Bug Fixes |
| 1224 |
|
| 1225 |
v2 - scm, cma: dat max_lmr_block_size is 32 bit, verbs max_mr_size is 64 bit |
| 1226 |
v2 - scm, cma: use direct SGE mappings from dat_lmr_triplet to ibv_sge |
| 1227 |
v2 - dtest: add flush EVD call after data transfer errors |
| 1228 |
v2 - scm: increase default MTU size from 1024 to 2048 |
| 1229 |
v2 - dapltest: reset server listen ports to avoid collisions during long runs |
| 1230 |
v2 - dapltest: avoid duplicating ports, increment based on ep/thread count |
| 1231 |
v2 - dapltest: fix assumptions that multiple EP's will connect in order |
| 1232 |
v2 - common: sync missing with when removing items off of EVD pending queue |
| 1233 |
v2 - scm: reduce open time with thread start up |
| 1234 |
v2 - scm: getsockopt optlen needs initialized to size of optval |
| 1235 |
v2 - scm: cr_thread cleanup |
| 1236 |
v2 - OFED and WinOF code sync |
| 1237 |
v2 - scm: remove unnecessary query gid/lid from connection phase code. |
| 1238 |
v2 - scm: add optional 64-bit counters, build with -DDAPL_COUNTERS. |
| 1239 |
v1,v2 - spec files missing Requires(post) statements for sed/coreutils |
| 1240 |
v1,v2 - dtest/dapltest: use $(top_builddir) for .la files during test builds |
| 1241 |
v1,v2 - scm: remove unecessary thread when using direct objects |
| 1242 |
v1,v2 - Fix SuSE 11 build issues, asm/atomic.h no longer exists |
| 1243 |
|
| 1244 |
* Build Notes: |
| 1245 |
|
| 1246 |
# NON_DEBUG build/install example for x86_64, OFED targets |
| 1247 |
./configure --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include" |
| 1248 |
make install |
| 1249 |
|
| 1250 |
# DEBUG build/install example for x86_64, using OFED targets |
| 1251 |
./configure --enable-debug --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include" |
| 1252 |
make install |
| 1253 |
|
| 1254 |
# COUNTERS build/install example for x86_64, using OFED targets |
| 1255 |
./configure --prefix /usr --sysconf=/etc --libdir /usr/lib64 LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include -DDAPL_COUNTERS" |
| 1256 |
make install |
| 1257 |
|
| 1258 |
* BKM for running new DAPL library on your cluster without any impact on existing OFED installation: |
| 1259 |
|
| 1260 |
Note: example for user /home/user1, (assumes /home/user1 is exported) and MLX4 adapter, port 1 |
| 1261 |
|
| 1262 |
Download latest 2.x package: http://www.openfabrics.org/downloads/dapl/dapl-2.0.19.tar.gz |
| 1263 |
|
| 1264 |
untar in /home/user1 |
| 1265 |
cd /home/user1/dapl-2.0.19 |
| 1266 |
./configure && make (build on node with OFED 1.3 or higher installed, dependency on verb/rdma_cm libraries) |
| 1267 |
|
| 1268 |
create /home/user1/dat.conf with following 2 lines. (entries with path to new libraries): |
| 1269 |
|
| 1270 |
ofa-v2-ib0 u2.0 nonthreadsafe default /home/user1/dapl-2.0.19/dapl/udapl/.libs/libdaplcma.so.1 dapl.2.0 "ib0 0" "" |
| 1271 |
ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default /home/user1/dapl-2.0.19/dapl/udapl/.libs/libdaploscm.so.2 dapl.2.0 "mlx4_0 1" "" |
| 1272 |
|
| 1273 |
Run uDAPL application or an MPI that uses uDAPL, with (assuming MLX4 connectx adapters) following: |
| 1274 |
|
| 1275 |
setenv DAT_OVERRIDE=/home/user1/dat.conf |
| 1276 |
|
| 1277 |
If running Intel MPI and uDAPL socket cm, set the following: |
| 1278 |
|
| 1279 |
setenv I_MPI_DEVICE=rdssm:ofa-v2-mlx4_0-1 |
| 1280 |
|
| 1281 |
if running Intel MPI and uDAPL rdma_cm, set the following: |
| 1282 |
|
| 1283 |
setenv I_MPI_DEVICE=rdssm:ofa-v2-ib0 |
| 1284 |
|
| 1285 |
------------------------- |
| 1286 |
|
| 1287 |
OFED 1.4 RELEASE NOTES |
| 1288 |
|
| 1289 |
NEW SINCE OFED 1.3.1 - new versions of uDAPL v1 (1.2.12-1) and v2 (2.0.15-1) |
| 1290 |
|
| 1291 |
* New Features |
| 1292 |
|
| 1293 |
1. The new socket CM provider, introduced in 1.2.8 and 2.0.11 packages, |
| 1294 |
assumes homogeneous cluster and will setup the QP's based on local HCA port |
| 1295 |
attributes and exchanges QP information via socket's using the hostname of |
| 1296 |
each node. IPoIB and rdma_cm are NOT required for this provider. QP attributes |
| 1297 |
can be adjusted via the following environment parameters: |
| 1298 |
|
| 1299 |
DAPL_ACK_TIMER (default=16 5 bits, 4.096us*2^ack_timer. 16 == 268ms) |
| 1300 |
DAPL_ACK_RETRY (default=7 3 bits, 7 * 268ms = 1.8 seconds) |
| 1301 |
DAPL_RNR_TIMER (default=12 5 bits, 12 == 64ms, 28 == 163ms, 31 == 491ms) |
| 1302 |
DAPL_RNR_RETRY (default=7 3 bits, 7 == infinite) |
| 1303 |
DAPL_IB_MTU (default=1024 limited to active MTU max) |
| 1304 |
|
| 1305 |
The new socket cm entries in /etc/dat.conf provide a link to the actual HCA |
| 1306 |
device and port. Example v1 and v2 entries for a Mellanox connectx device, port 1: |
| 1307 |
|
| 1308 |
OpenIB-mlx4_0-1 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "mlx4_0 1" "" |
| 1309 |
ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 1" "" |
| 1310 |
|
| 1311 |
This new socket cm provider, was successfully tested on the TATA CRL cluster |
| 1312 |
(#8 on Top500) with Intel MPI, achieving a HPLinpack score of 132.8TFlops on |
| 1313 |
1798 nodes, 14384 cores at ~76.9% of peak. DAPL_ACK_TIMER was increased to 21 |
| 1314 |
for this scale. |
| 1315 |
|
| 1316 |
2. New v2 definitions for IB unreliable datagram extension (only supported in |
| 1317 |
scm provider, libdaploscm.so.2) |
| 1318 |
|
| 1319 |
Extended EP dat_service_type, with DAT_IB_SERVICE_TYPE_UD |
| 1320 |
Add IB extension call dat_ib_post_send_ud(). |
| 1321 |
Add address handle definition for UD calls. |
| 1322 |
Add IB event definitions to provide remote AH via connect and connect requests |
| 1323 |
See dtestx (-d) source for example usage model |
| 1324 |
|
| 1325 |
* Bug Fixes |
| 1326 |
|
| 1327 |
v1,v2 - dapltest: trans test moves to cleanup stage before rdma_read processing is complete |
| 1328 |
v1,v2 - Fix static registration (dat.conf) to include sysconfdir override |
| 1329 |
v1,v2 - dat.conf: add default iwarp entry for eth2 |
| 1330 |
v1,v2 - dapl: adjust max_rdma_read_iov to 1 for iWARP devices |
| 1331 |
v1,v2 - dtest: reduce default IOV's for ep_create to support iWARP |
| 1332 |
v1,v2 - dtest: fix 32-bit build issues |
| 1333 |
v1,v2 - build: $(DESTDIR) prepend needed on install hooks for dat.conf |
| 1334 |
v2 - scm: UD shares EP;s which requires serialization |
| 1335 |
v2 - dapl: fixes for IB UD extensions in common code and socket cm provider. |
| 1336 |
v2 - dapl: add provider specific attribute query option for IB UD MTU size |
| 1337 |
v2 - dapl build: add correct CFLAGS, set non-debug build by default for v2 |
| 1338 |
v2 - dtestx: fix stack corruption problem with hostname strcpy |
| 1339 |
v2 - dapl extension: dapli_post_ext should always allocate cookie for requests. |
| 1340 |
v2 - dapltest: manpage - rdma write example incorrect |
| 1341 |
v1,v2 - dat, dapl, dtest, dapltest, providers: fix compiler warnings in dat common code |
| 1342 |
v1,v2 - dapl cma: debug message during query needs definition for inet_ntoa |
| 1343 |
v1,v2 - dapl scm: fix corner case that delivers duplicate disconnect events |
| 1344 |
v1,v2 - dat: include stddef.h for NULL definition in dat_platform_specific.h |
| 1345 |
v1,v2 - dapl: add debug messages during async and overflow events |
| 1346 |
v1,v2 - dapltest: add check for duplicate disconnect events in transaction test |
| 1347 |
v1,v2 - dapl scm: use correct device attribute for max_rdma_read_out, max_qp_init_rd_atom |
| 1348 |
v1,v2 - dapl scm: change IB RC qp inline and timer defaults. |
| 1349 |
v1,v2 - dapl scm: add mtu adjustments via environment, default = 1024. |
| 1350 |
v1,v2 - dapl scm: change connect and accept to non-blocking to avoid blocking user thread. |
| 1351 |
v1,v2 - dapl scm: update max_rdma_read_iov, max_rdma_write_iov EP attributes during query |
| 1352 |
v1,v2 - dat: allow TYPE_ERR messages to be turned off with DAT_DBG_TYPE |
| 1353 |
v1,v2 - dapl: remove needless terminating 0 in dto_op_str functions. |
| 1354 |
v1,v2 - dat: remove reference to doc/dat.conf in makefile.am |
| 1355 |
v1,v2 - dapl scm: fix ibv_destroy_cq busy error condition during dat_evd_free. |
| 1356 |
v1,v2 - dapl scm: add stdout logging for uname and gethostbyname errors during open. |
| 1357 |
v1,v2 - dapl scm: support global routing and set mtu based on active_mtu |
| 1358 |
v1,v2 - dapl: add opcode to string function to report opcode during failures. |
| 1359 |
v1,v2 - dapl: remove unused iov buffer allocation on the endpoint |
| 1360 |
v1,v2 - dapl: endpoint pending request count is wrong |
| 1361 |
|
| 1362 |
------------------------- |
| 1363 |
|
| 1364 |
OFED 1.3.1 RELEASE NOTES |
| 1365 |
|
| 1366 |
NEW SINCE OFED 1.3 - new versions of uDAPL v1 (1.2.7-1) and v2 (2.0.9-1) |
| 1367 |
|
| 1368 |
* New Features - None |
| 1369 |
|
| 1370 |
* Bug Fixes |
| 1371 |
v2 - add private data exchange with reject |
| 1372 |
v1,v2 - better error reporting in non-debug builds |
| 1373 |
v1,v2 - update only OFA entries in dat.conf, cooperate with non-ofa providers |
| 1374 |
v1,v2 - support for zero byte operations, iov==NULL |
| 1375 |
v1,v2 - multi-transport support for inline data and private data differences |
| 1376 |
v1,v2 - fix memory leaks and other reported bugs since OFED 1.3 |
| 1377 |
v1,v2 - dtest,dtestx,dapltest build issues on RHEL5.1 |
| 1378 |
v1,v2 - long delay during dat_ia_open when DNS not configured |
| 1379 |
v1,v2 - use rdma_read_in/out from ep_attr per consumer instead of HCA max |
| 1380 |
|
| 1381 |
------------------------- |
| 1382 |
|
| 1383 |
OFED 1.3 RELEASE NOTES |
| 1384 |
|
| 1385 |
NEW SINCE OFED 1.2 |
| 1386 |
|
| 1387 |
* New Features |
| 1388 |
|
| 1389 |
1. Add v2.0 library support for new 2.0 API Specification |
| 1390 |
2. Separate v1.2 library release to co-exist with v2.0 libraries. |
| 1391 |
3. New dat.conf with both 1.2 and 2.0 support |
| 1392 |
4. New v2.0 dtestx utilities to test IB extensions |
| 1393 |
|
| 1394 |
* Bug Fixes |
| 1395 |
|
| 1396 |
v1.2 and v2.0 |
| 1397 |
- uDAT: static/dynamic registry parsing fixes |
| 1398 |
- uDAPL: provider fixes for dat_psp_create_any |
| 1399 |
- dtest/dapltest: change default provider names to sync with dat.conf |
| 1400 |
- openib_cma: issues with destroy_cm_id and init/resp exchange |
| 1401 |
- dapltest: use gettimeofday instead of get_cycles for better portability |
| 1402 |
- dapltest: endian issue with mem_handle, mem_address |
| 1403 |
- dapltest fix to include inet_ntoa definitions |
| 1404 |
- fix build problems on 32-bit and 64-bit PowerPC |
| 1405 |
- cleanup packaging |
| 1406 |
|
| 1407 |
v2.0 |
| 1408 |
- set default config options to match spec file, --enable-debug --enable-ext-type=ib |
| 1409 |
- use unique devel target names, libdat2.so, /usr/include/dat2 |
| 1410 |
- dtestx fix memory leak, freeaddrinfo after getaddrinfo |
| 1411 |
- Fix for IB extended DTO cookie deallocation on inbound rdma_Write_immed |
| 1412 |
- WinOF: Update OFED code base to include WinOF changes, work from same code base |
| 1413 |
- WinOF: add DAT_API definition, __stdcall for windows, nothing for linux |
| 1414 |
- dtest: add dat_evd_query to check correct size |
| 1415 |
- openib_cma: add macro to convert SID to PORT |
| 1416 |
- dtest: endian support for exchanging RMR info |
| 1417 |
- openib_cma: lower default settings, inline and RDMA init/resp |
| 1418 |
- openib_cma: missing ia_query for max_iov_segments_per_rdma_write |
| 1419 |
|
| 1420 |
v1.2 |
| 1421 |
- openib_cma: turn down dbg noise level on rejects |
| 1422 |
- dtest: typo in memset |
| 1423 |
|
| 1424 |
|
| 1425 |
BUILD: v1 and v2 uDAPL source install/build instructions (redhat example): |
| 1426 |
|
| 1427 |
# cd to distribution SRPMS directory |
| 1428 |
cd /tmp/OFED-1.3/SRPMS |
| 1429 |
rpm -i dapl-1.2*.rpm |
| 1430 |
rpm -i dapl-2.0*.rpm |
| 1431 |
cd /usr/src/redhat/SOURCES |
| 1432 |
tar zxf dapl-1.2*.tgz |
| 1433 |
tar zxf dapl-2.0*.tgz |
| 1434 |
|
| 1435 |
# NON_DEBUG build example for x86_64, using OFED targets |
| 1436 |
|
| 1437 |
./configure --prefix /usr --sysconf=/etc --libdir /usr/lib64 |
| 1438 |
LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include" |
| 1439 |
|
| 1440 |
# build and install |
| 1441 |
|
| 1442 |
make |
| 1443 |
make install |
| 1444 |
|
| 1445 |
# DEBUG build example for x86_64, using OFED targets |
| 1446 |
|
| 1447 |
./configure --enable-debug --prefix /usr --sysconf=/etc --libdir /usr/lib64 |
| 1448 |
LDFLAGS=-L/usr/lib64 CPPFLAGS="-I/usr/include" |
| 1449 |
|
| 1450 |
# build and install |
| 1451 |
|
| 1452 |
make |
| 1453 |
make install |
| 1454 |
|
| 1455 |
# DEBUG messages: set environment variable DAPL_DBG_TYPE, default |
| 1456 |
mapping is 0x0003 |
| 1457 |
|
| 1458 |
DAPL_DBG_TYPE_ERR = 0x0001, |
| 1459 |
DAPL_DBG_TYPE_WARN = 0x0002, |
| 1460 |
DAPL_DBG_TYPE_EVD = 0x0004, |
| 1461 |
DAPL_DBG_TYPE_CM = 0x0008, |
| 1462 |
DAPL_DBG_TYPE_EP = 0x0010, |
| 1463 |
DAPL_DBG_TYPE_UTIL = 0x0020, |
| 1464 |
DAPL_DBG_TYPE_CALLBACK = 0x0040, |
| 1465 |
DAPL_DBG_TYPE_DTO_COMP_ERR= 0x0080, |
| 1466 |
DAPL_DBG_TYPE_API = 0x0100, |
| 1467 |
DAPL_DBG_TYPE_RTN = 0x0200, |
| 1468 |
DAPL_DBG_TYPE_EXCEPTION = 0x0400, |
| 1469 |
DAPL_DBG_TYPE_SRQ = 0x0800, |
| 1470 |
DAPL_DBG_TYPE_CNTR = 0x1000 |
| 1471 |
|
| 1472 |
------------------------- |
| 1473 |
|
| 1474 |
OFED 1.2 RELEASE NOTES |
| 1475 |
|
| 1476 |
NEW SINCE Gamma 3.2 and OFED 1.1 |
| 1477 |
|
| 1478 |
* New Features |
| 1479 |
|
| 1480 |
1. Added dtest and dapltest to the openfabrics build and utils rpm. |
| 1481 |
Includes manpages. |
| 1482 |
2. Added following enviroment variables to configure connection management |
| 1483 |
timers (default settings) for larger clusters: |
| 1484 |
|
| 1485 |
DAPL_CM_ARP_TIMEOUT_MS 4000 |
| 1486 |
DAPL_CM_ARP_RETRY_COUNT 15 |
| 1487 |
DAPL_CM_ROUTE_TIMEOUT_MS 4000 |
| 1488 |
DAPL_CM_ROUTE_RETRY_COUNT 15 |
| 1489 |
|
| 1490 |
* Bug Fixes |
| 1491 |
|
| 1492 |
+ Added support for new ib verbs client register event. No extra |
| 1493 |
processing required at the uDAPL level. |
| 1494 |
+ Fix some issues supporting create qp without recv cq handle or |
| 1495 |
recv qp resources. IB verbs assume a recv_cq handle and uDAPL |
| 1496 |
dapl_ep_create assumes there is always recv_sge resources specified. |
| 1497 |
+ Fix some timeout and long disconnect delay issues discovered during |
| 1498 |
scale-out testing. Added support to retry rdma_cm address and route |
| 1499 |
resolution with configuration options. Provide a disconnect call |
| 1500 |
when receiving the disconnect request to guarantee a disconnect reply |
| 1501 |
and event on the remote side. The rdma_disconnect was not being called |
| 1502 |
from dat_ep_disconnect() as a result of the state changing |
| 1503 |
to DISCONNECTED in the event callback. |
| 1504 |
+ Changes to support exchanging and validation of the device |
| 1505 |
responder_resources and the initiator_depth during conn establishment |
| 1506 |
+ Fix some build issues with dapltest on 32 bit arch, and on ia64 SUSE arch |
| 1507 |
+ Add support for multiple IB devices to dat.conf to support IPoIB HA failover |
| 1508 |
+ Fix atomic operation build problem with ia64 and RHEL5. |
| 1509 |
+ Add support to return local and remote port information with dat_ep_query |
| 1510 |
+ Cleanup RPM specfile for the dapl package, move to 1.2-1 release. |
| 1511 |
|
| 1512 |
NEW SINCE Gamma 3.1 and OFED 1.0 |
| 1513 |
|
| 1514 |
* BUG FIXES |
| 1515 |
|
| 1516 |
+ Update obsolete CLK_TCK to CLOCKS_PER_SEC |
| 1517 |
+ Fill out some unitialized fields in the ia_attr structure returned by |
| 1518 |
dat_ia_query(). |
| 1519 |
+ Update dtest to support multiple segments on rdma write and change |
| 1520 |
makefile to use OpenIB-cma by default. |
| 1521 |
+ Add support for dat_evd_set_unwaitable on a DTO evd in openib_cma |
| 1522 |
provider |
| 1523 |
+ Added errno reporting (message and return codes) during open to help |
| 1524 |
diagnose create thread issues. |
| 1525 |
+ Fix some suspicious inline assembly EIEIO_ON_SMP and ISYNC_ON_SMP |
| 1526 |
+ Fix IA64 build problems |
| 1527 |
+ Lower the reject debug message level so we don't see warnings when |
| 1528 |
consumers reject. |
| 1529 |
+ Added support for active side TIMED_OUT event from a provider. |
| 1530 |
+ Fix bug in dapls_ib_get_dat_event() call after adding new unreachable |
| 1531 |
event. |
| 1532 |
+ Update for new rdma_create_id() function signature. |
| 1533 |
+ Set max rdma read per EP attributes |
| 1534 |
+ Report the proper error and timeout events. |
| 1535 |
+ Socket CM fix to guard against using a loopback address as the local |
| 1536 |
device address. |
| 1537 |
+ Use the uCM set_option feature to adjust connect request timeout |
| 1538 |
retry values. |
| 1539 |
+ Fix to disallow any event after a disconnect event. |
| 1540 |
|
| 1541 |
* OFED 1.1 uDAPL source build instructions: |
| 1542 |
|
| 1543 |
cd /usr/local/ofed/src/openib-1.1/src/userspace/dapl |
| 1544 |
|
| 1545 |
# NON_DEBUG build configuration |
| 1546 |
|
| 1547 |
./configure --disable-libcheck --prefix /usr/local/ofed |
| 1548 |
--libdir /usr/local/ofed/lib64 LDFLAGS=-L/usr/local/ofed/lib64 |
| 1549 |
CPPFLAGS="-I../libibverbs/include -I../librdmacm/include" |
| 1550 |
|
| 1551 |
# build and install |
| 1552 |
|
| 1553 |
make |
| 1554 |
make install |
| 1555 |
|
| 1556 |
# DEBUG build configuration |
| 1557 |
|
| 1558 |
./configure --disable-libcheck --enable-debug --prefix /usr/local/ofed |
| 1559 |
--libdir /usr/local/ofed/lib64 LDFLAGS=-L/usr/local/ofed/lib64 |
| 1560 |
CPPFLAGS="-I../libibverbs/include -I../librdmacm/include" |
| 1561 |
|
| 1562 |
# build and install |
| 1563 |
|
| 1564 |
make |
| 1565 |
make install |
| 1566 |
|
| 1567 |
# DEBUG messages: set environment variable DAPL_DBG_TYPE, default |
| 1568 |
mapping is 0x0003 |
| 1569 |
|
| 1570 |
DAPL_DBG_TYPE_ERR = 0x0001, |
| 1571 |
DAPL_DBG_TYPE_WARN = 0x0002, |
| 1572 |
DAPL_DBG_TYPE_EVD = 0x0004, |
| 1573 |
DAPL_DBG_TYPE_CM = 0x0008, |
| 1574 |
DAPL_DBG_TYPE_EP = 0x0010, |
| 1575 |
DAPL_DBG_TYPE_UTIL = 0x0020, |
| 1576 |
DAPL_DBG_TYPE_CALLBACK = 0x0040, |
| 1577 |
DAPL_DBG_TYPE_DTO_COMP_ERR= 0x0080, |
| 1578 |
DAPL_DBG_TYPE_API = 0x0100, |
| 1579 |
DAPL_DBG_TYPE_RTN = 0x0200, |
| 1580 |
DAPL_DBG_TYPE_EXCEPTION = 0x0400, |
| 1581 |
DAPL_DBG_TYPE_SRQ = 0x0800, |
| 1582 |
DAPL_DBG_TYPE_CNTR = 0x1000 |
| 1583 |
|
| 1584 |
|
| 1585 |
Note: The udapl provider library libdaplscm.so is untested and |
| 1586 |
unsupported, thus customers should not use it. |
| 1587 |
It will be removed in the next OFED release. |
| 1588 |
|
| 1589 |
DAPL GAMMA 3.1 RELEASE NOTES |
| 1590 |
|
| 1591 |
This release of the DAPL reference implementation |
| 1592 |
is timed to coincide with the first release of the |
| 1593 |
Open Fabrics (www.openfabrics.org) software stack. |
| 1594 |
This release adds support for this new stack, which |
| 1595 |
is now the native Linux RDMA stack. |
| 1596 |
|
| 1597 |
This release also adds a new licensing option. In |
| 1598 |
addition to the Common Public License and BSD License, |
| 1599 |
the code can now be licensed under the terms of the GNU |
| 1600 |
General Public License (GPL) version 2. |
| 1601 |
|
| 1602 |
NEW SINCE Gamma 3.0 |
| 1603 |
|
| 1604 |
- GPL v2 added as a licensing option |
| 1605 |
- OpenFabrics (aka OpenIB) gen2 verbs support |
| 1606 |
- dapltest support for Solaris 10 |
| 1607 |
|
| 1608 |
* BUG FIXES |
| 1609 |
|
| 1610 |
+ Fixed a disconnect event processing race |
| 1611 |
+ Fix to destroy all QPs on IA close |
| 1612 |
+ Removed compiler warnings |
| 1613 |
+ Removed unused variables |
| 1614 |
+ And many more... |
| 1615 |
|
| 1616 |
DAPL GAMMA 3.0 RELEASE NOTES |
| 1617 |
|
| 1618 |
This is the first release based on version 1.2 of the spec. There |
| 1619 |
are some components, such a shared receive queues (SRQs), which |
| 1620 |
are not implemented yet. |
| 1621 |
|
| 1622 |
Once again there were numerous bug fixes submitted by the |
| 1623 |
DAPL community. |
| 1624 |
|
| 1625 |
NEW SINCE Beta 2.06 |
| 1626 |
|
| 1627 |
- DAT 1.2 headers |
| 1628 |
- DAT_IA_HANDLEs implemented as small integers |
| 1629 |
- Changed default device name to be "ia0a" |
| 1630 |
- Initial support for Linux 2.6.X kernels |
| 1631 |
- Updates to the OpenIB gen 1 provider |
| 1632 |
|
| 1633 |
* BUG FIXES |
| 1634 |
|
| 1635 |
+ Updated Makefile for differentiation between OS releases. |
| 1636 |
+ Updated atomic routines to use appropriate API |
| 1637 |
+ Removed unnecessary assert from atomic_dec. |
| 1638 |
+ Fixed bugs when freeing a PSP. |
| 1639 |
+ Fixed error codes returned by the DAT static registry. |
| 1640 |
+ Kernel updates for dat_strerror. |
| 1641 |
+ Cleaned up the transport layer/adapter interface to use DAPL |
| 1642 |
types rather than transport types. |
| 1643 |
+ Fixed ring buffer reallocation. |
| 1644 |
+ Removed old test/udapl/dapltest directory. |
| 1645 |
+ Fixed DAT_IA_HANDLE translation (from pointer to int and |
| 1646 |
vice versa) on 64-bit platforms. |
| 1647 |
|
| 1648 |
DAP BETA 2.06 RELEASE NOTES |
| 1649 |
|
| 1650 |
We are not planning any further releases of the Beta series, |
| 1651 |
which are based on the 1.1 version of the spec. There may be |
| 1652 |
further releases for bug fixes, but we anticipate the DAPL |
| 1653 |
community to move to the new 1.2 version of the spec and the |
| 1654 |
changes mandated in the reference implementation. |
| 1655 |
|
| 1656 |
The biggest item in this release is the first inclusion of the |
| 1657 |
OpenIB Gen 1 provider, an item generating a lot of interest in |
| 1658 |
the IB community. This implementation has graciously been |
| 1659 |
provided by the Mellanox team. The kdapl implementation is in |
| 1660 |
progress, and we imagine work will soon begin on Gen 2. |
| 1661 |
|
| 1662 |
There are also a handful of bug fixes available, as well as a long |
| 1663 |
awaited update to the endpoint design document. |
| 1664 |
|
| 1665 |
NEW SINCE Beta 2.05 |
| 1666 |
|
| 1667 |
- OpenIB gen 1 provider support has been added |
| 1668 |
- Added dapls_evd_post_generic_event(), routine to post generic |
| 1669 |
event types as requested by some providers. Also cleaned up |
| 1670 |
error reporting. |
| 1671 |
- Updated the endpoint design document in the doc/ directory. |
| 1672 |
|
| 1673 |
* BUG FIXES |
| 1674 |
|
| 1675 |
+ Cleaned up memory leak on close by freeing the HCA structure; |
| 1676 |
+ Removed bogus #defs for rdtsc calls on IA64. |
| 1677 |
+ Changed daptest thread types to use internal types for |
| 1678 |
portability & correctness |
| 1679 |
+ Various 64 bit enhancements & updates |
| 1680 |
+ Fixes to conformance test that were defining CONN_QUAL twice |
| 1681 |
and using it in different ways |
| 1682 |
+ Cleaned up private data handling in ep_connect & provider |
| 1683 |
support: we now avoid extra copy in connect code; reduced |
| 1684 |
stack requirements by using private_data structure in the EP; |
| 1685 |
removed provider variable. |
| 1686 |
+ Fixed problem in the dat conformance test where cno_wait would |
| 1687 |
attempt to dereference a timer value and SEGV. |
| 1688 |
+ Removed old vestiges of depricated POLLING_COMPLETIONS |
| 1689 |
conditionals. |
| 1690 |
|
| 1691 |
DAPL BETA 2.05 RELEASE NOTES |
| 1692 |
|
| 1693 |
This was to be a very minor release, the primary change was |
| 1694 |
going to be the new wording of the DAT license as contained in |
| 1695 |
the header for all source files. But the interest and |
| 1696 |
development occurring in DAPL provided some extra bug fixes, and |
| 1697 |
some new functionality that has been requested for a while. |
| 1698 |
|
| 1699 |
First, you may notice that every single source file was |
| 1700 |
changed. If you read the release notes from DAPL BETA 2.04, you |
| 1701 |
were warned this would happen. There was a legal issue with the |
| 1702 |
wording in the header, the end result was that every source file |
| 1703 |
was required to change the word 'either of' to 'both'. We've |
| 1704 |
been putting this change off as long as possible, but we wanted |
| 1705 |
to do it in a clean drop before we start working on DAT 1.2 |
| 1706 |
changes in the reference implementation, just to keep things |
| 1707 |
reasonably sane. |
| 1708 |
|
| 1709 |
kdapltest has enabled three of the subtests supported by |
| 1710 |
dapltest. The Performance test in particular has been very |
| 1711 |
useful to dapltest in getting minima and maxima. The Limit test |
| 1712 |
pushes the limits by allocating the maximum number of specific |
| 1713 |
resources. And the FFT tests are also available. |
| 1714 |
|
| 1715 |
Most vendors have supported shared memory regions for a while, |
| 1716 |
several of which have asked the reference implementation team to |
| 1717 |
provide a common implementation. Shared memory registration has |
| 1718 |
been tested on ibapi, and compiled into vapi. Both InfiniBand |
| 1719 |
providers have the restriction that a memory region must be |
| 1720 |
created before it can be shared; not all RDMA APIs are this way, |
| 1721 |
several allow you to declare a memory region shared when it is |
| 1722 |
registered. Hence, details of the implementation are hidden in |
| 1723 |
the provider layer, rather than forcing other APIs to do |
| 1724 |
something strange. |
| 1725 |
|
| 1726 |
This release also contains some changes that will allow dapl to |
| 1727 |
work on Opteron processors, as well as some preliminary support |
| 1728 |
for Power PC architecture. These features are not well tested |
| 1729 |
and may be incomplete at this time. |
| 1730 |
|
| 1731 |
Finally, we have been asked several times over the course of the |
| 1732 |
project for a canonical interface between the common and |
| 1733 |
provider layers. This release includes a dummy provider to meet |
| 1734 |
that need. Anyone should be able to download the release and do |
| 1735 |
a: |
| 1736 |
make VERBS=DUMMY |
| 1737 |
|
| 1738 |
And have a cleanly compiled dapl library. This will be useful |
| 1739 |
both to those porting new transport providers, as well as those |
| 1740 |
going to new machines. |
| 1741 |
|
| 1742 |
The DUMMY provider has been compiled on both Linux and Windows |
| 1743 |
machines. |
| 1744 |
|
| 1745 |
|
| 1746 |
NEW SINCE Beta 2.4 |
| 1747 |
- kdapltest enhancements: |
| 1748 |
* Limit subtests now work |
| 1749 |
* Performance subtests now work. |
| 1750 |
* FFT tests now work. |
| 1751 |
|
| 1752 |
- The VAPI headers have been refreshed by Mellanox |
| 1753 |
|
| 1754 |
- Initial Opteron and PPC support. |
| 1755 |
|
| 1756 |
- Atomic data types now have consistent treatment, allowing us to |
| 1757 |
use native data types other than integers. The Linux kdapl |
| 1758 |
uses atomic_t, allowing dapl to use the kernel macros and |
| 1759 |
eliminate the assembly code in dapl_osd.h |
| 1760 |
|
| 1761 |
- The license language was updated per the direction of the |
| 1762 |
DAT Collaborative. This two word change affected the header |
| 1763 |
of every file in the tree. |
| 1764 |
|
| 1765 |
- SHARED memory regions are now supported. |
| 1766 |
|
| 1767 |
- Initial support for the TOPSPIN provider. |
| 1768 |
|
| 1769 |
- Added a dummy provider, essentially the NULL provider. It's |
| 1770 |
purpose is to aid in porting and to clarify exactly what is |
| 1771 |
expected in a provider implementation. |
| 1772 |
|
| 1773 |
- Removed memory allocation from the DTO path for VAPI |
| 1774 |
|
| 1775 |
- cq_resize will now allow the CQ to be resized smaller. Not all |
| 1776 |
providers support this, but it's a provider problem, not a |
| 1777 |
limitation of the common code. |
| 1778 |
|
| 1779 |
* BUG FIXES |
| 1780 |
|
| 1781 |
+ Removed spurious lock in dapl_evd_connection_callb.c that |
| 1782 |
would have caused a deadlock. |
| 1783 |
+ The Async EVD was getting torn down too early, potentially |
| 1784 |
causing lost errors. Has been moved later in the teardown |
| 1785 |
process. |
| 1786 |
+ kDAPL replaced mem_map_reserve() with newer SetPageReserved() |
| 1787 |
for better Linux integration. |
| 1788 |
+ kdapltest no longer allocate large print buffers on the stack, |
| 1789 |
is more careful to ensure buffers don't overflow. |
| 1790 |
+ Put dapl_os_dbg_print() under DAPL_DBG conditional, it is |
| 1791 |
supposed to go away in a production build. |
| 1792 |
+ dapltest protocol version has been bumped to reflect the |
| 1793 |
change in the Service ID. |
| 1794 |
+ Corrected several instances of routines that did not adhere |
| 1795 |
to the DAT 1.1 error code scheme. |
| 1796 |
+ Cleaned up vapi ib_reject_connection to pass DAT types rather |
| 1797 |
than provider specific types. Also cleaned up naming interface |
| 1798 |
declarations and their use in vapi_cm.c; fixed incorrect |
| 1799 |
#ifdef for naming. |
| 1800 |
+ Initialize missing uDAPL provider attr, pz_support. |
| 1801 |
+ Changes for better layering: first, moved |
| 1802 |
dapl_lmr_convert_privileges to the provider layer as memory |
| 1803 |
permissions are clearly transport specific and are not always |
| 1804 |
defined in an integer bitfield; removed common routines for |
| 1805 |
lmr and rmr. Second, move init and release setup/teardown |
| 1806 |
routines into adapter_util.h, which defined the provider |
| 1807 |
interface. |
| 1808 |
+ Cleaned up the HCA name cruft that allowed different types |
| 1809 |
of names such as strings or ints to be dealt with in common |
| 1810 |
code; but all names are presented by the dat_registry as |
| 1811 |
strings, so pushed conversions down to the provider |
| 1812 |
level. Greatly simplifies names. |
| 1813 |
+ Changed deprecated true/false to DAT_TRUE/DAT_FALSE. |
| 1814 |
+ Removed old IB_HCA_NAME type in favor of char *. |
| 1815 |
+ Fixed race condition in kdapltest's use of dat_evd_dequeue. |
| 1816 |
+ Changed cast for SERVER_PORT_NUMBER to DAT_CONN_QUAL as it |
| 1817 |
should be. |
| 1818 |
+ Small code reorg to put the CNO into the EVD when it is |
| 1819 |
allocated, which simplifies things. |
| 1820 |
+ Removed gratuitous ib_hca_port_t and ib_send_op_type_t types, |
| 1821 |
replaced with standard int. |
| 1822 |
+ Pass a pointer to cqe debug routine, not a structure. Some |
| 1823 |
clean up of data types. |
| 1824 |
+ kdapl threads now invoke reparent_to_init() on exit to allow |
| 1825 |
threads to get cleaned up. |
| 1826 |
|
| 1827 |
|
| 1828 |
|
| 1829 |
DAPL BETA 2.04 RELEASE NOTES |
| 1830 |
|
| 1831 |
The big changes for this release involve a more strict adherence |
| 1832 |
to the original dapl architecture. Originally, only InfiniBand |
| 1833 |
providers were available, so allowing various data types and |
| 1834 |
event codes to show through into common code wasn't a big deal. |
| 1835 |
|
| 1836 |
But today, there are an increasing number of providers available |
| 1837 |
on a number of transports. Requiring an IP iWarp provider to |
| 1838 |
match up to InfiniBand events is silly, for example. |
| 1839 |
|
| 1840 |
Restructuring the code allows more flexibility in providing an |
| 1841 |
implementation. |
| 1842 |
|
| 1843 |
There are also a large number of bug fixes available in this |
| 1844 |
release, particularly in kdapl related code. |
| 1845 |
|
| 1846 |
Be warned that the next release will change every file in the |
| 1847 |
tree as we move to the newly approved DAT license. This is a |
| 1848 |
small change, but all files are affected. |
| 1849 |
|
| 1850 |
Future releases will also support to the soon to be ratified DAT |
| 1851 |
1.2 specification. |
| 1852 |
|
| 1853 |
This release has benefited from many bug reports and fixes from |
| 1854 |
a number of individuals and companies. On behalf of the DAPL |
| 1855 |
community, thank you! |
| 1856 |
|
| 1857 |
|
| 1858 |
NEW SINCE Beta 2.3 |
| 1859 |
|
| 1860 |
- Made several changes to be more rigorous on the layering |
| 1861 |
design of dapl. The intent is to make it easier for non |
| 1862 |
InfiniBand transports to use dapl. These changes include: |
| 1863 |
|
| 1864 |
* Revamped the ib_hca_open/close code to use an hca_ptr |
| 1865 |
rather than an ib_handle, giving the transport layer more |
| 1866 |
flexibility in assigning transport handles and resources. |
| 1867 |
|
| 1868 |
* Removed the CQD calls, they are specific to the IBM API; |
| 1869 |
folded this functionality into the provider open/close calls. |
| 1870 |
|
| 1871 |
* Moved VAPI, IBAPI transport specific items into a transport |
| 1872 |
structure placed inside of the HCA structure. Also updated |
| 1873 |
routines using these fields to use the new location. Cleaned |
| 1874 |
up provider knobs that have been exposed for too long. |
| 1875 |
|
| 1876 |
* Changed a number of provider routines to use DAPL structure |
| 1877 |
pointers rather than exposing provider handles & values. Moved |
| 1878 |
provider specific items out of common code, including provider |
| 1879 |
data types (e.g. ib_uint32_t). |
| 1880 |
|
| 1881 |
* Pushed provider completion codes and type back into the |
| 1882 |
provider layer. We no longer use EVD or CM completion types at |
| 1883 |
the common layer, instead we obtain the appropriate DAT type |
| 1884 |
from the provider and process only DAT types. |
| 1885 |
|
| 1886 |
* Change private_data handling such that we can now accommodate |
| 1887 |
variable length private data. |
| 1888 |
|
| 1889 |
- Remove DAT 1.0 cruft from the DAT header files. |
| 1890 |
|
| 1891 |
- Better spec compliance in headers and various routines. |
| 1892 |
|
| 1893 |
- Major updates to the VAPI implementation from |
| 1894 |
Mellanox. Includes initial kdapl implementation |
| 1895 |
|
| 1896 |
- Move kdapl platform specific support for hash routines into |
| 1897 |
OSD file. |
| 1898 |
|
| 1899 |
- Cleanups to make the code more readable, including comments |
| 1900 |
and certain variable and structure names. |
| 1901 |
|
| 1902 |
- Fixed CM_BUSTED code so that it works again: very useful for |
| 1903 |
new dapl ports where infrastructure is lacking. Also made |
| 1904 |
some fixes for IBHOSTS_NAMING conditional code. |
| 1905 |
|
| 1906 |
- Added DAPL_MERGE_CM_DTO as a compile time switch to support |
| 1907 |
EVD stream merging of CM and DTO events. Default is off. |
| 1908 |
|
| 1909 |
- 'Quit' test ported to kdapltest |
| 1910 |
|
| 1911 |
- uDAPL now builds on Linux 2.6 platform (SuSE 9.1). |
| 1912 |
|
| 1913 |
- kDAPL now builds for a larger range of Linux kernels, but |
| 1914 |
still lacks 2.6 support. |
| 1915 |
|
| 1916 |
- Added shared memory ID to LMR structure. Shared memory is |
| 1917 |
still not fully supported in the reference implementation, but |
| 1918 |
the common code will appear soon. |
| 1919 |
|
| 1920 |
* Bug fixes |
| 1921 |
- Various Makefiles fixed to use the correct dat registry |
| 1922 |
library in its new location (as of Beta 2.03) |
| 1923 |
- Simple reorg of dat headers files to be consistent with |
| 1924 |
the spec. |
| 1925 |
- fixed bug in vapi_dto.h recv macro where we could have an |
| 1926 |
uninitialized pointer. |
| 1927 |
- Simple fix in dat_dr.c to initialize a variable early in the |
| 1928 |
routine before errors occur. |
| 1929 |
- Removed private data pointers from a CONNECTED event, as |
| 1930 |
there should be no private data here. |
| 1931 |
- dat_strerror no longer returns an uninitialized pointer if |
| 1932 |
the error code is not recognized. |
| 1933 |
- dat_dup_connect() will reject 0 timeout values, per the |
| 1934 |
spec. |
| 1935 |
- Removed unused internal_hca_names parameter from |
| 1936 |
ib_enum_hcas() interface. |
| 1937 |
- Use a temporary DAT_EVENT for kdapl up-calls rather than |
| 1938 |
making assumptions about the current event queue. |
| 1939 |
- Relocated some platform dependent code to an OSD file. |
| 1940 |
- Eliminated several #ifdefs in .c files. |
| 1941 |
- Inserted a missing unlock() on an error path. |
| 1942 |
- Added bounds checking on size of private data to make sure |
| 1943 |
we don't overrun the buffer |
| 1944 |
- Fixed a kdapltest problem that caused a machine to panic if |
| 1945 |
the user hit ^C |
| 1946 |
- kdapltest now uses spin locks more appropriate for their |
| 1947 |
context, e.g. spin_lock_bh or spin_lock_irq. Under a |
| 1948 |
conditional. |
| 1949 |
- Fixed kdapltest loops that drain EVDs so they don't go into |
| 1950 |
endless loops. |
| 1951 |
- Fixed bug in dapl_llist_add_entry link list code. |
| 1952 |
- Better error reporting from provider code. |
| 1953 |
- Handle case of user trying to reap DTO completions on an |
| 1954 |
EP that has been freed. |
| 1955 |
- No longer hold lock when ep_free() calls into provider layer |
| 1956 |
- Fixed cr_accept() to not have an extra copy of |
| 1957 |
private_data. |
| 1958 |
- Verify private_data pointers before using them, avoid |
| 1959 |
panic. |
| 1960 |
- Fixed memory leak in kdapltest where print buffers were not |
| 1961 |
getting reclaimed. |
| 1962 |
|
| 1963 |
|
| 1964 |
|
| 1965 |
DAPL BETA 2.03 RELEASE NOTES |
| 1966 |
|
| 1967 |
There are some prominent features in this release: |
| 1968 |
1) dapltest/kdapltest. The dapltest test program has been |
| 1969 |
rearchitected such that a kernel version is now available |
| 1970 |
to test with kdapl. The most obvious change is a new |
| 1971 |
directory structure that more closely matches other core |
| 1972 |
dapl software. But there are a large number of changes |
| 1973 |
throughout the source files to accommodate both the |
| 1974 |
differences in udapl/kdapl interfaces, but also more mundane |
| 1975 |
things such as printing. |
| 1976 |
|
| 1977 |
The new dapltest is in the tree at ./test/dapltest, while the |
| 1978 |
old remains at ./test/udapl/dapltest. For this release, we |
| 1979 |
have maintained both versions. In a future release, perhaps |
| 1980 |
the next release, the old dapltest directory will be |
| 1981 |
removed. Ongoing development will only occur in the new tree. |
| 1982 |
|
| 1983 |
2) DAT 1.1 compliance. The DAT Collaborative has been busy |
| 1984 |
finalizing the 1.1 revision of the spec. The header files |
| 1985 |
have been reviewed and posted on the DAT Collaborative web |
| 1986 |
site, they are now in full compliance. |
| 1987 |
|
| 1988 |
The reference implementation has been at a 1.1 level for a |
| 1989 |
while. The current implementation has some features that will |
| 1990 |
be part of the 1.2 DAT specification, but only in places |
| 1991 |
where full compatibility can be maintained. |
| 1992 |
|
| 1993 |
3) The DAT Registry has undergone some positive changes for |
| 1994 |
robustness and support of more platforms. It now has the |
| 1995 |
ability to support several identical provider names |
| 1996 |
simultaneously, which enables the same dat.conf file to |
| 1997 |
support multiple platforms. The registry will open each |
| 1998 |
library and return when successful. For example, a dat.conf |
| 1999 |
file may contain multiple provider names for ex0a, each |
| 2000 |
pointing to a different library that may represent different |
| 2001 |
platforms or vendors. This simplifies distribution into |
| 2002 |
different environments by enabling the use of common |
| 2003 |
dat.conf files. |
| 2004 |
|
| 2005 |
In addition, there are a large number of bug fixes throughout |
| 2006 |
the code. Bug reports and fixes have come from a number of |
| 2007 |
companies. |
| 2008 |
|
| 2009 |
Also note that the Release notes are cleaned up, no longer |
| 2010 |
containing the complete text of previous releases. |
| 2011 |
|
| 2012 |
* EVDs no longer support DTO and CONNECTION event types on the |
| 2013 |
same EVD. NOTE: The problem is maintaining the event ordering |
| 2014 |
between two channels such that no DTO completes before a |
| 2015 |
connection is received; and no DTO completes after a |
| 2016 |
disconnect is received. For 90% of the cases this can be made |
| 2017 |
to work, but the remaining 10% will cause serious performance |
| 2018 |
degradation to get right. |
| 2019 |
|
| 2020 |
NEW SINCE Beta 2.2 |
| 2021 |
|
| 2022 |
* DAT 1.1 spec compliance. This includes some new types, error |
| 2023 |
codes, and moving structures around in the header files, |
| 2024 |
among other things. Note the Class bits of dat_error.h have |
| 2025 |
returned to a #define (from an enum) to cover the broadest |
| 2026 |
range of platforms. |
| 2027 |
|
| 2028 |
* Several additions for robustness, including handle and |
| 2029 |
pointer checking, better argument checking, state |
| 2030 |
verification, etc. Better recovery from error conditions, |
| 2031 |
and some assert()s have been replaced with 'if' statements to |
| 2032 |
handle the error. |
| 2033 |
|
| 2034 |
* EVDs now maintain the actual queue length, rather than the |
| 2035 |
requested amount. Both the DAT spec and IB (and other |
| 2036 |
transports) allow the underlying implementation to provide |
| 2037 |
more CQ entries than requested. |
| 2038 |
|
| 2039 |
Requests for the same number of entries contained by an EVD |
| 2040 |
return immediate success. |
| 2041 |
|
| 2042 |
* kDAPL enhancements: |
| 2043 |
- module parameters & OS support calls updated to work with |
| 2044 |
more recent Linux kernels. |
| 2045 |
- kDAPL build options changes to match the Linux kernel, vastly |
| 2046 |
reducing the size and making it more robust. |
| 2047 |
- kDAPL unload now works properly |
| 2048 |
- kDAPL takes a reference on the provider driver when it |
| 2049 |
obtains a verbs vector, to prevent an accidental unload |
| 2050 |
- Cleaned out all of the uDAPL cruft from the linux/osd files. |
| 2051 |
|
| 2052 |
* New dapltest (see above). |
| 2053 |
|
| 2054 |
* Added a new I/O trace facility, enabling a developer to debug |
| 2055 |
all I/O that are in progress or recently completed. Default |
| 2056 |
is OFF in the build. |
| 2057 |
|
| 2058 |
* 0 timeout connections now refused, per the spec. |
| 2059 |
|
| 2060 |
* Moved the remaining uDAPL specific files from the common/ |
| 2061 |
directory to udapl/. Also removed udapl files from the kdapl |
| 2062 |
build. |
| 2063 |
|
| 2064 |
* Bug fixes |
| 2065 |
- Better error reporting from provider layer |
| 2066 |
- Fixed race condition on reference counts for posting DTO |
| 2067 |
ops. |
| 2068 |
- Use DAT_COMPLETION_SUPPRESS_FLAG to suppress successful |
| 2069 |
completion of dapl_rmr_bind (instead of |
| 2070 |
DAT_COMPLEITON_UNSIGNALLED, which is for non-notification |
| 2071 |
completion). |
| 2072 |
- Verify psp_flags value per the spec |
| 2073 |
- Bug in psp_create_any() checking psp_flags fixed |
| 2074 |
- Fixed type of flags in ib_disconnect from |
| 2075 |
DAT_COMPLETION_FLAGS to DAT_CLOSE_FLAGS |
| 2076 |
- Removed hard coded check for ASYNC_EVD. Placed all EVD |
| 2077 |
prevention in evd_stream_merging_supported array, and |
| 2078 |
prevent ASYNC_EVD from being created by an app. |
| 2079 |
- ep_free() fixed to comply with the spec |
| 2080 |
- Replaced various printfs with dbg_log statements |
| 2081 |
- Fixed kDAPL interaction with the Linux kernel |
| 2082 |
- Corrected phy_register protottype |
| 2083 |
- Corrected kDAPL wait/wakeup synchronization |
| 2084 |
- Fixed kDAPL evd_kcreate() such that it no longer depends |
| 2085 |
on uDAPL only code. |
| 2086 |
- dapl_provider.h had wrong guard #def: changed DAT_PROVIDER_H |
| 2087 |
to DAPL_PROVIDER_H |
| 2088 |
- removed extra (and bogus) call to dapls_ib_completion_notify() |
| 2089 |
in evd_kcreate.c |
| 2090 |
- Inserted missing error code assignment in |
| 2091 |
dapls_rbuf_realloc() |
| 2092 |
- When a CONNECTED event arrives, make sure we are ready for |
| 2093 |
it, else something bad may have happened to the EP and we |
| 2094 |
just return; this replaces an explicit check for a single |
| 2095 |
error condition, replacing it with the general check for the |
| 2096 |
state capable of dealing with the request. |
| 2097 |
- Better context pointer verification. Removed locks around |
| 2098 |
call to ib_disconnect on an error path, which would result |
| 2099 |
in a deadlock. Added code for BROKEN events. |
| 2100 |
- Brought the vapi code more up to date: added conditional |
| 2101 |
compile switches, removed obsolete __ActivePort, deal |
| 2102 |
with 0 length DTO |
| 2103 |
- Several dapltest fixes to bring the code up to the 1.1 |
| 2104 |
specification. |
| 2105 |
- Fixed mismatched dalp_os_dbg_print() #else dapl_Dbg_Print(); |
| 2106 |
the latter was replaced with the former. |
| 2107 |
- ep_state_subtype() now includes UNCONNECTED. |
| 2108 |
- Added some missing ibapi error codes. |
| 2109 |
|
| 2110 |
|
| 2111 |
|
| 2112 |
NEW SINCE Beta 2.1 |
| 2113 |
|
| 2114 |
* Changes for Erratta and 1.1 Spec |
| 2115 |
- Removed DAT_NAME_NOT_FOUND, per DAT erratta |
| 2116 |
- EVD's with DTO and CONNECTION flags set no longer valid. |
| 2117 |
- Removed DAT_IS_SUCCESS macro |
| 2118 |
- Moved provider attribute structures from vendor files to udat.h |
| 2119 |
and kdat.h |
| 2120 |
- kdapl UPCALL_OBJECT now passed by reference |
| 2121 |
|
| 2122 |
* Completed dat_strerr return strings |
| 2123 |
|
| 2124 |
* Now support interrupted system calls |
| 2125 |
|
| 2126 |
* dapltest now used dat_strerror for error reporting. |
| 2127 |
|
| 2128 |
* Large number of files were formatted to meet project standard, |
| 2129 |
very cosmetic changes but improves readability and |
| 2130 |
maintainability. Also cleaned up a number of comments during |
| 2131 |
this effort. |
| 2132 |
|
| 2133 |
* dat_registry and RPM file changes (contributed by Steffen Persvold): |
| 2134 |
- Renamed the RPM name of the registry to be dat-registry |
| 2135 |
(renamed the .spec file too, some cvs add/remove needed) |
| 2136 |
- Added the ability to create RPMs as normal user (using |
| 2137 |
temporal paths), works on SuSE, Fedora, and RedHat. |
| 2138 |
- 'make rpm' now works even if you didn't build first. |
| 2139 |
- Changed to using the GNU __attribute__((constructor)) and |
| 2140 |
__attribute__((destructor)) on the dat_init functions, dat_init |
| 2141 |
and dat_fini. The old -init and -fini options to LD makes |
| 2142 |
applications crash on some platforms (Fedora for example). |
| 2143 |
- Added support for 64 bit platforms. |
| 2144 |
- Added code to allow multiple provider names in the registry, |
| 2145 |
primarily to support ia32 and ia64 libraries simultaneously. |
| 2146 |
Provider names are now kept in a list, the first successful |
| 2147 |
library open will be the provider. |
| 2148 |
|
| 2149 |
* Added initial infrastructure for DAPL_DCNTR, a feature that |
| 2150 |
will aid in debug and tuning of a dapl implementation. Partial |
| 2151 |
implementation only at this point. |
| 2152 |
|
| 2153 |
* Bug fixes |
| 2154 |
- Prevent debug messages from crashing dapl in EVD completions by |
| 2155 |
verifying the error code to ensure data is valid. |
| 2156 |
- Verify CNO before using it to clean up in evd_free() |
| 2157 |
- CNO timeouts now return correct error codes, per the spec. |
| 2158 |
- cr_accept now complies with the spec concerning connection |
| 2159 |
requests that go away before the accept is invoked. |
| 2160 |
- Verify valid EVD before posting connection evens on active side |
| 2161 |
of a connection. EP locking also corrected. |
| 2162 |
- Clean up of dapltest Makefile, no longer need to declare |
| 2163 |
DAT_THREADSAFE |
| 2164 |
- Fixed check of EP states to see if we need to disconnect an |
| 2165 |
IA is closed. |
| 2166 |
- ep_free() code reworked such that we can properly close a |
| 2167 |
connection pending EP. |
| 2168 |
- Changed disconnect processing to comply with the spec: user will |
| 2169 |
see a BROKEN event, not DISCONNECTED. |
| 2170 |
- If we get a DTO error, issue a disconnect to let the CM and |
| 2171 |
the user know the EP state changed to disconnect; checked IBA |
| 2172 |
spec to make sure we disconnect on correct error codes. |
| 2173 |
- ep_disconnect now properly deals with abrupt disconnects on the |
| 2174 |
active side of a connection. |
| 2175 |
- PSP now created in the correct state for psp_create_any(), making |
| 2176 |
it usable. |
| 2177 |
- dapl_evd_resize() now returns correct status, instead of always |
| 2178 |
DAT_NOT_IMPLEMENTED. |
| 2179 |
- dapl_evd_modify_cno() does better error checking before invoking |
| 2180 |
the provider layer, avoiding bugs. |
| 2181 |
- Simple change to allow dapl_evd_modify_cno() to set the CNO to |
| 2182 |
NULL, per the spec. |
| 2183 |
- Added required locking around call to dapl_sp_remove_cr. |
| 2184 |
|
| 2185 |
- Fixed problems related to dapl_ep_free: the new |
| 2186 |
disconnect(abrupt) allows us to do a more immediate teardown of |
| 2187 |
connections, removing the need for the MAGIC_EP_EXIT magic |
| 2188 |
number/state, which has been removed. Mmuch cleanup of paths, |
| 2189 |
and made more robust. |
| 2190 |
- Made changes to meet the spec, uDAPL 1.1 6.3.2.3: CNO is |
| 2191 |
triggered if there are waiters when the last EVD is removed |
| 2192 |
or when the IA is freed. |
| 2193 |
- Added code to deal with the provider synchronously telling us |
| 2194 |
a connection is unreachable, and generate the appropriate |
| 2195 |
event. |
| 2196 |
- Changed timer routine type from unsigned long to uintptr_t |
| 2197 |
to better fit with machine architectures. |
| 2198 |
- ep.param data now initialized in ep_create, not ep_alloc. |
| 2199 |
- Or Gerlitz provided updates to Mellanox files for evd_resize, |
| 2200 |
fw attributes, many others. Also implemented changes for correct |
| 2201 |
sizes on REP side of a connection request. |
| 2202 |
|
| 2203 |
|
| 2204 |
|
| 2205 |
NEW SINCE Beta 2.0 |
| 2206 |
|
| 2207 |
* dat_echo now DAT 1.1 compliant. Various small enhancements. |
| 2208 |
|
| 2209 |
* Revamped atomic_inc/dec to be void, the return value was never |
| 2210 |
used. This allows kdapl to use Linux kernel equivalents, and |
| 2211 |
is a small performance advantage. |
| 2212 |
|
| 2213 |
* kDAPL: dapl_evd_modify_upcall implemented and tested. |
| 2214 |
|
| 2215 |
* kDAPL: physical memory registration implemented and tested. |
| 2216 |
|
| 2217 |
* uDAPL now builds cleanly for non-debug versions. |
| 2218 |
|
| 2219 |
* Default RDMA credits increased to 8. |
| 2220 |
|
| 2221 |
* Default ACK_TIMEOUT now a reasonable value (2 sec vs old 2 |
| 2222 |
months). |
| 2223 |
|
| 2224 |
* Cleaned up dat_error.h, now 1.1 compliant in comments. |
| 2225 |
|
| 2226 |
* evd_resize initial implementation. Untested. |
| 2227 |
|
| 2228 |
* Bug fixes |
| 2229 |
- __KDAPL__ is defined in kdat_config.h, so apps don't need |
| 2230 |
to define it. |
| 2231 |
- Changed include file ordering in kdat.h to put kdat_config.h |
| 2232 |
first. |
| 2233 |
- resolved connection/tear-down race on the client side. |
| 2234 |
- kDAPL timeouts now scaled properly; fixed 3 orders of |
| 2235 |
magnitude difference. |
| 2236 |
- kDAPL EVD callbacks now get invoked for all completions; old |
| 2237 |
code would drop them in heavy utilization. |
| 2238 |
- Fixed error path in kDAPL evd creation, so we no longer |
| 2239 |
leak CNOs. |
| 2240 |
- create_psp_any returns correct error code if it can't create |
| 2241 |
a connection qualifier. |
| 2242 |
- lock fix in ibapi disconnect code. |
| 2243 |
- kDAPL INFINITE waits now work properly (non connection |
| 2244 |
waits) |
| 2245 |
- kDAPL driver unload now works properly |
| 2246 |
- dapl_lmr_[k]create now returns 1.1 error codes |
| 2247 |
- ibapi routines now return DAT 1.1 error codes |
| 2248 |
|
| 2249 |
|
| 2250 |
|
| 2251 |
NEW SINCE Beta 1.10 |
| 2252 |
|
| 2253 |
* kDAPL is now part of the DAPL distribution. See the release |
| 2254 |
notes above. |
| 2255 |
|
| 2256 |
The kDAPL 1.1 spec is now contained in the doc/ subdirectory. |
| 2257 |
|
| 2258 |
* Several files have been moved around as part of the kDAPL |
| 2259 |
checkin. Some files that were previously in udapl/ are now |
| 2260 |
in common/, some in common are now in udapl/. The goal was |
| 2261 |
to make sure files are properly located and make sense for |
| 2262 |
the build. |
| 2263 |
|
| 2264 |
* Source code formatting changes for consistency. |
| 2265 |
|
| 2266 |
* Bug fixes |
| 2267 |
- dapl_evd_create() was comparing the wrong bit combinations, |
| 2268 |
allowing bogus EVDs to be created. |
| 2269 |
- Removed code that swallowed zero length I/O requests, which |
| 2270 |
are allowed by the spec and are useful to applications. |
| 2271 |
- Locking in dapli_get_sp_ep was asymmetric; fixed it so the |
| 2272 |
routine will take and release the lock. Cosmetic change. |
| 2273 |
- dapl_get_consuemr_context() will now verify the pointer |
| 2274 |
argument 'context' is not NULL. |
| 2275 |
|
| 2276 |
|
| 2277 |
OBTAIN THE CODE |
| 2278 |
|
| 2279 |
To obtain the tree for your local machine you can check it |
| 2280 |
out of the source repository using CVS tools. CVS is common |
| 2281 |
on Unix systems and available as freeware on Windows machines. |
| 2282 |
The command to anonymously obtain the source code from |
| 2283 |
Source Forge (with no password) is: |
| 2284 |
|
| 2285 |
cvs -d:pserver:[email protected]:/cvsroot/dapl login |
| 2286 |
cvs -z3 -d:pserver:[email protected]:/cvsroot/dapl co . |
| 2287 |
|
| 2288 |
When prompted for a password, simply press the Enter key. |
| 2289 |
|
| 2290 |
Source Forge also contains explicit directions on how to become |
| 2291 |
a developer, as well as how to use different CVS commands. You may |
| 2292 |
also browse the source code using the URL: |
| 2293 |
|
| 2294 |
http://svn.sourceforge.net/viewvc/dapl/trunk/ |
| 2295 |
|
| 2296 |
SYSTEM REQUIREMENTS |
| 2297 |
|
| 2298 |
This project has been implemented on Red Hat Linux 7.3, SuSE |
| 2299 |
SLES 8, 9, and 10, Windows 2000, RHEL 3.0, 4.0 and 5.0 and a few |
| 2300 |
other Linux distrubutions. The structure of the code is designed |
| 2301 |
to allow other operating systems to easily be adapted. |
| 2302 |
|
| 2303 |
The DAPL team has used Mellanox Tavor based InfiniBand HCAs for |
| 2304 |
development, and continues with this platform. Our HCAs use the |
| 2305 |
IB verbs API submitted by IBM. Mellanox has contributed an |
| 2306 |
adapter layer using their VAPI verbs API. Either platform is |
| 2307 |
available to any group considering DAPL work. The structure of |
| 2308 |
the uDAPL source allows other provider API sets to be easily |
| 2309 |
integrated. |
| 2310 |
|
| 2311 |
The development team uses any one of three topologies: two HCAs |
| 2312 |
in a single machine; a single HCA in each of two machines; and |
| 2313 |
most commonly, a switch. Machines connected to a switch may have |
| 2314 |
more than one HCA. |
| 2315 |
|
| 2316 |
The DAPL Plugfest revealed that switches and HCAs available from |
| 2317 |
most vendors will interoperate with little trouble, given the |
| 2318 |
most recent releases of software. The dapl reference team makes |
| 2319 |
no recommendation on HCA or switch vendors. |
| 2320 |
|
| 2321 |
Explicit machine configurations are available upon request. |
| 2322 |
|
| 2323 |
IN THE TREE |
| 2324 |
|
| 2325 |
The DAPL tree contains source code for the uDAPL and kDAPL |
| 2326 |
implementations, and also includes tests and documentation. |
| 2327 |
|
| 2328 |
Included documentation has the base level API of the |
| 2329 |
providers: OpenFabrics, IBM Access, and Mellanox Verbs API. Also |
| 2330 |
included are a growing number of DAPL design documents which |
| 2331 |
lead the reader through specific DAPL subsystems. More |
| 2332 |
design documents are in progress and will appear in the tree in |
| 2333 |
the near future. |
| 2334 |
|
| 2335 |
A small number of test applications and a unit test framework |
| 2336 |
are also included. dapltest is the primary testing application |
| 2337 |
used by the DAPL team, it is capable of simulating a variety of |
| 2338 |
loads and exercises a large number of interfaces. Full |
| 2339 |
documentation is included for each of the tests. |
| 2340 |
|
| 2341 |
Recently, the dapl conformance test has been added to the source |
| 2342 |
repository. The test provides coverage of the most common |
| 2343 |
interfaces, doing both positive and negative testing. Vendors |
| 2344 |
providing DAPL implementation are strongly encouraged to run |
| 2345 |
this set of tests. |
| 2346 |
|
| 2347 |
MAKEFILE NOTES |
| 2348 |
|
| 2349 |
There are a number #ifdef's in the code that were necessary |
| 2350 |
during early development. They are disappearing as we |
| 2351 |
have time to take advantage of features and work available from |
| 2352 |
newer releases of provider software. These #ifdefs are not |
| 2353 |
documented as the intent is to remove them as soon as possible. |
| 2354 |
|
| 2355 |
CONTRIBUTIONS |
| 2356 |
|
| 2357 |
As is common to Source Forge projects, there are a small number |
| 2358 |
of developers directly associated with the source tree and having |
| 2359 |
privileges to change the tree. Requested updates, changes, bug |
| 2360 |
fixes, enhancements, or contributions should be sent to |
| 2361 |
James Lentini at [email protected] for review. We welcome your |
| 2362 |
contributions and expect the quality of the project will |
| 2363 |
improve thanks to your help. |
| 2364 |
|
| 2365 |
The core DAPL team is: |
| 2366 |
|
| 2367 |
James Lentini |
| 2368 |
Arlin Davis |
| 2369 |
Steve Sears |
| 2370 |
|
| 2371 |
... with contributions from a number of excellent engineers in |
| 2372 |
various companies contributing to the open source effort. |
| 2373 |
|
| 2374 |
|
| 2375 |
ONGOING WORK |
| 2376 |
|
| 2377 |
Not all of the DAPL spec is implemented at this time. |
| 2378 |
Functionality such as shared memory will probably not be |
| 2379 |
implemented by the reference implementation (there is a write up |
| 2380 |
on this in the doc/ area), and there are yet various cases where |
| 2381 |
work remains to be done. And of course, not all of the |
| 2382 |
implemented functionality has been tested yet. The DAPL team |
| 2383 |
continues to develop and test the tree with the intent of |
| 2384 |
completing the specification and delivering a robust and useful |
| 2385 |
implementation. |
| 2386 |
|
| 2387 |
|
| 2388 |
The DAPL Team |
| 2389 |
|