| 1 |
|
| 2 |
This file is provided under a dual BSD/GPLv2 license. When using or |
| 3 |
redistributing this file, you may do so under either license. |
| 4 |
|
| 5 |
GPL LICENSE SUMMARY |
| 6 |
|
| 7 |
Copyright(c) 2017 Intel Corporation. |
| 8 |
|
| 9 |
This program is free software; you can redistribute it and/or modify |
| 10 |
it under the terms of version 2 of the GNU General Public License as |
| 11 |
published by the Free Software Foundation. |
| 12 |
|
| 13 |
This program is distributed in the hope that it will be useful, but |
| 14 |
WITHOUT ANY WARRANTY; without even the implied warranty of |
| 15 |
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU |
| 16 |
General Public License for more details. |
| 17 |
|
| 18 |
Contact Information: |
| 19 |
Intel Corporation, www.intel.com |
| 20 |
|
| 21 |
BSD LICENSE |
| 22 |
|
| 23 |
Copyright(c) 2017 Intel Corporation. |
| 24 |
|
| 25 |
Redistribution and use in source and binary forms, with or without |
| 26 |
modification, are permitted provided that the following conditions |
| 27 |
are met: |
| 28 |
|
| 29 |
* Redistributions of source code must retain the above copyright |
| 30 |
notice, this list of conditions and the following disclaimer. |
| 31 |
* Redistributions in binary form must reproduce the above copyright |
| 32 |
notice, this list of conditions and the following disclaimer in |
| 33 |
the documentation and/or other materials provided with the |
| 34 |
distribution. |
| 35 |
* Neither the name of Intel Corporation nor the names of its |
| 36 |
contributors may be used to endorse or promote products derived |
| 37 |
from this software without specific prior written permission. |
| 38 |
|
| 39 |
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| 40 |
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| 41 |
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| 42 |
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT |
| 43 |
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, |
| 44 |
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT |
| 45 |
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, |
| 46 |
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY |
| 47 |
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT |
| 48 |
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE |
| 49 |
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| 50 |
|
| 51 |
Copyright (c) 2003-2017 Intel Corporation. All rights reserved. |
| 52 |
|
| 53 |
================================================================================ |
| 54 |
|
| 55 |
ABSTRACT |
| 56 |
-------- |
| 57 |
|
| 58 |
Discusses how to build, install and test the PSM2 library source code. |
| 59 |
|
| 60 |
Contains the following sections: |
| 61 |
|
| 62 |
- INTRODUCTION |
| 63 |
- DEPENDENCIES |
| 64 |
- BUILDING |
| 65 |
* BUILDING USING MAKEFILE |
| 66 |
* BUILDING USING RPMBUILD (CREATING SOURCE AND BINARY RPM'S) |
| 67 |
- INSTALLING |
| 68 |
* INSTALLING USING MAKEFILE |
| 69 |
* INSTALLING USING EITHER YUM OR DNF |
| 70 |
- TESTING |
| 71 |
- RELATED SOFTWARE TO PSM2 |
| 72 |
- SUPPORTING DOCUMENTATION |
| 73 |
|
| 74 |
INTRODUCTION |
| 75 |
============ |
| 76 |
|
| 77 |
This README file discusses how to build, install and test the PSM2 library |
| 78 |
source code. |
| 79 |
|
| 80 |
The PSM2 library supports a number of fabric media and stacks, and all of |
| 81 |
them run on version 7.X of Red Hat Enterprise Linux (abbreviated: RHEL), and |
| 82 |
SuSE SLES. |
| 83 |
|
| 84 |
Only the x86_64 architecture is supported. |
| 85 |
|
| 86 |
Building PSM2 is possible on RHEL 7.2+ as it ships with hfi1 kernel driver. |
| 87 |
For older RHEL 7.x versions and SuSE SLES, OPA is not natively supported |
| 88 |
in the kernel and therefore, building PSM2 is not possible unless |
| 89 |
you have the correct kernel-devel package or use latest versions of IFS. |
| 90 |
|
| 91 |
There are two mechanisms for building and installing the PSM2 library: |
| 92 |
|
| 93 |
1. Use provided Makefiles to build and install or |
| 94 |
2. Generate the *.rpm files which you can then install using either |
| 95 |
yum or dnf command |
| 96 |
|
| 97 |
DEPENDENCIES |
| 98 |
============ |
| 99 |
|
| 100 |
The following packages are required to build the PSM2 library source code: |
| 101 |
(all packages are for the x86_64 architecture) |
| 102 |
|
| 103 |
compat-rdma-devel |
| 104 |
gcc-4.8.2 |
| 105 |
glibc-devel |
| 106 |
glibc-headers |
| 107 |
kernel-headers |
| 108 |
|
| 109 |
Additional packages for GPU Direct support include: |
| 110 |
NVIDIA CUDA toolkit 8.0 or greater. Older versions are not supported. |
| 111 |
|
| 112 |
In addition to depending on these packages, root privileges are required to |
| 113 |
install the runtime libraries and development header files into standard |
| 114 |
system location. |
| 115 |
|
| 116 |
BUILDING |
| 117 |
======== |
| 118 |
|
| 119 |
The instructions below use $BASENAME, $PRODUCT and $RELEASE to refer to |
| 120 |
the base name of the tarball, RPM that will be generated and the product |
| 121 |
and release identifiers of the RPM. |
| 122 |
|
| 123 |
The base name of the RPM changes depending on which version/branch |
| 124 |
of code you derive the tar file from. |
| 125 |
|
| 126 |
Up until v10.2 of PSM2, the base name for the RPM is hfi1-psm. |
| 127 |
From v10.2 onwards, the base name will be libpsm2. The internal |
| 128 |
library remains unchanged and is still libpsm2.so.2. |
| 129 |
|
| 130 |
BUILDING USING MAKEFILES |
| 131 |
------------------------ |
| 132 |
|
| 133 |
1. Untar the tarball: |
| 134 |
$ tar zxvf $BASENAME-$PRODUCT-$RELEASE.tar.gz |
| 135 |
2. Change directory into the untarred location: |
| 136 |
$ cd $BASENAME-$PRODUCT-$RELEASE |
| 137 |
3. Build: |
| 138 |
3.1. To build with GNU C (gcc), run make on the command line: |
| 139 |
$ make |
| 140 |
- or - |
| 141 |
$ make CCARCH=gcc |
| 142 |
3.2 To build with Intel C (icc), specify the correct CCARCH: |
| 143 |
$ make CCARCH=icc |
| 144 |
3.3. To build with CUDA support, specify PSM_CUDA=1 |
| 145 |
on the command line along with the desired compiler: |
| 146 |
$ make PSM_CUDA=1 CCARCH=gcc |
| 147 |
- or - |
| 148 |
$ make PSM_CUDA=1 CCARCH=icc |
| 149 |
|
| 150 |
BUILDING USING RPMBUILD |
| 151 |
----------------------- |
| 152 |
|
| 153 |
1. Run this command from your $PWD to generate rpm, srpm files |
| 154 |
$ ./makesrpm.sh a |
| 155 |
|
| 156 |
This command results in the following collection of rpm's and source |
| 157 |
code rpm's under your $PWD/temp.X/ directory. |
| 158 |
("X" is the pid of the bash script that created the srpm and rpm files) |
| 159 |
(Result shown here for RHEL systems.) |
| 160 |
|
| 161 |
RPMS/x86_64/libpsm2-compat-10.3.7-1x86_64.rpm |
| 162 |
RPMS/x86_64/libpsm2-devel-10.3.7-1x86_64.rpm |
| 163 |
RPMS/x86_64/libpsm2-10.3.7-1x86_64.rpm |
| 164 |
RPMS/x86_64/libpsm2-debuginfo-10.3.7-1x86_64.rpm |
| 165 |
SRPMS/libpsm2-10.3.7-1.src.rpm |
| 166 |
|
| 167 |
1.1. Optionally for GPU Direct support run this command from your $PWD to |
| 168 |
generate rpm, srpm files |
| 169 |
$ ./makesrpm.sh a -cuda |
| 170 |
|
| 171 |
This command results in the following collection of rpm's and source code |
| 172 |
rpm's under your $PWD/temp.X/ directory. ("X" is the pid of the bash |
| 173 |
script that created the srpm and rpm files): |
| 174 |
RPMS/x86_64/libpsm2-10.3.7-1cuda.x86_64.rpm |
| 175 |
RPMS/x86_64/libpsm2-compat-10.3.7-1cuda.x86_64.rpm |
| 176 |
RPMS/x86_64/libpsm2-devel-10.3.7-1cuda.x86_64.rpm |
| 177 |
SRPMS/x86_64/libpsm2-10.3.7-1cuda.src.rpm |
| 178 |
|
| 179 |
On systems with SLES 12.3 or newer, the package name for the base libpsm2 |
| 180 |
RPM will be: |
| 181 |
libpsm2-2-10.3.7-1.x86_64.rpm |
| 182 |
|
| 183 |
Other supporting RPM package names will be as listed above. |
| 184 |
|
| 185 |
2. To build rpm files from the srpm file with Intel C (icc), specify the |
| 186 |
correct CCARCH in the rpmbuild environment: |
| 187 |
$ env CCARCH=icc rpmbuild --rebuild SRPMS/libpsm2-10.3.7-1.src.rpm |
| 188 |
|
| 189 |
INSTALLING |
| 190 |
========== |
| 191 |
|
| 192 |
INSTALLING USING MAKEFILE |
| 193 |
------------------------- |
| 194 |
|
| 195 |
Install the libraries and header files on the system (as root): |
| 196 |
$ make install |
| 197 |
|
| 198 |
The libraries will be installed in /usr/lib64, and the header files will |
| 199 |
be installed in /usr/include. |
| 200 |
|
| 201 |
This behavior can be altered by using the "DESTDIR" and "LIBDIR" variables on |
| 202 |
the "make install" command line. "DESTDIR" will add a leading path component |
| 203 |
to the overall install path and "LIBDIR" will change the path where libraries |
| 204 |
will be installed. For example, "make DESTDIR=/tmp/psm-install install" will |
| 205 |
install all files (libraries and headers) into "/tmp/psm-install/usr/...", |
| 206 |
"make DESTDIR=/tmp/psm-install LIBDIR=/libraries install" will install the |
| 207 |
libraries in "/tmp/psm-install/libraries" and the headers in |
| 208 |
"/tmp/psm-install/usr/include", and "make LIBDIR=/tmp/libs install" will |
| 209 |
install the libraries in "/tmp/libs" and the headers in "/usr/include". |
| 210 |
|
| 211 |
|
| 212 |
INSTALLING USING EITHER YUM OR DNF |
| 213 |
---------------------------------- |
| 214 |
|
| 215 |
You can install the rpm's and source rpm's previously built using rpmbuild using |
| 216 |
either the yum or dnf command as the root user. See the appropriate man page for |
| 217 |
details of installing rpm's. |
| 218 |
|
| 219 |
Note: It is also possible to use rpm command to install rpm's, but it is recommended |
| 220 |
that one use yum/dnf as rpm tool has issues with name changes and obsoletes tags. |
| 221 |
yum or dnf should be better able to resolve dependency issues. |
| 222 |
|
| 223 |
RELATED SOFTWARE TO PSM2 |
| 224 |
======================== |
| 225 |
|
| 226 |
MPI Libraries supported |
| 227 |
----------------------- |
| 228 |
A large number of open source (Open MPI, MVAPICH2) and Vendor MPI |
| 229 |
implementations support PSM2 for optimized communication on HCAs. Vendor MPI |
| 230 |
implementations (HP-MPI, Intel MPI 4.0 with PMI, Platform/Scali MPI) |
| 231 |
require that the PSM2 runtime libraries be installed and available on |
| 232 |
each node. Usually a configuration file or a command line switch to mpirun |
| 233 |
needs to be specified to utilize the PSM2 transport. |
| 234 |
|
| 235 |
Open MPI support |
| 236 |
--------------- |
| 237 |
If using a version of Open MPI that is not packaged within IFS release, it |
| 238 |
is required to use at least v1.10.4. Older versions are not supported. Since |
| 239 |
v1.10.4 is not in active development, it is further recommended to use upstream |
| 240 |
versions v2.1.2 or newer. |
| 241 |
|
| 242 |
If NVIDIA* CUDA* support is desired, you can use Open MPI built with CUDA* |
| 243 |
support provided by Intel in the IFS installer 10.4 or newer. This Open MPI |
| 244 |
build is identified with the "-cuda-hfi" tag to the Open MPI base version |
| 245 |
name. The NVIDIA* CUDA* support changes have also been accepted into v2.1.3, |
| 246 |
v3.0.1 and v3.1.0 branches of upstream Open MPI repository. |
| 247 |
|
| 248 |
PSM2 header and runtime files need to be installed on a node where the Open MPI |
| 249 |
build is performed. All compute nodes additionally should have the PSM2 runtime |
| 250 |
libraries available on them. Open MPI provides a standard configure, make and |
| 251 |
make install mechanism which will detect and build the relevant PSM2 network |
| 252 |
modules for Open MPI once the header and runtime files are detected. |
| 253 |
|
| 254 |
MVAPICH2 support |
| 255 |
---------------- |
| 256 |
MVAPICH2 supports PSM2 transport for optimized communication on HFI hardware. |
| 257 |
OPA IFS supports MVAPICH2 v2.1 (or later). PSM2 header and runtime files |
| 258 |
need to be installed on a node where MVAPICH2 builds are performed. All |
| 259 |
compute nodes should also have the PSM2 runtime libraries available on them. |
| 260 |
|
| 261 |
For building and installing MVAPICH2 with OPA support, refer to MVAPICH2 |
| 262 |
user guides here: |
| 263 |
http://mvapich.cse.ohio-state.edu/userguide/ |
| 264 |
|
| 265 |
(Note: Support for PSM2 is included in v2.2 and newer) |
| 266 |
|
| 267 |
OFED Support |
| 268 |
------------ |
| 269 |
Intel OPA is not yet included within OFED. But the hfi1 driver is available |
| 270 |
publicly at kernel.org. |
| 271 |
|
| 272 |
SUPPORTING DOCUMENTATION |
| 273 |
------------------------ |
| 274 |
PSM2 Programmer's Guide is published along with documentation for "IntelĀ® Omni-Path |
| 275 |
Host Fabric Interface PCIe Adapter 100 Series" |
| 276 |
(https://www.intel.com/content/www/us/en/support/articles/000016242/network-and-i-o/fabric-products.html) |
| 277 |
|
| 278 |
Refer to this document for description on APIs and environment variables that |
| 279 |
are available for use. For sample code on writing applications leveraging the |
| 280 |
PSM2 APIs, refer to Section 5. |
| 281 |
|
| 282 |
PSM Compatibility Support |
| 283 |
------------ |
| 284 |
|
| 285 |
libpsm2-compat suppports applications that use the PSM API instead of |
| 286 |
the PSM2 API, through a compatibility library. This library is an interface |
| 287 |
between PSM applications and the PSM2 API. |
| 288 |
|
| 289 |
If the system has an application that is coded to use PSM and has requirements |
| 290 |
to use PSM2 (i.e. the host has Omni-Path hardware), the compatibility library |
| 291 |
must be used. |
| 292 |
|
| 293 |
Please refer to your operating system's documentation to find how to modify the |
| 294 |
order in which system directories are searched for dynamic libraries. The |
| 295 |
libpsm2-compat version of libpsm_infinipath.so.1 must be earlier on the search |
| 296 |
path than that of libpsm-infinipath. Doing so allows applications coded to PSM |
| 297 |
to transparently use the PSM2 API and devices which require it. |
| 298 |
|
| 299 |
Please note that the installation path for the libpsm2-compat version of |
| 300 |
libpsm_infinipath.so.1 will differ depending on your operating system |
| 301 |
specifics. Common locations include: |
| 302 |
- /usr/lib64/psm2-compat/ |
| 303 |
- /usr/lib/psm2-compat/ |
| 304 |
|