Debian Codegraph @

README

raw
1
2 This file is provided under a dual BSD/GPLv2 license. When using or
3 redistributing this file, you may do so under either license.
4
5 GPL LICENSE SUMMARY
6
7 Copyright(c) 2017 Intel Corporation.
8
9 This program is free software; you can redistribute it and/or modify
10 it under the terms of version 2 of the GNU General Public License as
11 published by the Free Software Foundation.
12
13 This program is distributed in the hope that it will be useful, but
14 WITHOUT ANY WARRANTY; without even the implied warranty of
15 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
16 General Public License for more details.
17
18 Contact Information:
19 Intel Corporation, www.intel.com
20
21 BSD LICENSE
22
23 Copyright(c) 2017 Intel Corporation.
24
25 Redistribution and use in source and binary forms, with or without
26 modification, are permitted provided that the following conditions
27 are met:
28
29 * Redistributions of source code must retain the above copyright
30 notice, this list of conditions and the following disclaimer.
31 * Redistributions in binary form must reproduce the above copyright
32 notice, this list of conditions and the following disclaimer in
33 the documentation and/or other materials provided with the
34 distribution.
35 * Neither the name of Intel Corporation nor the names of its
36 contributors may be used to endorse or promote products derived
37 from this software without specific prior written permission.
38
39 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
40 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
41 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
42 A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
43 OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
44 SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
45 LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
46 DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
47 THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
48 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
49 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
50
51 Copyright (c) 2003-2017 Intel Corporation. All rights reserved.
52
53 ================================================================================
54
55 ABSTRACT
56 --------
57
58 Discusses how to build, install and test the PSM2 library source code.
59
60 Contains the following sections:
61
62 - INTRODUCTION
63 - DEPENDENCIES
64 - BUILDING
65 * BUILDING USING MAKEFILE
66 * BUILDING USING RPMBUILD (CREATING SOURCE AND BINARY RPM'S)
67 - INSTALLING
68 * INSTALLING USING MAKEFILE
69 * INSTALLING USING EITHER YUM OR DNF
70 - TESTING
71 - RELATED SOFTWARE TO PSM2
72 - SUPPORTING DOCUMENTATION
73
74 INTRODUCTION
75 ============
76
77 This README file discusses how to build, install and test the PSM2 library
78 source code.
79
80 The PSM2 library supports a number of fabric media and stacks, and all of
81 them run on version 7.X of Red Hat Enterprise Linux (abbreviated: RHEL), and
82 SuSE SLES.
83
84 Only the x86_64 architecture is supported.
85
86 Building PSM2 is possible on RHEL 7.2+ as it ships with hfi1 kernel driver.
87 For older RHEL 7.x versions and SuSE SLES, OPA is not natively supported
88 in the kernel and therefore, building PSM2 is not possible unless
89 you have the correct kernel-devel package or use latest versions of IFS.
90
91 There are two mechanisms for building and installing the PSM2 library:
92
93 1. Use provided Makefiles to build and install or
94 2. Generate the *.rpm files which you can then install using either
95 yum or dnf command
96
97 DEPENDENCIES
98 ============
99
100 The following packages are required to build the PSM2 library source code:
101 (all packages are for the x86_64 architecture)
102
103 compat-rdma-devel
104 gcc-4.8.2
105 glibc-devel
106 glibc-headers
107 kernel-headers
108
109 Additional packages for GPU Direct support include:
110 NVIDIA CUDA toolkit 8.0 or greater. Older versions are not supported.
111
112 In addition to depending on these packages, root privileges are required to
113 install the runtime libraries and development header files into standard
114 system location.
115
116 BUILDING
117 ========
118
119 The instructions below use $BASENAME, $PRODUCT and $RELEASE to refer to
120 the base name of the tarball, RPM that will be generated and the product
121 and release identifiers of the RPM.
122
123 The base name of the RPM changes depending on which version/branch
124 of code you derive the tar file from.
125
126 Up until v10.2 of PSM2, the base name for the RPM is hfi1-psm.
127 From v10.2 onwards, the base name will be libpsm2. The internal
128 library remains unchanged and is still libpsm2.so.2.
129
130 BUILDING USING MAKEFILES
131 ------------------------
132
133 1. Untar the tarball:
134 $ tar zxvf $BASENAME-$PRODUCT-$RELEASE.tar.gz
135 2. Change directory into the untarred location:
136 $ cd $BASENAME-$PRODUCT-$RELEASE
137 3. Build:
138 3.1. To build with GNU C (gcc), run make on the command line:
139 $ make
140 - or -
141 $ make CCARCH=gcc
142 3.2 To build with Intel C (icc), specify the correct CCARCH:
143 $ make CCARCH=icc
144 3.3. To build with CUDA support, specify PSM_CUDA=1
145 on the command line along with the desired compiler:
146 $ make PSM_CUDA=1 CCARCH=gcc
147 - or -
148 $ make PSM_CUDA=1 CCARCH=icc
149
150 BUILDING USING RPMBUILD
151 -----------------------
152
153 1. Run this command from your $PWD to generate rpm, srpm files
154 $ ./makesrpm.sh a
155
156 This command results in the following collection of rpm's and source
157 code rpm's under your $PWD/temp.X/ directory.
158 ("X" is the pid of the bash script that created the srpm and rpm files)
159 (Result shown here for RHEL systems.)
160
161 RPMS/x86_64/libpsm2-compat-10.3.7-1x86_64.rpm
162 RPMS/x86_64/libpsm2-devel-10.3.7-1x86_64.rpm
163 RPMS/x86_64/libpsm2-10.3.7-1x86_64.rpm
164 RPMS/x86_64/libpsm2-debuginfo-10.3.7-1x86_64.rpm
165 SRPMS/libpsm2-10.3.7-1.src.rpm
166
167 1.1. Optionally for GPU Direct support run this command from your $PWD to
168 generate rpm, srpm files
169 $ ./makesrpm.sh a -cuda
170
171 This command results in the following collection of rpm's and source code
172 rpm's under your $PWD/temp.X/ directory. ("X" is the pid of the bash
173 script that created the srpm and rpm files):
174 RPMS/x86_64/libpsm2-10.3.7-1cuda.x86_64.rpm
175 RPMS/x86_64/libpsm2-compat-10.3.7-1cuda.x86_64.rpm
176 RPMS/x86_64/libpsm2-devel-10.3.7-1cuda.x86_64.rpm
177 SRPMS/x86_64/libpsm2-10.3.7-1cuda.src.rpm
178
179 On systems with SLES 12.3 or newer, the package name for the base libpsm2
180 RPM will be:
181 libpsm2-2-10.3.7-1.x86_64.rpm
182
183 Other supporting RPM package names will be as listed above.
184
185 2. To build rpm files from the srpm file with Intel C (icc), specify the
186 correct CCARCH in the rpmbuild environment:
187 $ env CCARCH=icc rpmbuild --rebuild SRPMS/libpsm2-10.3.7-1.src.rpm
188
189 INSTALLING
190 ==========
191
192 INSTALLING USING MAKEFILE
193 -------------------------
194
195 Install the libraries and header files on the system (as root):
196 $ make install
197
198 The libraries will be installed in /usr/lib64, and the header files will
199 be installed in /usr/include.
200
201 This behavior can be altered by using the "DESTDIR" and "LIBDIR" variables on
202 the "make install" command line. "DESTDIR" will add a leading path component
203 to the overall install path and "LIBDIR" will change the path where libraries
204 will be installed. For example, "make DESTDIR=/tmp/psm-install install" will
205 install all files (libraries and headers) into "/tmp/psm-install/usr/...",
206 "make DESTDIR=/tmp/psm-install LIBDIR=/libraries install" will install the
207 libraries in "/tmp/psm-install/libraries" and the headers in
208 "/tmp/psm-install/usr/include", and "make LIBDIR=/tmp/libs install" will
209 install the libraries in "/tmp/libs" and the headers in "/usr/include".
210
211
212 INSTALLING USING EITHER YUM OR DNF
213 ----------------------------------
214
215 You can install the rpm's and source rpm's previously built using rpmbuild using
216 either the yum or dnf command as the root user. See the appropriate man page for
217 details of installing rpm's.
218
219 Note: It is also possible to use rpm command to install rpm's, but it is recommended
220 that one use yum/dnf as rpm tool has issues with name changes and obsoletes tags.
221 yum or dnf should be better able to resolve dependency issues.
222
223 RELATED SOFTWARE TO PSM2
224 ========================
225
226 MPI Libraries supported
227 -----------------------
228 A large number of open source (Open MPI, MVAPICH2) and Vendor MPI
229 implementations support PSM2 for optimized communication on HCAs. Vendor MPI
230 implementations (HP-MPI, Intel MPI 4.0 with PMI, Platform/Scali MPI)
231 require that the PSM2 runtime libraries be installed and available on
232 each node. Usually a configuration file or a command line switch to mpirun
233 needs to be specified to utilize the PSM2 transport.
234
235 Open MPI support
236 ---------------
237 If using a version of Open MPI that is not packaged within IFS release, it
238 is required to use at least v1.10.4. Older versions are not supported. Since
239 v1.10.4 is not in active development, it is further recommended to use upstream
240 versions v2.1.2 or newer.
241
242 If NVIDIA* CUDA* support is desired, you can use Open MPI built with CUDA*
243 support provided by Intel in the IFS installer 10.4 or newer. This Open MPI
244 build is identified with the "-cuda-hfi" tag to the Open MPI base version
245 name. The NVIDIA* CUDA* support changes have also been accepted into v2.1.3,
246 v3.0.1 and v3.1.0 branches of upstream Open MPI repository.
247
248 PSM2 header and runtime files need to be installed on a node where the Open MPI
249 build is performed. All compute nodes additionally should have the PSM2 runtime
250 libraries available on them. Open MPI provides a standard configure, make and
251 make install mechanism which will detect and build the relevant PSM2 network
252 modules for Open MPI once the header and runtime files are detected.
253
254 MVAPICH2 support
255 ----------------
256 MVAPICH2 supports PSM2 transport for optimized communication on HFI hardware.
257 OPA IFS supports MVAPICH2 v2.1 (or later). PSM2 header and runtime files
258 need to be installed on a node where MVAPICH2 builds are performed. All
259 compute nodes should also have the PSM2 runtime libraries available on them.
260
261 For building and installing MVAPICH2 with OPA support, refer to MVAPICH2
262 user guides here:
263 http://mvapich.cse.ohio-state.edu/userguide/
264
265 (Note: Support for PSM2 is included in v2.2 and newer)
266
267 OFED Support
268 ------------
269 Intel OPA is not yet included within OFED. But the hfi1 driver is available
270 publicly at kernel.org.
271
272 SUPPORTING DOCUMENTATION
273 ------------------------
274 PSM2 Programmer's Guide is published along with documentation for "IntelĀ® Omni-Path
275 Host Fabric Interface PCIe Adapter 100 Series"
276 (https://www.intel.com/content/www/us/en/support/articles/000016242/network-and-i-o/fabric-products.html)
277
278 Refer to this document for description on APIs and environment variables that
279 are available for use. For sample code on writing applications leveraging the
280 PSM2 APIs, refer to Section 5.
281
282 PSM Compatibility Support
283 ------------
284
285 libpsm2-compat suppports applications that use the PSM API instead of
286 the PSM2 API, through a compatibility library. This library is an interface
287 between PSM applications and the PSM2 API.
288
289 If the system has an application that is coded to use PSM and has requirements
290 to use PSM2 (i.e. the host has Omni-Path hardware), the compatibility library
291 must be used.
292
293 Please refer to your operating system's documentation to find how to modify the
294 order in which system directories are searched for dynamic libraries. The
295 libpsm2-compat version of libpsm_infinipath.so.1 must be earlier on the search
296 path than that of libpsm-infinipath. Doing so allows applications coded to PSM
297 to transparently use the PSM2 API and devices which require it.
298
299 Please note that the installation path for the libpsm2-compat version of
300 libpsm_infinipath.so.1 will differ depending on your operating system
301 specifics. Common locations include:
302 - /usr/lib64/psm2-compat/
303 - /usr/lib/psm2-compat/
304