Debian Codegraph @

README

raw
1 This is a truncated and poorly-formatted version of the documentation main page.
2 See https://www.open-mpi.org/projects/hwloc/doc/ for more.
3
4
5 hwloc Overview
6
7 The Hardware Locality (hwloc) software project aims at easing the process of
8 discovering hardware resources in parallel architectures. It offers
9 command-line tools and a C API for consulting these resources, their locality,
10 attributes, and interconnection. hwloc primarily aims at helping
11 high-performance computing (HPC) applications, but is also applicable to any
12 project seeking to exploit code and/or data locality on modern computing
13 platforms.
14
15 hwloc provides command line tools and a C API to obtain the hierarchical map of
16 key computing elements within a node, such as: NUMA memory nodes, shared
17 caches, processor packages, dies and cores, processing units (logical
18 processors or "threads") and even I/O devices. hwloc also gathers various
19 attributes such as cache and memory information, and is portable across a
20 variety of different operating systems and platforms.
21
22 hwloc primarily aims at helping high-performance computing (HPC) applications,
23 but is also applicable to any project seeking to exploit code and/or data
24 locality on modern computing platforms.
25
26 hwloc supports the following operating systems:
27
28 * Linux (with knowledge of cgroups and cpusets, memory targets/initiators,
29 etc.) on all supported hardware, including Intel Xeon Phi, ScaleMP vSMP,
30 and NumaScale NumaConnect.
31 * Solaris (with support for processor sets and logical domains)
32 * AIX
33 * Darwin / OS X
34 * FreeBSD and its variants (such as kFreeBSD/GNU)
35 * NetBSD
36 * HP-UX
37 * Microsoft Windows
38 * IBM BlueGene/Q Compute Node Kernel (CNK)
39
40 Since it uses standard Operating System information, hwloc's support is mostly
41 independant from the processor type (x86, powerpc, ...) and just relies on the
42 Operating System support. The main exception is BSD operating systems (NetBSD,
43 FreeBSD, etc.) because they do not provide support topology information, hence
44 hwloc uses an x86-only CPUID-based backend (which can be used for other OSes
45 too, see the Components and plugins section).
46
47 To check whether hwloc works on a particular machine, just try to build it and
48 run lstopo or lstopo-no-graphics. If some things do not look right (e.g. bogus
49 or missing cache information), see Questions and Bugs.
50
51 hwloc only reports the number of processors on unsupported operating systems;
52 no topology information is available.
53
54 For development and debugging purposes, hwloc also offers the ability to work
55 on "fake" topologies:
56
57 * Symmetrical tree of resources generated from a list of level arities, see
58 Synthetic topologies.
59 * Remote machine simulation through the gathering of topology as XML files,
60 see Importing and exporting topologies from/to XML files.
61
62 hwloc can display the topology in a human-readable format, either in graphical
63 mode (X11), or by exporting in one of several different formats, including:
64 plain text, LaTeX tikzpicture, PDF, PNG, and FIG (see Command-line Examples
65 below). Note that some of the export formats require additional support
66 libraries.
67
68 hwloc offers a programming interface for manipulating topologies and objects.
69 It also brings a powerful CPU bitmap API that is used to describe topology
70 objects location on physical/logical processors. See the Programming Interface
71 below. It may also be used to binding applications onto certain cores or memory
72 nodes. Several utility programs are also provided to ease command-line
73 manipulation of topology objects, binding of processes, and so on.
74
75 Bindings for several other languages are available from the project website.
76
77 Command-line Examples
78
79 On a 4-package 2-core machine with hyper-threading, the lstopo tool may show
80 the following graphical output:
81
82 [dudley]
83
84 Here's the equivalent output in textual form:
85
86 Machine
87 NUMANode L#0 (P#0)
88 Package L#0 + L3 L#0 (4096KB)
89 L2 L#0 (1024KB) + L1 L#0 (16KB) + Core L#0
90 PU L#0 (P#0)
91 PU L#1 (P#8)
92 L2 L#1 (1024KB) + L1 L#1 (16KB) + Core L#1
93 PU L#2 (P#4)
94 PU L#3 (P#12)
95 Package L#1 + L3 L#1 (4096KB)
96 L2 L#2 (1024KB) + L1 L#2 (16KB) + Core L#2
97 PU L#4 (P#1)
98 PU L#5 (P#9)
99 L2 L#3 (1024KB) + L1 L#3 (16KB) + Core L#3
100 PU L#6 (P#5)
101 PU L#7 (P#13)
102 Package L#2 + L3 L#2 (4096KB)
103 L2 L#4 (1024KB) + L1 L#4 (16KB) + Core L#4
104 PU L#8 (P#2)
105 PU L#9 (P#10)
106 L2 L#5 (1024KB) + L1 L#5 (16KB) + Core L#5
107 PU L#10 (P#6)
108 PU L#11 (P#14)
109 Package L#3 + L3 L#3 (4096KB)
110 L2 L#6 (1024KB) + L1 L#6 (16KB) + Core L#6
111 PU L#12 (P#3)
112 PU L#13 (P#11)
113 L2 L#7 (1024KB) + L1 L#7 (16KB) + Core L#7
114 PU L#14 (P#7)
115 PU L#15 (P#15)
116
117 Note that there is also an equivalent output in XML that is meant for exporting
118 /importing topologies but it is hardly readable to human-beings (see Importing
119 and exporting topologies from/to XML files for details).
120
121 On a 4-package 2-core Opteron NUMA machine (with two core cores disallowed by
122 the administrator), the lstopo tool may show the following graphical output
123 (with --disallowed for displaying disallowed objects):
124
125 [hagrid]
126
127 Here's the equivalent output in textual form:
128
129 Machine (32GB total)
130 Package L#0
131 NUMANode L#0 (P#0 8190MB)
132 L2 L#0 (1024KB) + L1 L#0 (64KB) + Core L#0 + PU L#0 (P#0)
133 L2 L#1 (1024KB) + L1 L#1 (64KB) + Core L#1 + PU L#1 (P#1)
134 Package L#1
135 NUMANode L#1 (P#1 8192MB)
136 L2 L#2 (1024KB) + L1 L#2 (64KB) + Core L#2 + PU L#2 (P#2)
137 L2 L#3 (1024KB) + L1 L#3 (64KB) + Core L#3 + PU L#3 (P#3)
138 Package L#2
139 NUMANode L#2 (P#2 8192MB)
140 L2 L#4 (1024KB) + L1 L#4 (64KB) + Core L#4 + PU L#4 (P#4)
141 L2 L#5 (1024KB) + L1 L#5 (64KB) + Core L#5 + PU L#5 (P#5)
142 Package L#3
143 NUMANode L#3 (P#3 8192MB)
144 L2 L#6 (1024KB) + L1 L#6 (64KB) + Core L#6 + PU L#6 (P#6)
145 L2 L#7 (1024KB) + L1 L#7 (64KB) + Core L#7 + PU L#7 (P#7)
146
147 On a 2-package quad-core Xeon (pre-Nehalem, with 2 dual-core dies into each
148 package):
149
150 [emmett]
151
152 Here's the same output in textual form:
153
154 Machine (total 16GB)
155 NUMANode L#0 (P#0 16GB)
156 Package L#0
157 L2 L#0 (4096KB)
158 L1 L#0 (32KB) + Core L#0 + PU L#0 (P#0)
159 L1 L#1 (32KB) + Core L#1 + PU L#1 (P#4)
160 L2 L#1 (4096KB)
161 L1 L#2 (32KB) + Core L#2 + PU L#2 (P#2)
162 L1 L#3 (32KB) + Core L#3 + PU L#3 (P#6)
163 Package L#1
164 L2 L#2 (4096KB)
165 L1 L#4 (32KB) + Core L#4 + PU L#4 (P#1)
166 L1 L#5 (32KB) + Core L#5 + PU L#5 (P#5)
167 L2 L#3 (4096KB)
168 L1 L#6 (32KB) + Core L#6 + PU L#6 (P#3)
169 L1 L#7 (32KB) + Core L#7 + PU L#7 (P#7)
170
171 Programming Interface
172
173 The basic interface is available in hwloc.h. Some higher-level functions are
174 available in hwloc/helper.h to reduce the need to manually manipulate objects
175 and follow links between them. Documentation for all these is provided later in
176 this document. Developers may also want to look at hwloc/inlines.h which
177 contains the actual inline code of some hwloc.h routines, and at this document,
178 which provides good higher-level topology traversal examples.
179
180 To precisely define the vocabulary used by hwloc, a Terms and Definitions
181 section is available and should probably be read first.
182
183 Each hwloc object contains a cpuset describing the list of processing units
184 that it contains. These bitmaps may be used for CPU binding and Memory binding.
185 hwloc offers an extensive bitmap manipulation interface in hwloc/bitmap.h.
186
187 Moreover, hwloc also comes with additional helpers for interoperability with
188 several commonly used environments. See the Interoperability With Other
189 Software section for details.
190
191 The complete API documentation is available in a full set of HTML pages, man
192 pages, and self-contained PDF files (formatted for both both US letter and A4
193 formats) in the source tarball in doc/doxygen-doc/.
194
195 NOTE: If you are building the documentation from a Git clone, you will need to
196 have Doxygen and pdflatex installed -- the documentation will be built during
197 the normal "make" process. The documentation is installed during "make install"
198 to $prefix/share/doc/hwloc/ and your systems default man page tree (under
199 $prefix, of course).
200
201 Portability
202
203 Operating System have varying support for CPU and memory binding, e.g. while
204 some Operating Systems provide interfaces for all kinds of CPU and memory
205 bindings, some others provide only interfaces for a limited number of kinds of
206 CPU and memory binding, and some do not provide any binding interface at all.
207 Hwloc's binding functions would then simply return the ENOSYS error (Function
208 not implemented), meaning that the underlying Operating System does not provide
209 any interface for them. CPU binding and Memory binding provide more information
210 on which hwloc binding functions should be preferred because interfaces for
211 them are usually available on the supported Operating Systems.
212
213 Similarly, the ability of reporting topology information varies from one
214 platform to another. As shown in Command-line Examples, hwloc can obtain
215 information on a wide variety of hardware topologies. However, some platforms
216 and/or operating system versions will only report a subset of this information.
217 For example, on an PPC64-based system with 8 cores (each with 2 hardware
218 threads) running a default 2.6.18-based kernel from RHEL 5.4, hwloc is only
219 able to glean information about NUMA nodes and processor units (PUs). No
220 information about caches, packages, or cores is available.
221
222 Here's the graphical output from lstopo on this platform when Simultaneous
223 Multi-Threading (SMT) is enabled:
224
225 [ppc64-with]
226
227 And here's the graphical output from lstopo on this platform when SMT is
228 disabled:
229
230 [ppc64-with]
231
232 Notice that hwloc only sees half the PUs when SMT is disabled. PU L#6, for
233 example, seems to change location from NUMA node #0 to #1. In reality, no PUs
234 "moved" -- they were simply re-numbered when hwloc only saw half as many (see
235 also Logical index in Indexes and Sets). Hence, PU L#6 in the SMT-disabled
236 picture probably corresponds to PU L#12 in the SMT-enabled picture.
237
238 This same "PUs have disappeared" effect can be seen on other platforms -- even
239 platforms / OSs that provide much more information than the above PPC64 system.
240 This is an unfortunate side-effect of how operating systems report information
241 to hwloc.
242
243 Note that upgrading the Linux kernel on the same PPC64 system mentioned above
244 to 2.6.34, hwloc is able to discover all the topology information. The
245 following picture shows the entire topology layout when SMT is enabled:
246
247 [ppc64-full]
248
249 Developers using the hwloc API or XML output for portable applications should
250 therefore be extremely careful to not make any assumptions about the structure
251 of data that is returned. For example, per the above reported PPC topology, it
252 is not safe to assume that PUs will always be descendants of cores.
253
254 Additionally, future hardware may insert new topology elements that are not
255 available in this version of hwloc. Long-lived applications that are meant to
256 span multiple different hardware platforms should also be careful about making
257 structure assumptions. For example, a new element may someday exist between a
258 core and a PU.
259
260 API Example
261
262 The following small C example (available in the source tree as ``doc/examples/
263 hwloc-hello.c'') prints the topology of the machine and performs some thread
264 and memory binding. More examples are available in the doc/examples/ directory
265 of the source tree.
266
267 /*
268 * SPDX-License-Identifier: BSD-3-Clause
269 * Copyright (c) 2009-2016 Inria. All rights reserved.
270 * Copyright (c) 2009-2011 Universit?eacute; Bordeaux
271 * Copyright (c) 2009-2010 Cisco Systems, Inc. All rights reserved.
272 * See COPYING in top-level directory.
273 *
274 * Example hwloc API program.
275 *
276 * See other examples under doc/examples/ in the source tree
277 * for more details.
278 *
279 * hwloc-hello.c
280 */
281 #include "hwloc.h"
282 #include <errno.h>
283 #include <stdio.h>
284 #include <string.h>
285 static void print_children(hwloc_topology_t topology, hwloc_obj_t obj,
286 int depth)
287 {
288 char type[32], attr[1024];
289 unsigned i;
290 hwloc_obj_type_snprintf(type, sizeof(type), obj, 0);
291 printf("%*s%s", 2*depth, "", type);
292 if (obj->os_index != (unsigned) -1)
293 printf("#%u", obj->os_index);
294 hwloc_obj_attr_snprintf(attr, sizeof(attr), obj, " ", 0);
295 if (*attr)
296 printf("(%s)", attr);
297 printf("\n");
298 for (i = 0; i < obj->arity; i++) {
299 print_children(topology, obj->children[i], depth + 1);
300 }
301 }
302 int main(void)
303 {
304 int depth;
305 unsigned i, n;
306 unsigned long size;
307 int levels;
308 char string[128];
309 int topodepth;
310 void *m;
311 hwloc_topology_t topology;
312 hwloc_cpuset_t cpuset;
313 hwloc_obj_t obj;
314 /* Allocate and initialize topology object. */
315 hwloc_topology_init(&topology);
316 /* ... Optionally, put detection configuration here to ignore
317 some objects types, define a synthetic topology, etc....
318 The default is to detect all the objects of the machine that
319 the caller is allowed to access. See Configure Topology
320 Detection. */
321 /* Perform the topology detection. */
322 hwloc_topology_load(topology);
323 /* Optionally, get some additional topology information
324 in case we need the topology depth later. */
325 topodepth = hwloc_topology_get_depth(topology);
326 /*****************************************************************
327 * First example:
328 * Walk the topology with an array style, from level 0 (always
329 * the system level) to the lowest level (always the proc level).
330 *****************************************************************/
331 for (depth = 0; depth < topodepth; depth++) {
332 printf("*** Objects at level %d\n", depth);
333 for (i = 0; i < hwloc_get_nbobjs_by_depth(topology, depth);
334 i++) {
335 hwloc_obj_type_snprintf(string, sizeof(string),
336 hwloc_get_obj_by_depth(topology, depth, i), 0);
337 printf("Index %u: %s\n", i, string);
338 }
339 }
340 /*****************************************************************
341 * Second example:
342 * Walk the topology with a tree style.
343 *****************************************************************/
344 printf("*** Printing overall tree\n");
345 print_children(topology, hwloc_get_root_obj(topology), 0);
346 /*****************************************************************
347 * Third example:
348 * Print the number of packages.
349 *****************************************************************/
350 depth = hwloc_get_type_depth(topology, HWLOC_OBJ_PACKAGE);
351 if (depth == HWLOC_TYPE_DEPTH_UNKNOWN) {
352 printf("*** The number of packages is unknown\n");
353 } else {
354 printf("*** %u package(s)\n",
355 hwloc_get_nbobjs_by_depth(topology, depth));
356 }
357 /*****************************************************************
358 * Fourth example:
359 * Compute the amount of cache that the first logical processor
360 * has above it.
361 *****************************************************************/
362 levels = 0;
363 size = 0;
364 for (obj = hwloc_get_obj_by_type(topology, HWLOC_OBJ_PU, 0);
365 obj;
366 obj = obj->parent)
367 if (hwloc_obj_type_is_cache(obj->type)) {
368 levels++;
369 size += obj->attr->cache.size;
370 }
371 printf("*** Logical processor 0 has %d caches totaling %luKB\n",
372 levels, size / 1024);
373 /*****************************************************************
374 * Fifth example:
375 * Bind to only one thread of the last core of the machine.
376 *
377 * First find out where cores are, or else smaller sets of CPUs if
378 * the OS doesn't have the notion of a "core".
379 *****************************************************************/
380 depth = hwloc_get_type_or_below_depth(topology, HWLOC_OBJ_CORE);
381 /* Get last core. */
382 obj = hwloc_get_obj_by_depth(topology, depth,
383 hwloc_get_nbobjs_by_depth(topology, depth) - 1);
384 if (obj) {
385 /* Get a copy of its cpuset that we may modify. */
386 cpuset = hwloc_bitmap_dup(obj->cpuset);
387 /* Get only one logical processor (in case the core is
388 SMT/hyper-threaded). */
389 hwloc_bitmap_singlify(cpuset);
390 /* And try to bind ourself there. */
391 if (hwloc_set_cpubind(topology, cpuset, 0)) {
392 char *str;
393 int error = errno;
394 hwloc_bitmap_asprintf(&str, obj->cpuset);
395 printf("Couldn't bind to cpuset %s: %s\n", str, strerror(error));
396 free(str);
397 }
398 /* Free our cpuset copy */
399 hwloc_bitmap_free(cpuset);
400 }
401 /*****************************************************************
402 * Sixth example:
403 * Allocate some memory on the last NUMA node, bind some existing
404 * memory to the last NUMA node.
405 *****************************************************************/
406 /* Get last node. There's always at least one. */
407 n = hwloc_get_nbobjs_by_type(topology, HWLOC_OBJ_NUMANODE);
408 obj = hwloc_get_obj_by_type(topology, HWLOC_OBJ_NUMANODE, n - 1);
409 size = 1024*1024;
410 m = hwloc_alloc_membind(topology, size, obj->nodeset,
411 HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_BYNODESET);
412 hwloc_free(topology, m, size);
413 m = malloc(size);
414 hwloc_set_area_membind(topology, m, size, obj->nodeset,
415 HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_BYNODESET);
416 free(m);
417 /* Destroy topology object. */
418 hwloc_topology_destroy(topology);
419 return 0;
420 }
421
422 hwloc provides a pkg-config executable to obtain relevant compiler and linker
423 flags. See Compiling software on top of hwloc's C API for details on building
424 program on top of hwloc's API using GNU Make or CMake.
425
426 On a machine 2 processor packages -- each package of which has two processing
427 cores -- the output from running hwloc-hello could be something like the
428 following:
429
430 shell$ ./hwloc-hello
431 Objects at level 0
432 Index 0: Machine
433 Objects at level 1
434 Index 0: Package#0
435 Index 1: Package#1
436 Objects at level 2
437 Index 0: Core#0
438 Index 1: Core#1
439 Index 2: Core#3
440 Index 3: Core#2
441 Objects at level 3
442 Index 0: PU#0
443 Index 1: PU#1
444 Index 2: PU#2
445 Index 3: PU#3
446 Printing overall tree
447 Machine
448 Package#0
449 Core#0
450 PU#0
451 Core#1
452 PU#1
453 Package#1
454 Core#3
455 PU#2
456 Core#2
457 PU#3
458 2 package(s)
459 Logical processor 0 has 0 caches totaling 0KB
460 shell$
461
462 Questions and Bugs
463
464 Bugs should be reported in the tracker (https://github.com/open-mpi/hwloc/
465 issues). Opening a new issue automatically displays lots of hints about how to
466 debug and report issues.
467
468 Questions may be sent to the users or developers mailing lists (https://
469 www.open-mpi.org/community/lists/hwloc.php).
470
471 There is also a #hwloc IRC channel on Libera Chat (irc.libera.chat).
472
473 History / Credits
474
475 hwloc is the evolution and merger of the libtopology project and the Portable
476 Linux Processor Affinity (PLPA) (https://www.open-mpi.org/projects/plpa/)
477 project. Because of functional and ideological overlap, these two code bases
478 and ideas were merged and released under the name "hwloc" as an Open MPI
479 sub-project.
480
481 libtopology was initially developed by the Inria Runtime Team-Project. PLPA was
482 initially developed by the Open MPI development team as a sub-project. Both are
483 now deprecated in favor of hwloc, which is distributed as an Open MPI
484 sub-project.
485
486
487
488
489 See https://www.open-mpi.org/projects/hwloc/doc/ for more hwloc documentation,
490 actual links to related pages, images, etc.