Debian Codegraph @

upstream/debian-trixie/man7/cpuset.7

raw
1 .\" Copyright (c) 2008 Silicon Graphics, Inc.
2 .\"
3 .\" Author: Paul Jackson (http://oss.sgi.com/projects/cpusets)
4 .\"
5 .\" SPDX-License-Identifier: GPL-2.0-only
6 .\"
7 .TH cpuset 7 2024-06-15 "Linux man-pages 6.9.1"
8 .SH NAME
9 cpuset \- confine processes to processor and memory node subsets
10 .SH DESCRIPTION
11 The cpuset filesystem is a pseudo-filesystem interface
12 to the kernel cpuset mechanism,
13 which is used to control the processor placement
14 and memory placement of processes.
15 It is commonly mounted at
16 .IR /dev/cpuset .
17 .P
18 On systems with kernels compiled with built in support for cpusets,
19 all processes are attached to a cpuset, and cpusets are always present.
20 If a system supports cpusets, then it will have the entry
21 .B nodev cpuset
22 in the file
23 .IR /proc/filesystems .
24 By mounting the cpuset filesystem (see the
25 .B EXAMPLES
26 section below),
27 the administrator can configure the cpusets on a system
28 to control the processor and memory placement of processes
29 on that system.
30 By default, if the cpuset configuration
31 on a system is not modified or if the cpuset filesystem
32 is not even mounted, then the cpuset mechanism,
33 though present, has no effect on the system's behavior.
34 .P
35 A cpuset defines a list of CPUs and memory nodes.
36 .P
37 The CPUs of a system include all the logical processing
38 units on which a process can execute, including, if present,
39 multiple processor cores within a package and Hyper-Threads
40 within a processor core.
41 Memory nodes include all distinct
42 banks of main memory; small and SMP systems typically have
43 just one memory node that contains all the system's main memory,
44 while NUMA (non-uniform memory access) systems have multiple memory nodes.
45 .P
46 Cpusets are represented as directories in a hierarchical
47 pseudo-filesystem, where the top directory in the hierarchy
48 .RI ( /dev/cpuset )
49 represents the entire system (all online CPUs and memory nodes)
50 and any cpuset that is the child (descendant) of
51 another parent cpuset contains a subset of that parent's
52 CPUs and memory nodes.
53 The directories and files representing cpusets have normal
54 filesystem permissions.
55 .P
56 Every process in the system belongs to exactly one cpuset.
57 A process is confined to run only on the CPUs in
58 the cpuset it belongs to, and to allocate memory only
59 on the memory nodes in that cpuset.
60 When a process
61 .BR fork (2)s,
62 the child process is placed in the same cpuset as its parent.
63 With sufficient privilege, a process may be moved from one
64 cpuset to another and the allowed CPUs and memory nodes
65 of an existing cpuset may be changed.
66 .P
67 When the system begins booting, a single cpuset is
68 defined that includes all CPUs and memory nodes on the
69 system, and all processes are in that cpuset.
70 During the boot process, or later during normal system operation,
71 other cpusets may be created, as subdirectories of this top cpuset,
72 under the control of the system administrator,
73 and processes may be placed in these other cpusets.
74 .P
75 Cpusets are integrated with the
76 .BR sched_setaffinity (2)
77 scheduling affinity mechanism and the
78 .BR mbind (2)
79 and
80 .BR set_mempolicy (2)
81 memory-placement mechanisms in the kernel.
82 Neither of these mechanisms let a process make use
83 of a CPU or memory node that is not allowed by that process's cpuset.
84 If changes to a process's cpuset placement conflict with these
85 other mechanisms, then cpuset placement is enforced
86 even if it means overriding these other mechanisms.
87 The kernel accomplishes this overriding by silently
88 restricting the CPUs and memory nodes requested by
89 these other mechanisms to those allowed by the
90 invoking process's cpuset.
91 This can result in these
92 other calls returning an error, if for example, such
93 a call ends up requesting an empty set of CPUs or
94 memory nodes, after that request is restricted to
95 the invoking process's cpuset.
96 .P
97 Typically, a cpuset is used to manage
98 the CPU and memory-node confinement for a set of
99 cooperating processes such as a batch scheduler job, and these
100 other mechanisms are used to manage the placement of
101 individual processes or memory regions within that set or job.
102 .SH FILES
103 Each directory below
104 .I /dev/cpuset
105 represents a cpuset and contains a fixed set of pseudo-files
106 describing the state of that cpuset.
107 .P
108 New cpusets are created using the
109 .BR mkdir (2)
110 system call or the
111 .BR mkdir (1)
112 command.
113 The properties of a cpuset, such as its flags, allowed
114 CPUs and memory nodes, and attached processes, are queried and modified
115 by reading or writing to the appropriate file in that cpuset's directory,
116 as listed below.
117 .P
118 The pseudo-files in each cpuset directory are automatically created when
119 the cpuset is created, as a result of the
120 .BR mkdir (2)
121 invocation.
122 It is not possible to directly add or remove these pseudo-files.
123 .P
124 A cpuset directory that contains no child cpuset directories,
125 and has no attached processes, can be removed using
126 .BR rmdir (2)
127 or
128 .BR rmdir (1).
129 It is not necessary, or possible,
130 to remove the pseudo-files inside the directory before removing it.
131 .P
132 The pseudo-files in each cpuset directory are
133 small text files that may be read and
134 written using traditional shell utilities such as
135 .BR cat (1),
136 and
137 .BR echo (1),
138 or from a program by using file I/O library functions or system calls,
139 such as
140 .BR open (2),
141 .BR read (2),
142 .BR write (2),
143 and
144 .BR close (2).
145 .P
146 The pseudo-files in a cpuset directory represent internal kernel
147 state and do not have any persistent image on disk.
148 Each of these per-cpuset files is listed and described below.
149 .\" ====================== tasks ======================
150 .TP
151 .I tasks
152 List of the process IDs (PIDs) of the processes in that cpuset.
153 The list is formatted as a series of ASCII
154 decimal numbers, each followed by a newline.
155 A process may be added to a cpuset (automatically removing
156 it from the cpuset that previously contained it) by writing its
157 PID to that cpuset's
158 .I tasks
159 file (with or without a trailing newline).
160 .IP
161 .B Warning:
162 only one PID may be written to the
163 .I tasks
164 file at a time.
165 If a string is written that contains more
166 than one PID, only the first one will be used.
167 .\" =================== notify_on_release ===================
168 .TP
169 .I notify_on_release
170 Flag (0 or 1).
171 If set (1), that cpuset will receive special handling
172 after it is released, that is, after all processes cease using
173 it (i.e., terminate or are moved to a different cpuset)
174 and all child cpuset directories have been removed.
175 See the \fBNotify On Release\fR section, below.
176 .\" ====================== cpus ======================
177 .TP
178 .I cpuset.cpus
179 List of the physical numbers of the CPUs on which processes
180 in that cpuset are allowed to execute.
181 See \fBList Format\fR below for a description of the
182 format of
183 .IR cpus .
184 .IP
185 The CPUs allowed to a cpuset may be changed by
186 writing a new list to its
187 .I cpus
188 file.
189 .\" ==================== cpu_exclusive ====================
190 .TP
191 .I cpuset.cpu_exclusive
192 Flag (0 or 1).
193 If set (1), the cpuset has exclusive use of
194 its CPUs (no sibling or cousin cpuset may overlap CPUs).
195 By default, this is off (0).
196 Newly created cpusets also initially default this to off (0).
197 .IP
198 Two cpusets are
199 .I sibling
200 cpusets if they share the same parent cpuset in the
201 .I /dev/cpuset
202 hierarchy.
203 Two cpusets are
204 .I cousin
205 cpusets if neither is the ancestor of the other.
206 Regardless of the
207 .I cpu_exclusive
208 setting, if one cpuset is the ancestor of another,
209 and if both of these cpusets have nonempty
210 .IR cpus ,
211 then their
212 .I cpus
213 must overlap, because the
214 .I cpus
215 of any cpuset are always a subset of the
216 .I cpus
217 of its parent cpuset.
218 .\" ====================== mems ======================
219 .TP
220 .I cpuset.mems
221 List of memory nodes on which processes in this cpuset are
222 allowed to allocate memory.
223 See \fBList Format\fR below for a description of the
224 format of
225 .IR mems .
226 .\" ==================== mem_exclusive ====================
227 .TP
228 .I cpuset.mem_exclusive
229 Flag (0 or 1).
230 If set (1), the cpuset has exclusive use of
231 its memory nodes (no sibling or cousin may overlap).
232 Also if set (1), the cpuset is a \fBHardwall\fR cpuset (see below).
233 By default, this is off (0).
234 Newly created cpusets also initially default this to off (0).
235 .IP
236 Regardless of the
237 .I mem_exclusive
238 setting, if one cpuset is the ancestor of another,
239 then their memory nodes must overlap, because the memory
240 nodes of any cpuset are always a subset of the memory nodes
241 of that cpuset's parent cpuset.
242 .\" ==================== mem_hardwall ====================
243 .TP
244 .IR cpuset.mem_hardwall " (since Linux 2.6.26)"
245 Flag (0 or 1).
246 If set (1), the cpuset is a \fBHardwall\fR cpuset (see below).
247 Unlike \fBmem_exclusive\fR,
248 there is no constraint on whether cpusets
249 marked \fBmem_hardwall\fR may have overlapping
250 memory nodes with sibling or cousin cpusets.
251 By default, this is off (0).
252 Newly created cpusets also initially default this to off (0).
253 .\" ==================== memory_migrate ====================
254 .TP
255 .IR cpuset.memory_migrate " (since Linux 2.6.16)"
256 Flag (0 or 1).
257 If set (1), then memory migration is enabled.
258 By default, this is off (0).
259 See the \fBMemory Migration\fR section, below.
260 .\" ==================== memory_pressure ====================
261 .TP
262 .IR cpuset.memory_pressure " (since Linux 2.6.16)"
263 A measure of how much memory pressure the processes in this
264 cpuset are causing.
265 See the \fBMemory Pressure\fR section, below.
266 Unless
267 .I memory_pressure_enabled
268 is enabled, always has value zero (0).
269 This file is read-only.
270 See the
271 .B WARNINGS
272 section, below.
273 .\" ================= memory_pressure_enabled =================
274 .TP
275 .IR cpuset.memory_pressure_enabled " (since Linux 2.6.16)"
276 Flag (0 or 1).
277 This file is present only in the root cpuset, normally
278 .IR /dev/cpuset .
279 If set (1), the
280 .I memory_pressure
281 calculations are enabled for all cpusets in the system.
282 By default, this is off (0).
283 See the
284 \fBMemory Pressure\fR section, below.
285 .\" ================== memory_spread_page ==================
286 .TP
287 .IR cpuset.memory_spread_page " (since Linux 2.6.17)"
288 Flag (0 or 1).
289 If set (1), pages in the kernel page cache
290 (filesystem buffers) are uniformly spread across the cpuset.
291 By default, this is off (0) in the top cpuset,
292 and inherited from the parent cpuset in
293 newly created cpusets.
294 See the \fBMemory Spread\fR section, below.
295 .\" ================== memory_spread_slab ==================
296 .TP
297 .IR cpuset.memory_spread_slab " (since Linux 2.6.17)"
298 Flag (0 or 1).
299 If set (1), the kernel slab caches
300 for file I/O (directory and inode structures) are
301 uniformly spread across the cpuset.
302 By default, is off (0) in the top cpuset,
303 and inherited from the parent cpuset in
304 newly created cpusets.
305 See the \fBMemory Spread\fR section, below.
306 .\" ================== sched_load_balance ==================
307 .TP
308 .IR cpuset.sched_load_balance " (since Linux 2.6.24)"
309 Flag (0 or 1).
310 If set (1, the default) the kernel will
311 automatically load balance processes in that cpuset over
312 the allowed CPUs in that cpuset.
313 If cleared (0) the
314 kernel will avoid load balancing processes in this cpuset,
315 .I unless
316 some other cpuset with overlapping CPUs has its
317 .I sched_load_balance
318 flag set.
319 See \fBScheduler Load Balancing\fR, below, for further details.
320 .\" ================== sched_relax_domain_level ==================
321 .TP
322 .IR cpuset.sched_relax_domain_level " (since Linux 2.6.26)"
323 Integer, between \-1 and a small positive value.
324 The
325 .I sched_relax_domain_level
326 controls the width of the range of CPUs over which the kernel scheduler
327 performs immediate rebalancing of runnable tasks across CPUs.
328 If
329 .I sched_load_balance
330 is disabled, then the setting of
331 .I sched_relax_domain_level
332 does not matter, as no such load balancing is done.
333 If
334 .I sched_load_balance
335 is enabled, then the higher the value of the
336 .IR sched_relax_domain_level ,
337 the wider
338 the range of CPUs over which immediate load balancing is attempted.
339 See \fBScheduler Relax Domain Level\fR, below, for further details.
340 .\" ================== proc cpuset ==================
341 .P
342 In addition to the above pseudo-files in each directory below
343 .IR /dev/cpuset ,
344 each process has a pseudo-file,
345 .IR /proc/ pid /cpuset ,
346 that displays the path of the process's cpuset directory
347 relative to the root of the cpuset filesystem.
348 .\" ================== proc status ==================
349 .P
350 Also the
351 .IR /proc/ pid /status
352 file for each process has four added lines,
353 displaying the process's
354 .I Cpus_allowed
355 (on which CPUs it may be scheduled) and
356 .I Mems_allowed
357 (on which memory nodes it may obtain memory),
358 in the two formats \fBMask Format\fR and \fBList Format\fR (see below)
359 as shown in the following example:
360 .P
361 .in +4n
362 .EX
363 Cpus_allowed: ffffffff,ffffffff,ffffffff,ffffffff
364 Cpus_allowed_list: 0\-127
365 Mems_allowed: ffffffff,ffffffff
366 Mems_allowed_list: 0\-63
367 .EE
368 .in
369 .P
370 The "allowed" fields were added in Linux 2.6.24;
371 the "allowed_list" fields were added in Linux 2.6.26.
372 .\" ================== EXTENDED CAPABILITIES ==================
373 .SH EXTENDED CAPABILITIES
374 In addition to controlling which
375 .I cpus
376 and
377 .I mems
378 a process is allowed to use, cpusets provide the following
379 extended capabilities.
380 .\" ================== Exclusive Cpusets ==================
381 .SS Exclusive cpusets
382 If a cpuset is marked
383 .I cpu_exclusive
384 or
385 .IR mem_exclusive ,
386 no other cpuset, other than a direct ancestor or descendant,
387 may share any of the same CPUs or memory nodes.
388 .P
389 A cpuset that is
390 .I mem_exclusive
391 restricts kernel allocations for
392 buffer cache pages and other internal kernel data pages
393 commonly shared by the kernel across
394 multiple users.
395 All cpusets, whether
396 .I mem_exclusive
397 or not, restrict allocations of memory for user space.
398 This enables configuring a
399 system so that several independent jobs can share common kernel data,
400 while isolating each job's user allocation in
401 its own cpuset.
402 To do this, construct a large
403 .I mem_exclusive
404 cpuset to hold all the jobs, and construct child,
405 .RI non- mem_exclusive
406 cpusets for each individual job.
407 Only a small amount of kernel memory,
408 such as requests from interrupt handlers, is allowed to be
409 placed on memory nodes
410 outside even a
411 .I mem_exclusive
412 cpuset.
413 .\" ================== Hardwall ==================
414 .SS Hardwall
415 A cpuset that has
416 .I mem_exclusive
417 or
418 .I mem_hardwall
419 set is a
420 .I hardwall
421 cpuset.
422 A
423 .I hardwall
424 cpuset restricts kernel allocations for page, buffer,
425 and other data commonly shared by the kernel across multiple users.
426 All cpusets, whether
427 .I hardwall
428 or not, restrict allocations of memory for user space.
429 .P
430 This enables configuring a system so that several independent
431 jobs can share common kernel data, such as filesystem pages,
432 while isolating each job's user allocation in its own cpuset.
433 To do this, construct a large
434 .I hardwall
435 cpuset to hold
436 all the jobs, and construct child cpusets for each individual
437 job which are not
438 .I hardwall
439 cpusets.
440 .P
441 Only a small amount of kernel memory, such as requests from
442 interrupt handlers, is allowed to be taken outside even a
443 .I hardwall
444 cpuset.
445 .\" ================== Notify On Release ==================
446 .SS Notify on release
447 If the
448 .I notify_on_release
449 flag is enabled (1) in a cpuset,
450 then whenever the last process in the cpuset leaves
451 (exits or attaches to some other cpuset)
452 and the last child cpuset of that cpuset is removed,
453 the kernel will run the command
454 .IR /sbin/cpuset_release_agent ,
455 supplying the pathname (relative to the mount point of the
456 cpuset filesystem) of the abandoned cpuset.
457 This enables automatic removal of abandoned cpusets.
458 .P
459 The default value of
460 .I notify_on_release
461 in the root cpuset at system boot is disabled (0).
462 The default value of other cpusets at creation
463 is the current value of their parent's
464 .I notify_on_release
465 setting.
466 .P
467 The command
468 .I /sbin/cpuset_release_agent
469 is invoked, with the name
470 .RI ( /dev/cpuset
471 relative path)
472 of the to-be-released cpuset in
473 .IR argv[1] .
474 .P
475 The usual contents of the command
476 .I /sbin/cpuset_release_agent
477 is simply the shell script:
478 .P
479 .in +4n
480 .EX
481 #!/bin/sh
482 rmdir /dev/cpuset/$1
483 .EE
484 .in
485 .P
486 As with other flag values below, this flag can
487 be changed by writing an ASCII
488 number 0 or 1 (with optional trailing newline)
489 into the file, to clear or set the flag, respectively.
490 .\" ================== Memory Pressure ==================
491 .SS Memory pressure
492 The
493 .I memory_pressure
494 of a cpuset provides a simple per-cpuset running average of
495 the rate that the processes in a cpuset are attempting to free up in-use
496 memory on the nodes of the cpuset to satisfy additional memory requests.
497 .P
498 This enables batch managers that are monitoring jobs running in dedicated
499 cpusets to efficiently detect what level of memory pressure that job
500 is causing.
501 .P
502 This is useful both on tightly managed systems running a wide mix of
503 submitted jobs, which may choose to terminate or reprioritize jobs that
504 are trying to use more memory than allowed on the nodes assigned them,
505 and with tightly coupled, long-running, massively parallel scientific
506 computing jobs that will dramatically fail to meet required performance
507 goals if they start to use more memory than allowed to them.
508 .P
509 This mechanism provides a very economical way for the batch manager
510 to monitor a cpuset for signs of memory pressure.
511 It's up to the batch manager or other user code to decide
512 what action to take if it detects signs of memory pressure.
513 .P
514 Unless memory pressure calculation is enabled by setting the pseudo-file
515 .IR /dev/cpuset/cpuset.memory_pressure_enabled ,
516 it is not computed for any cpuset, and reads from any
517 .I memory_pressure
518 always return zero, as represented by the ASCII string "0\[rs]n".
519 See the \fBWARNINGS\fR section, below.
520 .P
521 A per-cpuset, running average is employed for the following reasons:
522 .IP \[bu] 3
523 Because this meter is per-cpuset rather than per-process or per virtual
524 memory region, the system load imposed by a batch scheduler monitoring
525 this metric is sharply reduced on large systems, because a scan of
526 the tasklist can be avoided on each set of queries.
527 .IP \[bu]
528 Because this meter is a running average rather than an accumulating
529 counter, a batch scheduler can detect memory pressure with a
530 single read, instead of having to read and accumulate results
531 for a period of time.
532 .IP \[bu]
533 Because this meter is per-cpuset rather than per-process,
534 the batch scheduler can obtain the key information\[em]memory
535 pressure in a cpuset\[em]with a single read, rather than having to
536 query and accumulate results over all the (dynamically changing)
537 set of processes in the cpuset.
538 .P
539 The
540 .I memory_pressure
541 of a cpuset is calculated using a per-cpuset simple digital filter
542 that is kept within the kernel.
543 For each cpuset, this filter tracks
544 the recent rate at which processes attached to that cpuset enter the
545 kernel direct reclaim code.
546 .P
547 The kernel direct reclaim code is entered whenever a process has to
548 satisfy a memory page request by first finding some other page to
549 repurpose, due to lack of any readily available already free pages.
550 Dirty filesystem pages are repurposed by first writing them
551 to disk.
552 Unmodified filesystem buffer pages are repurposed
553 by simply dropping them, though if that page is needed again, it
554 will have to be reread from disk.
555 .P
556 The
557 .I cpuset.memory_pressure
558 file provides an integer number representing the recent (half-life of
559 10 seconds) rate of entries to the direct reclaim code caused by any
560 process in the cpuset, in units of reclaims attempted per second,
561 times 1000.
562 .\" ================== Memory Spread ==================
563 .SS Memory spread
564 There are two Boolean flag files per cpuset that control where the
565 kernel allocates pages for the filesystem buffers and related
566 in-kernel data structures.
567 They are called
568 .I cpuset.memory_spread_page
569 and
570 .IR cpuset.memory_spread_slab .
571 .P
572 If the per-cpuset Boolean flag file
573 .I cpuset.memory_spread_page
574 is set, then
575 the kernel will spread the filesystem buffers (page cache) evenly
576 over all the nodes that the faulting process is allowed to use, instead
577 of preferring to put those pages on the node where the process is running.
578 .P
579 If the per-cpuset Boolean flag file
580 .I cpuset.memory_spread_slab
581 is set,
582 then the kernel will spread some filesystem-related slab caches,
583 such as those for inodes and directory entries, evenly over all the nodes
584 that the faulting process is allowed to use, instead of preferring to
585 put those pages on the node where the process is running.
586 .P
587 The setting of these flags does not affect the data segment
588 (see
589 .BR brk (2))
590 or stack segment pages of a process.
591 .P
592 By default, both kinds of memory spreading are off and the kernel
593 prefers to allocate memory pages on the node local to where the
594 requesting process is running.
595 If that node is not allowed by the
596 process's NUMA memory policy or cpuset configuration or if there are
597 insufficient free memory pages on that node, then the kernel looks
598 for the nearest node that is allowed and has sufficient free memory.
599 .P
600 When new cpusets are created, they inherit the memory spread settings
601 of their parent.
602 .P
603 Setting memory spreading causes allocations for the affected page or
604 slab caches to ignore the process's NUMA memory policy and be spread
605 instead.
606 However, the effect of these changes in memory placement
607 caused by cpuset-specified memory spreading is hidden from the
608 .BR mbind (2)
609 or
610 .BR set_mempolicy (2)
611 calls.
612 These two NUMA memory policy calls always appear to behave as if
613 no cpuset-specified memory spreading is in effect, even if it is.
614 If cpuset memory spreading is subsequently turned off, the NUMA
615 memory policy most recently specified by these calls is automatically
616 reapplied.
617 .P
618 Both
619 .I cpuset.memory_spread_page
620 and
621 .I cpuset.memory_spread_slab
622 are Boolean flag files.
623 By default, they contain "0", meaning that the feature is off
624 for that cpuset.
625 If a "1" is written to that file, that turns the named feature on.
626 .P
627 Cpuset-specified memory spreading behaves similarly to what is known
628 (in other contexts) as round-robin or interleave memory placement.
629 .P
630 Cpuset-specified memory spreading can provide substantial performance
631 improvements for jobs that:
632 .IP \[bu] 3
633 need to place thread-local data on
634 memory nodes close to the CPUs which are running the threads that most
635 frequently access that data; but also
636 .IP \[bu]
637 need to access large filesystem data sets that must to be spread
638 across the several nodes in the job's cpuset in order to fit.
639 .P
640 Without this policy,
641 the memory allocation across the nodes in the job's cpuset
642 can become very uneven,
643 especially for jobs that might have just a single
644 thread initializing or reading in the data set.
645 .\" ================== Memory Migration ==================
646 .SS Memory migration
647 Normally, under the default setting (disabled) of
648 .IR cpuset.memory_migrate ,
649 once a page is allocated (given a physical page
650 of main memory), then that page stays on whatever node it
651 was allocated, so long as it remains allocated, even if the
652 cpuset's memory-placement policy
653 .I mems
654 subsequently changes.
655 .P
656 When memory migration is enabled in a cpuset, if the
657 .I mems
658 setting of the cpuset is changed, then any memory page in use by any
659 process in the cpuset that is on a memory node that is no longer
660 allowed will be migrated to a memory node that is allowed.
661 .P
662 Furthermore, if a process is moved into a cpuset with
663 .I memory_migrate
664 enabled, any memory pages it uses that were on memory nodes allowed
665 in its previous cpuset, but which are not allowed in its new cpuset,
666 will be migrated to a memory node allowed in the new cpuset.
667 .P
668 The relative placement of a migrated page within
669 the cpuset is preserved during these migration operations if possible.
670 For example,
671 if the page was on the second valid node of the prior cpuset,
672 then the page will be placed on the second valid node of the new cpuset,
673 if possible.
674 .\" ================== Scheduler Load Balancing ==================
675 .SS Scheduler load balancing
676 The kernel scheduler automatically load balances processes.
677 If one CPU is underutilized,
678 the kernel will look for processes on other more
679 overloaded CPUs and move those processes to the underutilized CPU,
680 within the constraints of such placement mechanisms as cpusets and
681 .BR sched_setaffinity (2).
682 .P
683 The algorithmic cost of load balancing and its impact on key shared
684 kernel data structures such as the process list increases more than
685 linearly with the number of CPUs being balanced.
686 For example, it
687 costs more to load balance across one large set of CPUs than it does
688 to balance across two smaller sets of CPUs, each of half the size
689 of the larger set.
690 (The precise relationship between the number of CPUs being balanced
691 and the cost of load balancing depends
692 on implementation details of the kernel process scheduler, which is
693 subject to change over time, as improved kernel scheduler algorithms
694 are implemented.)
695 .P
696 The per-cpuset flag
697 .I sched_load_balance
698 provides a mechanism to suppress this automatic scheduler load
699 balancing in cases where it is not needed and suppressing it would have
700 worthwhile performance benefits.
701 .P
702 By default, load balancing is done across all CPUs, except those
703 marked isolated using the kernel boot time "isolcpus=" argument.
704 (See \fBScheduler Relax Domain Level\fR, below, to change this default.)
705 .P
706 This default load balancing across all CPUs is not well suited to
707 the following two situations:
708 .IP \[bu] 3
709 On large systems, load balancing across many CPUs is expensive.
710 If the system is managed using cpusets to place independent jobs
711 on separate sets of CPUs, full load balancing is unnecessary.
712 .IP \[bu]
713 Systems supporting real-time on some CPUs need to minimize
714 system overhead on those CPUs, including avoiding process load
715 balancing if that is not needed.
716 .P
717 When the per-cpuset flag
718 .I sched_load_balance
719 is enabled (the default setting),
720 it requests load balancing across
721 all the CPUs in that cpuset's allowed CPUs,
722 ensuring that load balancing can move a process (not otherwise pinned,
723 as by
724 .BR sched_setaffinity (2))
725 from any CPU in that cpuset to any other.
726 .P
727 When the per-cpuset flag
728 .I sched_load_balance
729 is disabled, then the
730 scheduler will avoid load balancing across the CPUs in that cpuset,
731 \fIexcept\fR in so far as is necessary because some overlapping cpuset
732 has
733 .I sched_load_balance
734 enabled.
735 .P
736 So, for example, if the top cpuset has the flag
737 .I sched_load_balance
738 enabled, then the scheduler will load balance across all
739 CPUs, and the setting of the
740 .I sched_load_balance
741 flag in other cpusets has no effect,
742 as we're already fully load balancing.
743 .P
744 Therefore in the above two situations, the flag
745 .I sched_load_balance
746 should be disabled in the top cpuset, and only some of the smaller,
747 child cpusets would have this flag enabled.
748 .P
749 When doing this, you don't usually want to leave any unpinned processes in
750 the top cpuset that might use nontrivial amounts of CPU, as such processes
751 may be artificially constrained to some subset of CPUs, depending on
752 the particulars of this flag setting in descendant cpusets.
753 Even if such a process could use spare CPU cycles in some other CPUs,
754 the kernel scheduler might not consider the possibility of
755 load balancing that process to the underused CPU.
756 .P
757 Of course, processes pinned to a particular CPU can be left in a cpuset
758 that disables
759 .I sched_load_balance
760 as those processes aren't going anywhere else anyway.
761 .\" ================== Scheduler Relax Domain Level ==================
762 .SS Scheduler relax domain level
763 The kernel scheduler performs immediate load balancing whenever
764 a CPU becomes free or another task becomes runnable.
765 This load
766 balancing works to ensure that as many CPUs as possible are usefully
767 employed running tasks.
768 The kernel also performs periodic load
769 balancing off the software clock described in
770 .BR time (7).
771 The setting of
772 .I sched_relax_domain_level
773 applies only to immediate load balancing.
774 Regardless of the
775 .I sched_relax_domain_level
776 setting, periodic load balancing is attempted over all CPUs
777 (unless disabled by turning off
778 .IR sched_load_balance .)
779 In any case, of course, tasks will be scheduled to run only on
780 CPUs allowed by their cpuset, as modified by
781 .BR sched_setaffinity (2)
782 system calls.
783 .P
784 On small systems, such as those with just a few CPUs, immediate load
785 balancing is useful to improve system interactivity and to minimize
786 wasteful idle CPU cycles.
787 But on large systems, attempting immediate
788 load balancing across a large number of CPUs can be more costly than
789 it is worth, depending on the particular performance characteristics
790 of the job mix and the hardware.
791 .P
792 The exact meaning of the small integer values of
793 .I sched_relax_domain_level
794 will depend on internal
795 implementation details of the kernel scheduler code and on the
796 non-uniform architecture of the hardware.
797 Both of these will evolve
798 over time and vary by system architecture and kernel version.
799 .P
800 As of this writing, when this capability was introduced in Linux
801 2.6.26, on certain popular architectures, the positive values of
802 .I sched_relax_domain_level
803 have the following meanings.
804 .P
805 .PD 0
806 .TP
807 .B 1
808 Perform immediate load balancing across Hyper-Thread
809 siblings on the same core.
810 .TP
811 .B 2
812 Perform immediate load balancing across other cores in the same package.
813 .TP
814 .B 3
815 Perform immediate load balancing across other CPUs
816 on the same node or blade.
817 .TP
818 .B 4
819 Perform immediate load balancing across over several
820 (implementation detail) nodes [On NUMA systems].
821 .TP
822 .B 5
823 Perform immediate load balancing across over all CPUs
824 in system [On NUMA systems].
825 .PD
826 .P
827 The
828 .I sched_relax_domain_level
829 value of zero (0) always means
830 don't perform immediate load balancing,
831 hence that load balancing is done only periodically,
832 not immediately when a CPU becomes available or another task becomes
833 runnable.
834 .P
835 The
836 .I sched_relax_domain_level
837 value of minus one (\-1)
838 always means use the system default value.
839 The system default value can vary by architecture and kernel version.
840 This system default value can be changed by kernel
841 boot-time "relax_domain_level=" argument.
842 .P
843 In the case of multiple overlapping cpusets which have conflicting
844 .I sched_relax_domain_level
845 values, then the highest such value
846 applies to all CPUs in any of the overlapping cpusets.
847 In such cases,
848 .B \-1
849 is the lowest value,
850 overridden by any other value,
851 and
852 .B 0
853 is the next lowest value.
854 .SH FORMATS
855 The following formats are used to represent sets of
856 CPUs and memory nodes.
857 .\" ================== Mask Format ==================
858 .SS Mask format
859 The \fBMask Format\fR is used to represent CPU and memory-node bit masks
860 in the
861 .IR /proc/ pid /status
862 file.
863 .P
864 This format displays each 32-bit
865 word in hexadecimal (using ASCII characters "0" - "9" and "a" - "f");
866 words are filled with leading zeros, if required.
867 For masks longer than one word, a comma separator is used between words.
868 Words are displayed in big-endian
869 order, which has the most significant bit first.
870 The hex digits within a word are also in big-endian order.
871 .P
872 The number of 32-bit words displayed is the minimum number needed to
873 display all bits of the bit mask, based on the size of the bit mask.
874 .P
875 Examples of the \fBMask Format\fR:
876 .P
877 .in +4n
878 .EX
879 00000001 # just bit 0 set
880 40000000,00000000,00000000 # just bit 94 set
881 00000001,00000000,00000000 # just bit 64 set
882 000000ff,00000000 # bits 32\-39 set
883 00000000,000e3862 # 1,5,6,11\-13,17\-19 set
884 .EE
885 .in
886 .P
887 A mask with bits 0, 1, 2, 4, 8, 16, 32, and 64 set displays as:
888 .P
889 .in +4n
890 .EX
891 00000001,00000001,00010117
892 .EE
893 .in
894 .P
895 The first "1" is for bit 64, the
896 second for bit 32, the third for bit 16, the fourth for bit 8, the
897 fifth for bit 4, and the "7" is for bits 2, 1, and 0.
898 .\" ================== List Format ==================
899 .SS List format
900 The \fBList Format\fR for
901 .I cpus
902 and
903 .I mems
904 is a comma-separated list of CPU or memory-node
905 numbers and ranges of numbers, in ASCII decimal.
906 .P
907 Examples of the \fBList Format\fR:
908 .P
909 .in +4n
910 .EX
911 0\-4,9 # bits 0, 1, 2, 3, 4, and 9 set
912 0\-2,7,12\-14 # bits 0, 1, 2, 7, 12, 13, and 14 set
913 .EE
914 .in
915 .\" ================== RULES ==================
916 .SH RULES
917 The following rules apply to each cpuset:
918 .IP \[bu] 3
919 Its CPUs and memory nodes must be a (possibly equal)
920 subset of its parent's.
921 .IP \[bu]
922 It can be marked
923 .I cpu_exclusive
924 only if its parent is.
925 .IP \[bu]
926 It can be marked
927 .I mem_exclusive
928 only if its parent is.
929 .IP \[bu]
930 If it is
931 .IR cpu_exclusive ,
932 its CPUs may not overlap any sibling.
933 .IP \[bu]
934 If it is
935 .IR mem_exclusive ,
936 its memory nodes may not overlap any sibling.
937 .\" ================== PERMISSIONS ==================
938 .SH PERMISSIONS
939 The permissions of a cpuset are determined by the permissions
940 of the directories and pseudo-files in the cpuset filesystem,
941 normally mounted at
942 .IR /dev/cpuset .
943 .P
944 For instance, a process can put itself in some other cpuset (than
945 its current one) if it can write the
946 .I tasks
947 file for that cpuset.
948 This requires execute permission on the encompassing directories
949 and write permission on the
950 .I tasks
951 file.
952 .P
953 An additional constraint is applied to requests to place some
954 other process in a cpuset.
955 One process may not attach another to
956 a cpuset unless it would have permission to send that process
957 a signal (see
958 .BR kill (2)).
959 .P
960 A process may create a child cpuset if it can access and write the
961 parent cpuset directory.
962 It can modify the CPUs or memory nodes
963 in a cpuset if it can access that cpuset's directory (execute
964 permissions on the each of the parent directories) and write the
965 corresponding
966 .I cpus
967 or
968 .I mems
969 file.
970 .P
971 There is one minor difference between the manner in which these
972 permissions are evaluated and the manner in which normal filesystem
973 operation permissions are evaluated.
974 The kernel interprets
975 relative pathnames starting at a process's current working directory.
976 Even if one is operating on a cpuset file, relative pathnames
977 are interpreted relative to the process's current working directory,
978 not relative to the process's current cpuset.
979 The only ways that
980 cpuset paths relative to a process's current cpuset can be used are
981 if either the process's current working directory is its cpuset
982 (it first did a
983 .B cd
984 or
985 .BR chdir (2)
986 to its cpuset directory beneath
987 .IR /dev/cpuset ,
988 which is a bit unusual)
989 or if some user code converts the relative cpuset path to a
990 full filesystem path.
991 .P
992 In theory, this means that user code should specify cpusets
993 using absolute pathnames, which requires knowing the mount point of
994 the cpuset filesystem (usually, but not necessarily,
995 .IR /dev/cpuset ).
996 In practice, all user level code that this author is aware of
997 simply assumes that if the cpuset filesystem is mounted, then
998 it is mounted at
999 .IR /dev/cpuset .
1000 Furthermore, it is common practice for carefully written
1001 user code to verify the presence of the pseudo-file
1002 .I /dev/cpuset/tasks
1003 in order to verify that the cpuset pseudo-filesystem
1004 is currently mounted.
1005 .\" ================== WARNINGS ==================
1006 .SH WARNINGS
1007 .SS Enabling memory_pressure
1008 By default, the per-cpuset file
1009 .I cpuset.memory_pressure
1010 always contains zero (0).
1011 Unless this feature is enabled by writing "1" to the pseudo-file
1012 .IR /dev/cpuset/cpuset.memory_pressure_enabled ,
1013 the kernel does
1014 not compute per-cpuset
1015 .IR memory_pressure .
1016 .SS Using the echo command
1017 When using the
1018 .B echo
1019 command at the shell prompt to change the values of cpuset files,
1020 beware that the built-in
1021 .B echo
1022 command in some shells does not display an error message if the
1023 .BR write (2)
1024 system call fails.
1025 .\" Gack! csh(1)'s echo does this
1026 For example, if the command:
1027 .P
1028 .in +4n
1029 .EX
1030 echo 19 > cpuset.mems
1031 .EE
1032 .in
1033 .P
1034 failed because memory node 19 was not allowed (perhaps
1035 the current system does not have a memory node 19), then the
1036 .B echo
1037 command might not display any error.
1038 It is better to use the
1039 .B /bin/echo
1040 external command to change cpuset file settings, as this
1041 command will display
1042 .BR write (2)
1043 errors, as in the example:
1044 .P
1045 .in +4n
1046 .EX
1047 /bin/echo 19 > cpuset.mems
1048 /bin/echo: write error: Invalid argument
1049 .EE
1050 .in
1051 .\" ================== EXCEPTIONS ==================
1052 .SH EXCEPTIONS
1053 .SS Memory placement
1054 Not all allocations of system memory are constrained by cpusets,
1055 for the following reasons.
1056 .P
1057 If hot-plug functionality is used to remove all the CPUs that are
1058 currently assigned to a cpuset, then the kernel will automatically
1059 update the
1060 .I cpus_allowed
1061 of all processes attached to CPUs in that cpuset
1062 to allow all CPUs.
1063 When memory hot-plug functionality for removing
1064 memory nodes is available, a similar exception is expected to apply
1065 there as well.
1066 In general, the kernel prefers to violate cpuset placement,
1067 rather than starving a process that has had all its allowed CPUs or
1068 memory nodes taken offline.
1069 User code should reconfigure cpusets to refer only to online CPUs
1070 and memory nodes when using hot-plug to add or remove such resources.
1071 .P
1072 A few kernel-critical, internal memory-allocation requests, marked
1073 GFP_ATOMIC, must be satisfied immediately.
1074 The kernel may drop some
1075 request or malfunction if one of these allocations fail.
1076 If such a request cannot be satisfied within the current process's cpuset,
1077 then we relax the cpuset, and look for memory anywhere we can find it.
1078 It's better to violate the cpuset than stress the kernel.
1079 .P
1080 Allocations of memory requested by kernel drivers while processing
1081 an interrupt lack any relevant process context, and are not confined
1082 by cpusets.
1083 .SS Renaming cpusets
1084 You can use the
1085 .BR rename (2)
1086 system call to rename cpusets.
1087 Only simple renaming is supported; that is, changing the name of a cpuset
1088 directory is permitted, but moving a directory into
1089 a different directory is not permitted.
1090 .\" ================== ERRORS ==================
1091 .SH ERRORS
1092 The Linux kernel implementation of cpusets sets
1093 .I errno
1094 to specify the reason for a failed system call affecting cpusets.
1095 .P
1096 The possible
1097 .I errno
1098 settings and their meaning when set on
1099 a failed cpuset call are as listed below.
1100 .TP
1101 .B E2BIG
1102 Attempted a
1103 .BR write (2)
1104 on a special cpuset file
1105 with a length larger than some kernel-determined upper
1106 limit on the length of such writes.
1107 .TP
1108 .B EACCES
1109 Attempted to
1110 .BR write (2)
1111 the process ID (PID) of a process to a cpuset
1112 .I tasks
1113 file when one lacks permission to move that process.
1114 .TP
1115 .B EACCES
1116 Attempted to add, using
1117 .BR write (2),
1118 a CPU or memory node to a cpuset, when that CPU or memory node was
1119 not already in its parent.
1120 .TP
1121 .B EACCES
1122 Attempted to set, using
1123 .BR write (2),
1124 .I cpuset.cpu_exclusive
1125 or
1126 .I cpuset.mem_exclusive
1127 on a cpuset whose parent lacks the same setting.
1128 .TP
1129 .B EACCES
1130 Attempted to
1131 .BR write (2)
1132 a
1133 .I cpuset.memory_pressure
1134 file.
1135 .TP
1136 .B EACCES
1137 Attempted to create a file in a cpuset directory.
1138 .TP
1139 .B EBUSY
1140 Attempted to remove, using
1141 .BR rmdir (2),
1142 a cpuset with attached processes.
1143 .TP
1144 .B EBUSY
1145 Attempted to remove, using
1146 .BR rmdir (2),
1147 a cpuset with child cpusets.
1148 .TP
1149 .B EBUSY
1150 Attempted to remove
1151 a CPU or memory node from a cpuset
1152 that is also in a child of that cpuset.
1153 .TP
1154 .B EEXIST
1155 Attempted to create, using
1156 .BR mkdir (2),
1157 a cpuset that already exists.
1158 .TP
1159 .B EEXIST
1160 Attempted to
1161 .BR rename (2)
1162 a cpuset to a name that already exists.
1163 .TP
1164 .B EFAULT
1165 Attempted to
1166 .BR read (2)
1167 or
1168 .BR write (2)
1169 a cpuset file using
1170 a buffer that is outside the writing processes accessible address space.
1171 .TP
1172 .B EINVAL
1173 Attempted to change a cpuset, using
1174 .BR write (2),
1175 in a way that would violate a
1176 .I cpu_exclusive
1177 or
1178 .I mem_exclusive
1179 attribute of that cpuset or any of its siblings.
1180 .TP
1181 .B EINVAL
1182 Attempted to
1183 .BR write (2)
1184 an empty
1185 .I cpuset.cpus
1186 or
1187 .I cpuset.mems
1188 list to a cpuset which has attached processes or child cpusets.
1189 .TP
1190 .B EINVAL
1191 Attempted to
1192 .BR write (2)
1193 a
1194 .I cpuset.cpus
1195 or
1196 .I cpuset.mems
1197 list which included a range with the second number smaller than
1198 the first number.
1199 .TP
1200 .B EINVAL
1201 Attempted to
1202 .BR write (2)
1203 a
1204 .I cpuset.cpus
1205 or
1206 .I cpuset.mems
1207 list which included an invalid character in the string.
1208 .TP
1209 .B EINVAL
1210 Attempted to
1211 .BR write (2)
1212 a list to a
1213 .I cpuset.cpus
1214 file that did not include any online CPUs.
1215 .TP
1216 .B EINVAL
1217 Attempted to
1218 .BR write (2)
1219 a list to a
1220 .I cpuset.mems
1221 file that did not include any online memory nodes.
1222 .TP
1223 .B EINVAL
1224 Attempted to
1225 .BR write (2)
1226 a list to a
1227 .I cpuset.mems
1228 file that included a node that held no memory.
1229 .TP
1230 .B EIO
1231 Attempted to
1232 .BR write (2)
1233 a string to a cpuset
1234 .I tasks
1235 file that
1236 does not begin with an ASCII decimal integer.
1237 .TP
1238 .B EIO
1239 Attempted to
1240 .BR rename (2)
1241 a cpuset into a different directory.
1242 .TP
1243 .B ENAMETOOLONG
1244 Attempted to
1245 .BR read (2)
1246 a
1247 .IR /proc/ pid /cpuset
1248 file for a cpuset path that is longer than the kernel page size.
1249 .TP
1250 .B ENAMETOOLONG
1251 Attempted to create, using
1252 .BR mkdir (2),
1253 a cpuset whose base directory name is longer than 255 characters.
1254 .TP
1255 .B ENAMETOOLONG
1256 Attempted to create, using
1257 .BR mkdir (2),
1258 a cpuset whose full pathname,
1259 including the mount point (typically "/dev/cpuset/") prefix,
1260 is longer than 4095 characters.
1261 .TP
1262 .B ENODEV
1263 The cpuset was removed by another process at the same time as a
1264 .BR write (2)
1265 was attempted on one of the pseudo-files in the cpuset directory.
1266 .TP
1267 .B ENOENT
1268 Attempted to create, using
1269 .BR mkdir (2),
1270 a cpuset in a parent cpuset that doesn't exist.
1271 .TP
1272 .B ENOENT
1273 Attempted to
1274 .BR access (2)
1275 or
1276 .BR open (2)
1277 a nonexistent file in a cpuset directory.
1278 .TP
1279 .B ENOMEM
1280 Insufficient memory is available within the kernel; can occur
1281 on a variety of system calls affecting cpusets, but only if the
1282 system is extremely short of memory.
1283 .TP
1284 .B ENOSPC
1285 Attempted to
1286 .BR write (2)
1287 the process ID (PID)
1288 of a process to a cpuset
1289 .I tasks
1290 file when the cpuset had an empty
1291 .I cpuset.cpus
1292 or empty
1293 .I cpuset.mems
1294 setting.
1295 .TP
1296 .B ENOSPC
1297 Attempted to
1298 .BR write (2)
1299 an empty
1300 .I cpuset.cpus
1301 or
1302 .I cpuset.mems
1303 setting to a cpuset that
1304 has tasks attached.
1305 .TP
1306 .B ENOTDIR
1307 Attempted to
1308 .BR rename (2)
1309 a nonexistent cpuset.
1310 .TP
1311 .B EPERM
1312 Attempted to remove a file from a cpuset directory.
1313 .TP
1314 .B ERANGE
1315 Specified a
1316 .I cpuset.cpus
1317 or
1318 .I cpuset.mems
1319 list to the kernel which included a number too large for the kernel
1320 to set in its bit masks.
1321 .TP
1322 .B ESRCH
1323 Attempted to
1324 .BR write (2)
1325 the process ID (PID) of a nonexistent process to a cpuset
1326 .I tasks
1327 file.
1328 .\" ================== VERSIONS ==================
1329 .SH VERSIONS
1330 Cpusets appeared in Linux 2.6.12.
1331 .\" ================== NOTES ==================
1332 .SH NOTES
1333 Despite its name, the
1334 .I pid
1335 parameter is actually a thread ID,
1336 and each thread in a threaded group can be attached to a different
1337 cpuset.
1338 The value returned from a call to
1339 .BR gettid (2)
1340 can be passed in the argument
1341 .IR pid .
1342 .\" ================== BUGS ==================
1343 .SH BUGS
1344 .I cpuset.memory_pressure
1345 cpuset files can be opened
1346 for writing, creation, or truncation, but then the
1347 .BR write (2)
1348 fails with
1349 .I errno
1350 set to
1351 .BR EACCES ,
1352 and the creation and truncation options on
1353 .BR open (2)
1354 have no effect.
1355 .\" ================== EXAMPLES ==================
1356 .SH EXAMPLES
1357 The following examples demonstrate querying and setting cpuset
1358 options using shell commands.
1359 .SS Creating and attaching to a cpuset.
1360 To create a new cpuset and attach the current command shell to it,
1361 the steps are:
1362 .P
1363 .PD 0
1364 .IP (1) 5
1365 mkdir /dev/cpuset (if not already done)
1366 .IP (2)
1367 mount \-t cpuset none /dev/cpuset (if not already done)
1368 .IP (3)
1369 Create the new cpuset using
1370 .BR mkdir (1).
1371 .IP (4)
1372 Assign CPUs and memory nodes to the new cpuset.
1373 .IP (5)
1374 Attach the shell to the new cpuset.
1375 .PD
1376 .P
1377 For example, the following sequence of commands will set up a cpuset
1378 named "Charlie", containing just CPUs 2 and 3, and memory node 1,
1379 and then attach the current shell to that cpuset.
1380 .P
1381 .in +4n
1382 .EX
1383 .RB "$" " mkdir /dev/cpuset"
1384 .RB "$" " mount \-t cpuset cpuset /dev/cpuset"
1385 .RB "$" " cd /dev/cpuset"
1386 .RB "$" " mkdir Charlie"
1387 .RB "$" " cd Charlie"
1388 .RB "$" " /bin/echo 2\-3 > cpuset.cpus"
1389 .RB "$" " /bin/echo 1 > cpuset.mems"
1390 .RB "$" " /bin/echo $$ > tasks"
1391 # The current shell is now running in cpuset Charlie
1392 # The next line should display \[aq]/Charlie\[aq]
1393 .RB "$" " cat /proc/self/cpuset"
1394 .EE
1395 .in
1396 .\"
1397 .SS Migrating a job to different memory nodes.
1398 To migrate a job (the set of processes attached to a cpuset)
1399 to different CPUs and memory nodes in the system, including moving
1400 the memory pages currently allocated to that job,
1401 perform the following steps.
1402 .P
1403 .PD 0
1404 .IP (1) 5
1405 Let's say we want to move the job in cpuset
1406 .I alpha
1407 (CPUs 4\[en]7 and memory nodes 2\[en]3) to a new cpuset
1408 .I beta
1409 (CPUs 16\[en]19 and memory nodes 8\[en]9).
1410 .IP (2)
1411 First create the new cpuset
1412 .IR beta .
1413 .IP (3)
1414 Then allow CPUs 16\[en]19 and memory nodes 8\[en]9 in
1415 .IR beta .
1416 .IP (4)
1417 Then enable
1418 .I memory_migration
1419 in
1420 .IR beta .
1421 .IP (5)
1422 Then move each process from
1423 .I alpha
1424 to
1425 .IR beta .
1426 .PD
1427 .P
1428 The following sequence of commands accomplishes this.
1429 .P
1430 .in +4n
1431 .EX
1432 .RB "$" " cd /dev/cpuset"
1433 .RB "$" " mkdir beta"
1434 .RB "$" " cd beta"
1435 .RB "$" " /bin/echo 16\-19 > cpuset.cpus"
1436 .RB "$" " /bin/echo 8\-9 > cpuset.mems"
1437 .RB "$" " /bin/echo 1 > cpuset.memory_migrate"
1438 .RB "$" " while read i; do /bin/echo $i; done < ../alpha/tasks > tasks"
1439 .EE
1440 .in
1441 .P
1442 The above should move any processes in
1443 .I alpha
1444 to
1445 .IR beta ,
1446 and any memory held by these processes on memory nodes 2\[en]3 to memory
1447 nodes 8\[en]9, respectively.
1448 .P
1449 Notice that the last step of the above sequence did not do:
1450 .P
1451 .in +4n
1452 .EX
1453 .RB "$" " cp ../alpha/tasks tasks"
1454 .EE
1455 .in
1456 .P
1457 The
1458 .I while
1459 loop, rather than the seemingly easier use of the
1460 .BR cp (1)
1461 command, was necessary because
1462 only one process PID at a time may be written to the
1463 .I tasks
1464 file.
1465 .P
1466 The same effect (writing one PID at a time) as the
1467 .I while
1468 loop can be accomplished more efficiently, in fewer keystrokes and in
1469 syntax that works on any shell, but alas more obscurely, by using the
1470 .B \-u
1471 (unbuffered) option of
1472 .BR sed (1):
1473 .P
1474 .in +4n
1475 .EX
1476 .RB "$" " sed \-un p < ../alpha/tasks > tasks"
1477 .EE
1478 .in
1479 .\" ================== SEE ALSO ==================
1480 .SH SEE ALSO
1481 .BR taskset (1),
1482 .BR get_mempolicy (2),
1483 .BR getcpu (2),
1484 .BR mbind (2),
1485 .BR sched_getaffinity (2),
1486 .BR sched_setaffinity (2),
1487 .BR sched_setscheduler (2),
1488 .BR set_mempolicy (2),
1489 .BR CPU_SET (3),
1490 .BR proc (5),
1491 .BR cgroups (7),
1492 .BR numa (7),
1493 .BR sched (7),
1494 .BR migratepages (8),
1495 .BR numactl (8)
1496 .P
1497 .I Documentation/admin\-guide/cgroup\-v1/cpusets.rst
1498 in the Linux kernel source tree
1499 .\" commit 45ce80fb6b6f9594d1396d44dd7e7c02d596fef8
1500 (or
1501 .I Documentation/cgroup\-v1/cpusets.txt
1502 before Linux 4.18, and
1503 .I Documentation/cpusets.txt
1504 before Linux 2.6.29)