Debian Codegraph @

upstream/debian-unstable/man7/cpuset.7

raw
1 .\" Copyright 2008, Silicon Graphics, Inc.
2 .\" Copyright, the authors of the Linux man-pages project
3 .\"
4 .\" SPDX-License-Identifier: GPL-2.0-only
5 .\"
6 .TH cpuset 7 2026-02-08 "Linux man-pages 6.18"
7 .SH NAME
8 cpuset \- confine processes to processor and memory node subsets
9 .SH DESCRIPTION
10 The cpuset filesystem is a pseudo-filesystem interface
11 to the kernel cpuset mechanism,
12 which is used to control the processor placement
13 and memory placement of processes.
14 It is commonly mounted at
15 .IR /dev/cpuset .
16 .P
17 On systems with kernels compiled with built in support for cpusets,
18 all processes are attached to a cpuset, and cpusets are always present.
19 If a system supports cpusets, then it will have the entry
20 .B nodev cpuset
21 in the file
22 .IR /proc/filesystems .
23 By mounting the cpuset filesystem (see the
24 .B EXAMPLES
25 section below),
26 the administrator can configure the cpusets on a system
27 to control the processor and memory placement of processes
28 on that system.
29 By default, if the cpuset configuration
30 on a system is not modified or if the cpuset filesystem
31 is not even mounted, then the cpuset mechanism,
32 though present, has no effect on the system's behavior.
33 .P
34 A cpuset defines a list of CPUs and memory nodes.
35 .P
36 The CPUs of a system include
37 all the logical processing units on which a process can execute,
38 including,
39 if present,
40 multiple processor cores within a package
41 and Hyper-Threads within a processor core.
42 Memory nodes include all distinct banks of main memory;
43 small and SMP systems typically have
44 just one memory node that contains all the system's main memory,
45 while NUMA (non-uniform memory access) systems have multiple memory nodes.
46 .P
47 Cpusets are represented as directories in a hierarchical pseudo-filesystem,
48 where the top directory in the hierarchy
49 .RI ( /dev/cpuset )
50 represents the entire system (all online CPUs and memory nodes)
51 and any cpuset that is the child (descendant) of another parent cpuset
52 contains a subset of that parent's CPUs and memory nodes.
53 The directories and files representing cpusets
54 have normal filesystem permissions.
55 .P
56 Every process in the system belongs to exactly one cpuset.
57 A process is confined to run
58 only on the CPUs in the cpuset it belongs to,
59 and to allocate memory
60 only on the memory nodes in that cpuset.
61 When a process
62 .BR fork (2)s,
63 the child process is placed in the same cpuset as its parent.
64 With sufficient privilege,
65 a process may be moved from one cpuset to another
66 and the allowed CPUs and memory nodes of an existing cpuset
67 may be changed.
68 .P
69 When the system begins booting,
70 a single cpuset is defined
71 that includes all CPUs and memory nodes on the system,
72 and all processes are in that cpuset.
73 During the boot process,
74 or later during normal system operation,
75 other cpusets may be created,
76 as subdirectories of this top cpuset,
77 under the control of the system administrator,
78 and processes may be placed in these other cpusets.
79 .P
80 Cpusets are integrated with the
81 .BR sched_setaffinity (2)
82 scheduling affinity mechanism and the
83 .BR mbind (2)
84 and
85 .BR set_mempolicy (2)
86 memory-placement mechanisms in the kernel.
87 Neither of these mechanisms let a process
88 make use of a CPU or memory node that is not allowed by that process's cpuset.
89 If changes to a process's cpuset placement
90 conflict with these other mechanisms,
91 then cpuset placement is enforced
92 even if it means overriding these other mechanisms.
93 The kernel accomplishes this overriding
94 by silently restricting the CPUs and memory nodes
95 requested by these other mechanisms
96 to those allowed by the invoking process's cpuset.
97 This can result in these other calls returning an error,
98 if for example,
99 such a call ends up requesting an empty set of CPUs or memory nodes,
100 after that request is restricted to the invoking process's cpuset.
101 .P
102 Typically,
103 a cpuset is used to manage
104 the CPU and memory-node confinement for a set of cooperating processes,
105 such as a batch scheduler job,
106 and these other mechanisms are used to manage
107 the placement of individual processes or memory regions
108 within that set or job.
109 .SH FILES
110 Each directory below
111 .I /dev/cpuset
112 represents a cpuset and contains a fixed set of pseudo-files
113 describing the state of that cpuset.
114 .P
115 New cpusets are created using the
116 .BR mkdir (2)
117 system call or the
118 .BR mkdir (1)
119 command.
120 The properties of a cpuset,
121 such as its flags,
122 allowed CPUs and memory nodes,
123 and attached processes,
124 are queried and modified
125 by reading or writing to the appropriate file in that cpuset's directory,
126 as listed below.
127 .P
128 The pseudo-files in each cpuset directory
129 are automatically created when the cpuset is created,
130 as a result of the
131 .BR mkdir (2)
132 invocation.
133 It is not possible to directly add or remove these pseudo-files.
134 .P
135 A cpuset directory that contains no child cpuset directories,
136 and has no attached processes,
137 can be removed using
138 .BR rmdir (2)
139 or
140 .BR rmdir (1).
141 It is not necessary, or possible,
142 to remove the pseudo-files inside the directory before removing it.
143 .P
144 The pseudo-files in each cpuset directory
145 are small text files that may be
146 read and written using traditional shell utilities such as
147 .BR cat (1),
148 and
149 .BR echo (1),
150 or from a program by using file I/O library functions or system calls,
151 such as
152 .BR open (2),
153 .BR read (2),
154 .BR write (2),
155 and
156 .BR close (2).
157 .P
158 The pseudo-files in a cpuset directory represent internal kernel state
159 and do not have any persistent image on disk.
160 Each of these per-cpuset files is listed and described below.
161 .\" ====================== tasks ======================
162 .TP
163 .I tasks
164 List of the process IDs (PIDs) of the processes in that cpuset.
165 The list is formatted as a series of
166 ASCII decimal numbers,
167 each followed by a newline.
168 A process may be added to a cpuset
169 (automatically removing it from the cpuset that previously contained it)
170 by writing its PID to that cpuset's
171 .I tasks
172 file
173 (with or without a trailing newline).
174 .IP
175 .B Warning:
176 only one PID may be written to the
177 .I tasks
178 file at a time.
179 If a string is written that contains more than one PID,
180 only the first one will be used.
181 .\" =================== notify_on_release ===================
182 .TP
183 .I notify_on_release
184 Flag (0 or 1).
185 If set (1),
186 that cpuset will receive special handling after it is released,
187 that is,
188 after all processes cease using it
189 (i.e., terminate or are moved to a different cpuset)
190 and all child cpuset directories have been removed.
191 See the
192 .B Notify On Release \" Sx
193 section, below.
194 .\" ====================== cpus ======================
195 .TP
196 .I cpuset.cpus
197 List of the physical numbers of the CPUs
198 on which processes in that cpuset are allowed to execute.
199 See
200 .B List Format \" Sx
201 below for a description of the format of
202 .IR cpus .
203 .IP
204 The CPUs allowed to a cpuset may be changed by
205 writing a new list to its
206 .I cpus
207 file.
208 .\" ==================== cpu_exclusive ====================
209 .TP
210 .I cpuset.cpu_exclusive
211 Flag (0 or 1).
212 If set (1),
213 the cpuset has exclusive use of its CPUs
214 (no sibling or cousin cpuset may overlap CPUs).
215 By default,
216 this is off (0).
217 Newly created cpusets also initially default this to off (0).
218 .IP
219 Two cpusets are
220 .I sibling
221 cpusets if they share the same parent cpuset in the
222 .I /dev/cpuset
223 hierarchy.
224 Two cpusets are
225 .I cousin
226 cpusets if neither is the ancestor of the other.
227 Regardless of the
228 .I cpu_exclusive
229 setting,
230 if one cpuset is the ancestor of another,
231 and if both of these cpusets have nonempty
232 .IR cpus ,
233 then their
234 .I cpus
235 must overlap,
236 because the
237 .I cpus
238 of any cpuset are always a subset of the
239 .I cpus
240 of its parent cpuset.
241 .\" ====================== mems ======================
242 .TP
243 .I cpuset.mems
244 List of memory nodes on which
245 processes in this cpuset
246 are allowed to allocate memory.
247 See
248 .B List Format \" Sx
249 below for a description of the format of
250 .IR mems .
251 .\" ==================== mem_exclusive ====================
252 .TP
253 .I cpuset.mem_exclusive
254 Flag (0 or 1).
255 If set (1),
256 the cpuset has exclusive use of its memory nodes
257 (no sibling or cousin may overlap).
258 Also if set (1),
259 the cpuset is a
260 .B Hardwall \" Sx
261 cpuset (see below).
262 By default,
263 this is off (0).
264 Newly created cpusets also initially default this to off (0).
265 .IP
266 Regardless of the
267 .I mem_exclusive
268 setting,
269 if one cpuset is the ancestor of another,
270 then their memory nodes must overlap,
271 because
272 the memory nodes of any cpuset
273 are always a subset of
274 the memory nodes of that cpuset's parent cpuset.
275 .\" ==================== mem_hardwall ====================
276 .TP
277 .IR cpuset.mem_hardwall " (since Linux 2.6.26)"
278 Flag (0 or 1).
279 If set (1),
280 the cpuset is a
281 .B Hardwall
282 cpuset (see below).
283 Unlike
284 .BR mem_exclusive ,
285 there is no constraint on whether cpusets marked
286 .B mem_hardwall
287 may have overlapping memory nodes with sibling or cousin cpusets.
288 By default,
289 this is off (0).
290 Newly created cpusets also initially default this to off (0).
291 .\" ==================== memory_migrate ====================
292 .TP
293 .IR cpuset.memory_migrate " (since Linux 2.6.16)"
294 Flag (0 or 1).
295 If set (1),
296 then memory migration is enabled.
297 By default,
298 this is off (0).
299 See the
300 .B Memory Migration \" Sx
301 section, below.
302 .\" ==================== memory_pressure ====================
303 .TP
304 .IR cpuset.memory_pressure " (since Linux 2.6.16)"
305 A measure of how much memory pressure the processes in this cpuset are causing.
306 See the
307 .B Memory Pressure \" Sx
308 section,
309 below.
310 Unless
311 .I memory_pressure_enabled
312 is enabled,
313 always has value zero (0).
314 This file is read-only.
315 See the
316 .B WARNINGS
317 section,
318 below.
319 .\" ================= memory_pressure_enabled =================
320 .TP
321 .IR cpuset.memory_pressure_enabled " (since Linux 2.6.16)"
322 Flag (0 or 1).
323 This file is present only in the root cpuset,
324 normally
325 .IR /dev/cpuset .
326 If set (1),
327 the
328 .I memory_pressure
329 calculations are enabled for all cpusets in the system.
330 By default,
331 this is off (0).
332 See the
333 .B Memory Pressure \" Sx
334 section,
335 below.
336 .\" ================== memory_spread_page ==================
337 .TP
338 .IR cpuset.memory_spread_page " (since Linux 2.6.17)"
339 Flag (0 or 1).
340 If set (1),
341 pages in the kernel page cache
342 (filesystem buffers)
343 are uniformly spread across the cpuset.
344 By default,
345 this is off (0) in the top cpuset,
346 and inherited from the parent cpuset
347 in newly created cpusets.
348 See the
349 .B Memory Spread \" Sx
350 section,
351 below.
352 .\" ================== memory_spread_slab ==================
353 .TP
354 .IR cpuset.memory_spread_slab " (since Linux 2.6.17)"
355 Flag (0 or 1).
356 If set (1),
357 the kernel slab caches for file I/O
358 (directory and inode structures)
359 are uniformly spread across the cpuset.
360 By default,
361 this is off (0) in the top cpuset,
362 and inherited from the parent cpuset
363 in newly created cpusets.
364 See the
365 .B Memory Spread \" Sx
366 section,
367 below.
368 .\" ================== sched_load_balance ==================
369 .TP
370 .IR cpuset.sched_load_balance " (since Linux 2.6.24)"
371 Flag (0 or 1).
372 If set (1, the default)
373 the kernel will automatically load balance processes in that cpuset
374 over the allowed CPUs in that cpuset.
375 If cleared (0),
376 the kernel will avoid load balancing processes in this cpuset,
377 .I unless
378 some other cpuset with overlapping CPUs has its
379 .I sched_load_balance
380 flag set.
381 See
382 .BR "Scheduler Load Balancing" , \" Sx
383 below,
384 for further details.
385 .\" ================== sched_relax_domain_level ==================
386 .TP
387 .IR cpuset.sched_relax_domain_level " (since Linux 2.6.26)"
388 Integer,
389 between \-1 and a small positive value.
390 The
391 .I sched_relax_domain_level
392 controls the width of the range of CPUs over which
393 the kernel scheduler performs
394 immediate rebalancing of runnable tasks across CPUs.
395 If
396 .I sched_load_balance
397 is disabled,
398 then the setting of
399 .I sched_relax_domain_level
400 does not matter,
401 as no such load balancing is done.
402 If
403 .I sched_load_balance
404 is enabled,
405 then,
406 the higher the value of the
407 .IR sched_relax_domain_level ,
408 the wider the range of CPUs over which immediate load balancing is attempted.
409 See
410 .BR "Scheduler Relax Domain Level" , \" Sx
411 below, for further details.
412 .\" ================== proc cpuset ==================
413 .P
414 In addition to the above pseudo-files in each directory below
415 .IR /dev/cpuset ,
416 each process has a pseudo-file,
417 .IR /proc/ pid /cpuset ,
418 that displays the path of the process's cpuset directory
419 relative to the root of the cpuset filesystem.
420 .\" ================== proc status ==================
421 .P
422 Also the
423 .IR /proc/ pid /status
424 file for each process has four added lines,
425 displaying the process's
426 .I Cpus_allowed
427 (on which CPUs it may be scheduled) and
428 .I Mems_allowed
429 (on which memory nodes it may obtain memory),
430 in the two formats
431 .B Mask Format \" Sx
432 and
433 .B List Format \" Sx
434 (see below)
435 as shown in the following example:
436 .P
437 .in +4n
438 .EX
439 Cpus_allowed: ffffffff,ffffffff,ffffffff,ffffffff
440 Cpus_allowed_list: 0\-127
441 Mems_allowed: ffffffff,ffffffff
442 Mems_allowed_list: 0\-63
443 .EE
444 .in
445 .P
446 The "allowed" fields were added in Linux 2.6.24;
447 the "allowed_list" fields were added in Linux 2.6.26.
448 .\" ================== EXTENDED CAPABILITIES ==================
449 .SH EXTENDED CAPABILITIES
450 In addition to controlling which
451 .I cpus
452 and
453 .I mems
454 a process is allowed to use, cpusets provide the following
455 extended capabilities.
456 .\" ================== Exclusive Cpusets ==================
457 .SS Exclusive cpusets
458 If a cpuset is marked
459 .I cpu_exclusive
460 or
461 .IR mem_exclusive ,
462 no other cpuset, other than a direct ancestor or descendant,
463 may share any of the same CPUs or memory nodes.
464 .P
465 A cpuset that is
466 .I mem_exclusive
467 restricts kernel allocations for
468 buffer cache pages and other internal kernel data pages
469 commonly shared by the kernel across
470 multiple users.
471 All cpusets, whether
472 .I mem_exclusive
473 or not, restrict allocations of memory for user space.
474 This enables configuring a
475 system so that several independent jobs can share common kernel data,
476 while isolating each job's user allocation in
477 its own cpuset.
478 To do this, construct a large
479 .I mem_exclusive
480 cpuset to hold all the jobs, and construct child,
481 .RI non- mem_exclusive
482 cpusets for each individual job.
483 Only a small amount of kernel memory,
484 such as requests from interrupt handlers, is allowed to be
485 placed on memory nodes
486 outside even a
487 .I mem_exclusive
488 cpuset.
489 .\" ================== Hardwall ==================
490 .SS Hardwall
491 A cpuset that has
492 .I mem_exclusive
493 or
494 .I mem_hardwall
495 set is a
496 .I hardwall
497 cpuset.
498 A
499 .I hardwall
500 cpuset restricts kernel allocations for page, buffer,
501 and other data commonly shared by the kernel across multiple users.
502 All cpusets, whether
503 .I hardwall
504 or not, restrict allocations of memory for user space.
505 .P
506 This enables configuring a system so that several independent
507 jobs can share common kernel data, such as filesystem pages,
508 while isolating each job's user allocation in its own cpuset.
509 To do this, construct a large
510 .I hardwall
511 cpuset to hold
512 all the jobs, and construct child cpusets for each individual
513 job which are not
514 .I hardwall
515 cpusets.
516 .P
517 Only a small amount of kernel memory, such as requests from
518 interrupt handlers, is allowed to be taken outside even a
519 .I hardwall
520 cpuset.
521 .\" ================== Notify On Release ==================
522 .SS Notify on release
523 If the
524 .I notify_on_release
525 flag is enabled (1) in a cpuset,
526 then whenever the last process in the cpuset leaves
527 (exits or attaches to some other cpuset)
528 and the last child cpuset of that cpuset is removed,
529 the kernel will run the command
530 .IR /sbin/cpuset_release_agent ,
531 supplying the pathname (relative to the mount point of the
532 cpuset filesystem) of the abandoned cpuset.
533 This enables automatic removal of abandoned cpusets.
534 .P
535 The default value of
536 .I notify_on_release
537 in the root cpuset at system boot is disabled (0).
538 The default value of other cpusets at creation
539 is the current value of their parent's
540 .I notify_on_release
541 setting.
542 .P
543 The command
544 .I /sbin/cpuset_release_agent
545 is invoked, with the name
546 .RI ( /dev/cpuset
547 relative path)
548 of the to-be-released cpuset in
549 .IR argv[1] .
550 .P
551 The usual contents of the command
552 .I /sbin/cpuset_release_agent
553 is simply the shell script:
554 .P
555 .in +4n
556 .EX
557 #!/bin/sh
558 rmdir /dev/cpuset/$1
559 .EE
560 .in
561 .P
562 As with other flag values below, this flag can
563 be changed by writing an ASCII
564 number 0 or 1 (with optional trailing newline)
565 into the file, to clear or set the flag, respectively.
566 .\" ================== Memory Pressure ==================
567 .SS Memory pressure
568 The
569 .I memory_pressure
570 of a cpuset provides a simple per-cpuset running average of
571 the rate that the processes in a cpuset are attempting to free up in-use
572 memory on the nodes of the cpuset to satisfy additional memory requests.
573 .P
574 This enables batch managers that are monitoring jobs running in dedicated
575 cpusets to efficiently detect what level of memory pressure that job
576 is causing.
577 .P
578 This is useful both on tightly managed systems running a wide mix of
579 submitted jobs, which may choose to terminate or reprioritize jobs that
580 are trying to use more memory than allowed on the nodes assigned them,
581 and with tightly coupled, long-running, massively parallel scientific
582 computing jobs that will dramatically fail to meet required performance
583 goals if they start to use more memory than allowed to them.
584 .P
585 This mechanism provides a very economical way for the batch manager
586 to monitor a cpuset for signs of memory pressure.
587 It's up to the batch manager or other user code to decide
588 what action to take if it detects signs of memory pressure.
589 .P
590 Unless memory pressure calculation is enabled by setting the pseudo-file
591 .IR /dev/cpuset/cpuset.memory_pressure_enabled ,
592 it is not computed for any cpuset, and reads from any
593 .I memory_pressure
594 always return zero, as represented by the ASCII string "0\[rs]n".
595 See the
596 .B WARNINGS \" Sx
597 section, below.
598 .P
599 A per-cpuset, running average is employed for the following reasons:
600 .IP \[bu] 3
601 Because this meter is per-cpuset rather than per-process or per virtual
602 memory region, the system load imposed by a batch scheduler monitoring
603 this metric is sharply reduced on large systems, because a scan of
604 the tasklist can be avoided on each set of queries.
605 .IP \[bu]
606 Because this meter is a running average rather than an accumulating
607 counter, a batch scheduler can detect memory pressure with a
608 single read, instead of having to read and accumulate results
609 for a period of time.
610 .IP \[bu]
611 Because this meter is per-cpuset rather than per-process,
612 the batch scheduler can obtain the key information\[em]memory
613 pressure in a cpuset\[em]with a single read, rather than having to
614 query and accumulate results over all the (dynamically changing)
615 set of processes in the cpuset.
616 .P
617 The
618 .I memory_pressure
619 of a cpuset is calculated using a per-cpuset simple digital filter
620 that is kept within the kernel.
621 For each cpuset, this filter tracks
622 the recent rate at which processes attached to that cpuset enter the
623 kernel direct reclaim code.
624 .P
625 The kernel direct reclaim code is entered whenever a process has to
626 satisfy a memory page request by first finding some other page to
627 repurpose, due to lack of any readily available already free pages.
628 Dirty filesystem pages are repurposed by first writing them
629 to disk.
630 Unmodified filesystem buffer pages are repurposed
631 by simply dropping them, though if that page is needed again, it
632 will have to be reread from disk.
633 .P
634 The
635 .I cpuset.memory_pressure
636 file provides an integer number representing the recent (half-life of
637 10 seconds) rate of entries to the direct reclaim code caused by any
638 process in the cpuset, in units of reclaims attempted per second,
639 times 1000.
640 .\" ================== Memory Spread ==================
641 .SS Memory spread
642 There are two Boolean flag files per cpuset that control where the
643 kernel allocates pages for the filesystem buffers and related
644 in-kernel data structures.
645 They are called
646 .I cpuset.memory_spread_page
647 and
648 .IR cpuset.memory_spread_slab .
649 .P
650 If the per-cpuset Boolean flag file
651 .I cpuset.memory_spread_page
652 is set, then
653 the kernel will spread the filesystem buffers (page cache) evenly
654 over all the nodes that the faulting process is allowed to use, instead
655 of preferring to put those pages on the node where the process is running.
656 .P
657 If the per-cpuset Boolean flag file
658 .I cpuset.memory_spread_slab
659 is set,
660 then the kernel will spread some filesystem-related slab caches,
661 such as those for inodes and directory entries, evenly over all the nodes
662 that the faulting process is allowed to use, instead of preferring to
663 put those pages on the node where the process is running.
664 .P
665 The setting of these flags does not affect the data segment
666 (see
667 .BR brk (2))
668 or stack segment pages of a process.
669 .P
670 By default, both kinds of memory spreading are off and the kernel
671 prefers to allocate memory pages on the node local to where the
672 requesting process is running.
673 If that node is not allowed by the
674 process's NUMA memory policy or cpuset configuration or if there are
675 insufficient free memory pages on that node, then the kernel looks
676 for the nearest node that is allowed and has sufficient free memory.
677 .P
678 When new cpusets are created, they inherit the memory spread settings
679 of their parent.
680 .P
681 Setting memory spreading causes allocations for the affected page or
682 slab caches to ignore the process's NUMA memory policy and be spread
683 instead.
684 However, the effect of these changes in memory placement
685 caused by cpuset-specified memory spreading is hidden from the
686 .BR mbind (2)
687 or
688 .BR set_mempolicy (2)
689 calls.
690 These two NUMA memory policy calls always appear to behave as if
691 no cpuset-specified memory spreading is in effect, even if it is.
692 If cpuset memory spreading is subsequently turned off, the NUMA
693 memory policy most recently specified by these calls is automatically
694 reapplied.
695 .P
696 Both
697 .I cpuset.memory_spread_page
698 and
699 .I cpuset.memory_spread_slab
700 are Boolean flag files.
701 By default, they contain "0", meaning that the feature is off
702 for that cpuset.
703 If a "1" is written to that file, that turns the named feature on.
704 .P
705 Cpuset-specified memory spreading behaves similarly to what is known
706 (in other contexts) as round-robin or interleave memory placement.
707 .P
708 Cpuset-specified memory spreading can provide substantial performance
709 improvements for jobs that:
710 .IP \[bu] 3
711 need to place thread-local data on
712 memory nodes close to the CPUs
713 which are running the threads
714 that most frequently access that data;
715 but also
716 .IP \[bu]
717 need to access large filesystem data sets
718 that must to be spread across the several nodes in the job's cpuset
719 in order to fit.
720 .P
721 Without this policy,
722 the memory allocation across the nodes in the job's cpuset
723 can become very uneven,
724 especially for jobs that might have
725 just a single thread initializing or reading in the data set.
726 .\" ================== Memory Migration ==================
727 .SS Memory migration
728 Normally,
729 under the default setting (disabled) of
730 .IR cpuset.memory_migrate ,
731 once a page is allocated
732 (given a physical page of main memory),
733 then that page stays on whatever node it was allocated,
734 so long as it remains allocated,
735 even if the cpuset's memory-placement policy
736 .I mems
737 subsequently changes.
738 .P
739 When memory migration is enabled in a cpuset,
740 if the
741 .I mems
742 setting of the cpuset is changed,
743 then any memory page in use
744 by any process in the cpuset that is
745 on a memory node that is no longer allowed
746 will be migrated to a memory node that is allowed.
747 .P
748 Furthermore,
749 if a process is moved into a cpuset with
750 .I memory_migrate
751 enabled,
752 any memory pages it uses that were on memory nodes
753 allowed in its previous cpuset,
754 but which are not allowed in its new cpuset,
755 will be migrated to a memory node allowed in the new cpuset.
756 .P
757 The relative placement of a migrated page within the cpuset
758 is preserved during these migration operations if possible.
759 For example,
760 if the page was on the second valid node of the prior cpuset,
761 then the page will be placed on the second valid node of the new cpuset,
762 if possible.
763 .\" ================== Scheduler Load Balancing ==================
764 .SS Scheduler load balancing
765 The kernel scheduler automatically load balances processes.
766 If one CPU is underutilized,
767 the kernel will look for processes on other more
768 overloaded CPUs and move those processes to the underutilized CPU,
769 within the constraints of such placement mechanisms as cpusets and
770 .BR sched_setaffinity (2).
771 .P
772 The algorithmic cost of load balancing and its impact on key shared
773 kernel data structures such as the process list increases more than
774 linearly with the number of CPUs being balanced.
775 For example, it
776 costs more to load balance across one large set of CPUs than it does
777 to balance across two smaller sets of CPUs, each of half the size
778 of the larger set.
779 (The precise relationship between the number of CPUs being balanced
780 and the cost of load balancing depends
781 on implementation details of the kernel process scheduler, which is
782 subject to change over time, as improved kernel scheduler algorithms
783 are implemented.)
784 .P
785 The per-cpuset flag
786 .I sched_load_balance
787 provides a mechanism to suppress this automatic scheduler load
788 balancing in cases where it is not needed and suppressing it would have
789 worthwhile performance benefits.
790 .P
791 By default, load balancing is done across all CPUs, except those
792 marked isolated using the kernel boot time "isolcpus=" argument.
793 (See
794 .BR "Scheduler Relax Domain Level" , \" Sx
795 below, to change this default.)
796 .P
797 This default load balancing across all CPUs is not well suited to
798 the following two situations:
799 .IP \[bu] 3
800 On large systems, load balancing across many CPUs is expensive.
801 If the system is managed using cpusets to place independent jobs
802 on separate sets of CPUs, full load balancing is unnecessary.
803 .IP \[bu]
804 Systems supporting real-time on some CPUs need to minimize
805 system overhead on those CPUs, including avoiding process load
806 balancing if that is not needed.
807 .P
808 When the per-cpuset flag
809 .I sched_load_balance
810 is enabled (the default setting),
811 it requests load balancing across
812 all the CPUs in that cpuset's allowed CPUs,
813 ensuring that load balancing can move a process (not otherwise pinned,
814 as by
815 .BR sched_setaffinity (2))
816 from any CPU in that cpuset to any other.
817 .P
818 When the per-cpuset flag
819 .I sched_load_balance
820 is disabled, then the
821 scheduler will avoid load balancing across the CPUs in that cpuset,
822 .I except
823 in so far as is necessary because some overlapping cpuset
824 has
825 .I sched_load_balance
826 enabled.
827 .P
828 So, for example, if the top cpuset has the flag
829 .I sched_load_balance
830 enabled, then the scheduler will load balance across all
831 CPUs, and the setting of the
832 .I sched_load_balance
833 flag in other cpusets has no effect,
834 as we're already fully load balancing.
835 .P
836 Therefore in the above two situations, the flag
837 .I sched_load_balance
838 should be disabled in the top cpuset, and only some of the smaller,
839 child cpusets would have this flag enabled.
840 .P
841 When doing this, you don't usually want to leave any unpinned processes in
842 the top cpuset that might use nontrivial amounts of CPU, as such processes
843 may be artificially constrained to some subset of CPUs, depending on
844 the particulars of this flag setting in descendant cpusets.
845 Even if such a process could use spare CPU cycles in some other CPUs,
846 the kernel scheduler might not consider the possibility of
847 load balancing that process to the underused CPU.
848 .P
849 Of course, processes pinned to a particular CPU can be left in a cpuset
850 that disables
851 .I sched_load_balance
852 as those processes aren't going anywhere else anyway.
853 .\" ================== Scheduler Relax Domain Level ==================
854 .SS Scheduler relax domain level
855 The kernel scheduler performs immediate load balancing whenever
856 a CPU becomes free or another task becomes runnable.
857 This load
858 balancing works to ensure that as many CPUs as possible are usefully
859 employed running tasks.
860 The kernel also performs periodic load
861 balancing off the software clock described in
862 .BR time (7).
863 The setting of
864 .I sched_relax_domain_level
865 applies only to immediate load balancing.
866 Regardless of the
867 .I sched_relax_domain_level
868 setting, periodic load balancing is attempted over all CPUs
869 (unless disabled by turning off
870 .IR sched_load_balance .)
871 In any case, of course, tasks will be scheduled to run only on
872 CPUs allowed by their cpuset, as modified by
873 .BR sched_setaffinity (2)
874 system calls.
875 .P
876 On small systems, such as those with just a few CPUs, immediate load
877 balancing is useful to improve system interactivity and to minimize
878 wasteful idle CPU cycles.
879 But on large systems, attempting immediate
880 load balancing across a large number of CPUs can be more costly than
881 it is worth, depending on the particular performance characteristics
882 of the job mix and the hardware.
883 .P
884 The exact meaning of the small integer values of
885 .I sched_relax_domain_level
886 will depend on internal
887 implementation details of the kernel scheduler code and on the
888 non-uniform architecture of the hardware.
889 Both of these will evolve
890 over time and vary by system architecture and kernel version.
891 .P
892 As of this writing, when this capability was introduced in Linux
893 2.6.26, on certain popular architectures, the positive values of
894 .I sched_relax_domain_level
895 have the following meanings.
896 .P
897 .PD 0
898 .TP
899 .B 1
900 Perform immediate load balancing across Hyper-Thread
901 siblings on the same core.
902 .TP
903 .B 2
904 Perform immediate load balancing across other cores in the same package.
905 .TP
906 .B 3
907 Perform immediate load balancing across other CPUs
908 on the same node or blade.
909 .TP
910 .B 4
911 Perform immediate load balancing across over several
912 (implementation detail) nodes [On NUMA systems].
913 .TP
914 .B 5
915 Perform immediate load balancing across over all CPUs
916 in system [On NUMA systems].
917 .PD
918 .P
919 The
920 .I sched_relax_domain_level
921 value of zero (0) always means
922 don't perform immediate load balancing,
923 hence that load balancing is done only periodically,
924 not immediately when a CPU becomes available or another task becomes
925 runnable.
926 .P
927 The
928 .I sched_relax_domain_level
929 value of minus one (\-1)
930 always means use the system default value.
931 The system default value can vary by architecture and kernel version.
932 This system default value can be changed by kernel
933 boot-time "relax_domain_level=" argument.
934 .P
935 In the case of multiple overlapping cpusets which have conflicting
936 .I sched_relax_domain_level
937 values, then the highest such value
938 applies to all CPUs in any of the overlapping cpusets.
939 In such cases,
940 .B \-1
941 is the lowest value,
942 overridden by any other value,
943 and
944 .B 0
945 is the next lowest value.
946 .SH FORMATS
947 The following formats are used to represent sets of
948 CPUs and memory nodes.
949 .\" ================== Mask Format ==================
950 .SS Mask format
951 The Mask Format is used to
952 represent CPU and memory-node bit masks in the
953 .IR /proc/ pid /status
954 file.
955 .P
956 This format displays each 32-bit
957 word in hexadecimal (using ASCII characters "0" - "9" and "a" - "f");
958 words are filled with leading zeros, if required.
959 For masks longer than one word, a comma separator is used between words.
960 Words are displayed in big-endian
961 order, which has the most significant bit first.
962 The hex digits within a word are also in big-endian order.
963 .P
964 The number of 32-bit words displayed is the minimum number needed to
965 display all bits of the bit mask, based on the size of the bit mask.
966 .P
967 Examples of the
968 .BR Mask\~Format :
969 .P
970 .in +4n
971 .EX
972 00000001 # just bit 0 set
973 40000000,00000000,00000000 # just bit 94 set
974 00000001,00000000,00000000 # just bit 64 set
975 000000ff,00000000 # bits 32\-39 set
976 00000000,000e3862 # 1,5,6,11\-13,17\-19 set
977 .EE
978 .in
979 .P
980 A mask with bits 0, 1, 2, 4, 8, 16, 32, and 64 set displays as:
981 .P
982 .in +4n
983 .EX
984 00000001,00000001,00010117
985 .EE
986 .in
987 .P
988 The first "1" is for bit 64, the
989 second for bit 32, the third for bit 16, the fourth for bit 8, the
990 fifth for bit 4, and the "7" is for bits 2, 1, and 0.
991 .\" ================== List Format ==================
992 .SS List format
993 The List Format for
994 .I cpus
995 and
996 .I mems
997 is a comma-separated list of CPU or memory-node
998 numbers and ranges of numbers, in ASCII decimal.
999 .P
1000 Examples of the
1001 .BR List\~Format :
1002 .P
1003 .in +4n
1004 .EX
1005 0\-4,9 # bits 0, 1, 2, 3, 4, and 9 set
1006 0\-2,7,12\-14 # bits 0, 1, 2, 7, 12, 13, and 14 set
1007 .EE
1008 .in
1009 .\" ================== RULES ==================
1010 .SH RULES
1011 The following rules apply to each cpuset:
1012 .IP \[bu] 3
1013 Its CPUs and memory nodes must be a (possibly equal)
1014 subset of its parent's.
1015 .IP \[bu]
1016 It can be marked
1017 .I cpu_exclusive
1018 only if its parent is.
1019 .IP \[bu]
1020 It can be marked
1021 .I mem_exclusive
1022 only if its parent is.
1023 .IP \[bu]
1024 If it is
1025 .IR cpu_exclusive ,
1026 its CPUs may not overlap any sibling.
1027 .IP \[bu]
1028 If it is
1029 .IR mem_exclusive ,
1030 its memory nodes may not overlap any sibling.
1031 .\" ================== PERMISSIONS ==================
1032 .SH PERMISSIONS
1033 The permissions of a cpuset are determined by the permissions
1034 of the directories and pseudo-files in the cpuset filesystem,
1035 normally mounted at
1036 .IR /dev/cpuset .
1037 .P
1038 For instance, a process can put itself in some other cpuset (than
1039 its current one) if it can write the
1040 .I tasks
1041 file for that cpuset.
1042 This requires execute permission on the encompassing directories
1043 and write permission on the
1044 .I tasks
1045 file.
1046 .P
1047 An additional constraint is applied to requests to place some
1048 other process in a cpuset.
1049 One process may not attach another to
1050 a cpuset unless it would have permission to send that process
1051 a signal (see
1052 .BR kill (2)).
1053 .P
1054 A process may create a child cpuset if it can access and write the
1055 parent cpuset directory.
1056 It can modify the CPUs or memory nodes
1057 in a cpuset if it can access that cpuset's directory (execute
1058 permissions on the each of the parent directories) and write the
1059 corresponding
1060 .I cpus
1061 or
1062 .I mems
1063 file.
1064 .P
1065 There is one minor difference between the manner in which these
1066 permissions are evaluated and the manner in which normal filesystem
1067 operation permissions are evaluated.
1068 The kernel interprets
1069 relative pathnames starting at a process's current working directory.
1070 Even if one is operating on a cpuset file, relative pathnames
1071 are interpreted relative to the process's current working directory,
1072 not relative to the process's current cpuset.
1073 The only ways that
1074 cpuset paths relative to a process's current cpuset can be used are
1075 if either the process's current working directory is its cpuset
1076 (it first did a
1077 .B cd
1078 or
1079 .BR chdir (2)
1080 to its cpuset directory beneath
1081 .IR /dev/cpuset ,
1082 which is a bit unusual)
1083 or if some user code converts the relative cpuset path to a
1084 full filesystem path.
1085 .P
1086 In theory, this means that user code should specify cpusets
1087 using absolute pathnames, which requires knowing the mount point of
1088 the cpuset filesystem (usually, but not necessarily,
1089 .IR /dev/cpuset ).
1090 In practice, all user level code that this author is aware of
1091 simply assumes that if the cpuset filesystem is mounted, then
1092 it is mounted at
1093 .IR /dev/cpuset .
1094 Furthermore, it is common practice for carefully written
1095 user code to verify the presence of the pseudo-file
1096 .I /dev/cpuset/tasks
1097 in order to verify that the cpuset pseudo-filesystem
1098 is currently mounted.
1099 .\" ================== WARNINGS ==================
1100 .SH WARNINGS
1101 .SS Enabling memory_pressure
1102 By default, the per-cpuset file
1103 .I cpuset.memory_pressure
1104 always contains zero (0).
1105 Unless this feature is enabled by writing "1" to the pseudo-file
1106 .IR /dev/cpuset/cpuset.memory_pressure_enabled ,
1107 the kernel does
1108 not compute per-cpuset
1109 .IR memory_pressure .
1110 .SS Using the echo command
1111 When using the
1112 .B echo
1113 command at the shell prompt to change the values of cpuset files,
1114 beware that the built-in
1115 .B echo
1116 command in some shells does not display an error message if the
1117 .BR write (2)
1118 system call fails.
1119 .\" Gack! csh(1)'s echo does this
1120 For example, if the command:
1121 .P
1122 .in +4n
1123 .EX
1124 echo 19 > cpuset.mems
1125 .EE
1126 .in
1127 .P
1128 failed because memory node 19 was not allowed (perhaps
1129 the current system does not have a memory node 19), then the
1130 .B echo
1131 command might not display any error.
1132 It is better to use the
1133 .B /bin/echo
1134 external command to change cpuset file settings, as this
1135 command will display
1136 .BR write (2)
1137 errors, as in the example:
1138 .P
1139 .in +4n
1140 .EX
1141 /bin/echo 19 > cpuset.mems
1142 /bin/echo: write error: Invalid argument
1143 .EE
1144 .in
1145 .\" ================== EXCEPTIONS ==================
1146 .SH EXCEPTIONS
1147 .SS Memory placement
1148 Not all allocations of system memory are constrained by cpusets,
1149 for the following reasons.
1150 .P
1151 If hot-plug functionality is used to remove all the CPUs that are
1152 currently assigned to a cpuset,
1153 then the kernel will automatically
1154 update the
1155 .I cpus_allowed
1156 of all processes attached to CPUs in that cpuset
1157 to allow all CPUs.
1158 When memory hot-plug functionality for removing memory nodes is available,
1159 a similar exception is expected to apply there as well.
1160 In general,
1161 the kernel prefers to violate cpuset placement,
1162 rather than starving a process that has had all its allowed CPUs or
1163 memory nodes taken offline.
1164 User code should reconfigure cpusets to refer only to online CPUs
1165 and memory nodes when using hot-plug to add or remove such resources.
1166 .P
1167 A few kernel-critical,
1168 internal memory-allocation requests,
1169 marked GFP_ATOMIC,
1170 must be satisfied immediately.
1171 The kernel may drop some request or malfunction
1172 if one of these allocations fail.
1173 If such a request cannot be satisfied within the current process's cpuset,
1174 then we relax the cpuset,
1175 and look for memory anywhere we can find it.
1176 It's better to violate the cpuset than stress the kernel.
1177 .P
1178 Allocations of memory requested by kernel drivers
1179 while processing an interrupt
1180 lack any relevant process context,
1181 and are not confined by cpusets.
1182 .SS Renaming cpusets
1183 You can use the
1184 .BR rename (2)
1185 system call to rename cpusets.
1186 Only simple renaming is supported;
1187 that is,
1188 changing the name of a cpuset directory is permitted,
1189 but moving a directory into a different directory is not permitted.
1190 .\" ================== ERRORS ==================
1191 .SH ERRORS
1192 The Linux kernel implementation of cpusets sets
1193 .I errno
1194 to specify the reason for a failed system call affecting cpusets.
1195 .P
1196 The possible
1197 .I errno
1198 settings and their meaning when set on
1199 a failed cpuset call are as listed below.
1200 .TP
1201 .B E2BIG
1202 Attempted a
1203 .BR write (2)
1204 on a special cpuset file
1205 with a length larger than some kernel-determined upper
1206 limit on the length of such writes.
1207 .TP
1208 .B EACCES
1209 Attempted to
1210 .BR write (2)
1211 the process ID (PID) of a process to a cpuset
1212 .I tasks
1213 file when one lacks permission to move that process.
1214 .TP
1215 .B EACCES
1216 Attempted to add,
1217 using
1218 .BR write (2),
1219 a CPU or memory node to a cpuset,
1220 when that CPU or memory node
1221 was not already in its parent.
1222 .TP
1223 .B EACCES
1224 Attempted to set, using
1225 .BR write (2),
1226 .I cpuset.cpu_exclusive
1227 or
1228 .I cpuset.mem_exclusive
1229 on a cpuset whose parent lacks the same setting.
1230 .TP
1231 .B EACCES
1232 Attempted to
1233 .BR write (2)
1234 a
1235 .I cpuset.memory_pressure
1236 file.
1237 .TP
1238 .B EACCES
1239 Attempted to create a file in a cpuset directory.
1240 .TP
1241 .B EBUSY
1242 Attempted to remove, using
1243 .BR rmdir (2),
1244 a cpuset with attached processes.
1245 .TP
1246 .B EBUSY
1247 Attempted to remove, using
1248 .BR rmdir (2),
1249 a cpuset with child cpusets.
1250 .TP
1251 .B EBUSY
1252 Attempted to remove
1253 a CPU or memory node from a cpuset
1254 that is also in a child of that cpuset.
1255 .TP
1256 .B EEXIST
1257 Attempted to create, using
1258 .BR mkdir (2),
1259 a cpuset that already exists.
1260 .TP
1261 .B EEXIST
1262 Attempted to
1263 .BR rename (2)
1264 a cpuset to a name that already exists.
1265 .TP
1266 .B EFAULT
1267 Attempted to
1268 .BR read (2)
1269 or
1270 .BR write (2)
1271 a cpuset file using
1272 a buffer that is outside the writing processes accessible address space.
1273 .TP
1274 .B EINVAL
1275 Attempted to change a cpuset, using
1276 .BR write (2),
1277 in a way that would violate a
1278 .I cpu_exclusive
1279 or
1280 .I mem_exclusive
1281 attribute of that cpuset or any of its siblings.
1282 .TP
1283 .B EINVAL
1284 Attempted to
1285 .BR write (2)
1286 an empty
1287 .I cpuset.cpus
1288 or
1289 .I cpuset.mems
1290 list to a cpuset which has attached processes or child cpusets.
1291 .TP
1292 .B EINVAL
1293 Attempted to
1294 .BR write (2)
1295 a
1296 .I cpuset.cpus
1297 or
1298 .I cpuset.mems
1299 list which included a range with the second number smaller than
1300 the first number.
1301 .TP
1302 .B EINVAL
1303 Attempted to
1304 .BR write (2)
1305 a
1306 .I cpuset.cpus
1307 or
1308 .I cpuset.mems
1309 list which included an invalid character in the string.
1310 .TP
1311 .B EINVAL
1312 Attempted to
1313 .BR write (2)
1314 a list to a
1315 .I cpuset.cpus
1316 file that did not include any online CPUs.
1317 .TP
1318 .B EINVAL
1319 Attempted to
1320 .BR write (2)
1321 a list to a
1322 .I cpuset.mems
1323 file that did not include any online memory nodes.
1324 .TP
1325 .B EINVAL
1326 Attempted to
1327 .BR write (2)
1328 a list to a
1329 .I cpuset.mems
1330 file that included a node that held no memory.
1331 .TP
1332 .B EIO
1333 Attempted to
1334 .BR write (2)
1335 a string to a cpuset
1336 .I tasks
1337 file that
1338 does not begin with an ASCII decimal integer.
1339 .TP
1340 .B EIO
1341 Attempted to
1342 .BR rename (2)
1343 a cpuset into a different directory.
1344 .TP
1345 .B ENAMETOOLONG
1346 Attempted to
1347 .BR read (2)
1348 a
1349 .IR /proc/ pid /cpuset
1350 file for a cpuset path that is longer than the kernel page size.
1351 .TP
1352 .B ENAMETOOLONG
1353 Attempted to create, using
1354 .BR mkdir (2),
1355 a cpuset whose base directory name is longer than 255 characters.
1356 .TP
1357 .B ENAMETOOLONG
1358 Attempted to create, using
1359 .BR mkdir (2),
1360 a cpuset whose full pathname,
1361 including the mount point (typically "/dev/cpuset/") prefix,
1362 is longer than 4095 characters.
1363 .TP
1364 .B ENODEV
1365 The cpuset was removed by another process at the same time as a
1366 .BR write (2)
1367 was attempted on one of the pseudo-files in the cpuset directory.
1368 .TP
1369 .B ENOENT
1370 Attempted to create, using
1371 .BR mkdir (2),
1372 a cpuset in a parent cpuset that doesn't exist.
1373 .TP
1374 .B ENOENT
1375 Attempted to
1376 .BR access (2)
1377 or
1378 .BR open (2)
1379 a nonexistent file in a cpuset directory.
1380 .TP
1381 .B ENOMEM
1382 Insufficient memory is available within the kernel;
1383 can occur on a variety of system calls affecting cpusets,
1384 but only if the system is extremely short of memory.
1385 .TP
1386 .B ENOSPC
1387 Attempted to
1388 .BR write (2)
1389 the process ID (PID)
1390 of a process to a cpuset
1391 .I tasks
1392 file when the cpuset had an empty
1393 .I cpuset.cpus
1394 or empty
1395 .I cpuset.mems
1396 setting.
1397 .TP
1398 .B ENOSPC
1399 Attempted to
1400 .BR write (2)
1401 an empty
1402 .I cpuset.cpus
1403 or
1404 .I cpuset.mems
1405 setting to a cpuset that
1406 has tasks attached.
1407 .TP
1408 .B ENOTDIR
1409 Attempted to
1410 .BR rename (2)
1411 a nonexistent cpuset.
1412 .TP
1413 .B EPERM
1414 Attempted to remove a file from a cpuset directory.
1415 .TP
1416 .B ERANGE
1417 Specified a
1418 .I cpuset.cpus
1419 or
1420 .I cpuset.mems
1421 list to the kernel which included a number too large for the kernel
1422 to set in its bit masks.
1423 .TP
1424 .B ESRCH
1425 Attempted to
1426 .BR write (2)
1427 the process ID (PID) of a nonexistent process to a cpuset
1428 .I tasks
1429 file.
1430 .\" ================== VERSIONS ==================
1431 .SH VERSIONS
1432 Cpusets appeared in Linux 2.6.12.
1433 .\" ================== NOTES ==================
1434 .SH NOTES
1435 Despite its name, the
1436 .I pid
1437 parameter is actually a thread ID,
1438 and each thread in a threaded group can be attached to a different
1439 cpuset.
1440 The value returned from a call to
1441 .BR gettid (2)
1442 can be passed in the argument
1443 .IR pid .
1444 .\" ================== BUGS ==================
1445 .SH BUGS
1446 .I cpuset.memory_pressure
1447 cpuset files can be opened
1448 for writing, creation, or truncation, but then the
1449 .BR write (2)
1450 fails with
1451 .I errno
1452 set to
1453 .BR EACCES ,
1454 and the creation and truncation options on
1455 .BR open (2)
1456 have no effect.
1457 .\" ================== EXAMPLES ==================
1458 .SH EXAMPLES
1459 The following examples demonstrate querying and setting cpuset
1460 options using shell commands.
1461 .SS Creating and attaching to a cpuset.
1462 To create a new cpuset and attach the current command shell to it,
1463 the steps are:
1464 .P
1465 .PD 0
1466 .IP (1) 5
1467 mkdir /dev/cpuset (if not already done)
1468 .IP (2)
1469 mount \-t cpuset none /dev/cpuset (if not already done)
1470 .IP (3)
1471 Create the new cpuset using
1472 .BR mkdir (1).
1473 .IP (4)
1474 Assign CPUs and memory nodes to the new cpuset.
1475 .IP (5)
1476 Attach the shell to the new cpuset.
1477 .PD
1478 .P
1479 For example, the following sequence of commands will set up a cpuset
1480 named "Charlie", containing just CPUs 2 and 3, and memory node 1,
1481 and then attach the current shell to that cpuset.
1482 .P
1483 .in +4n
1484 .EX
1485 .RB "$" " mkdir /dev/cpuset"
1486 .RB "$" " mount \-t cpuset cpuset /dev/cpuset"
1487 .RB "$" " cd /dev/cpuset"
1488 .RB "$" " mkdir Charlie"
1489 .RB "$" " cd Charlie"
1490 .RB "$" " /bin/echo 2\-3 > cpuset.cpus"
1491 .RB "$" " /bin/echo 1 > cpuset.mems"
1492 .RB "$" " /bin/echo $$ > tasks"
1493 # The current shell is now running in cpuset Charlie
1494 # The next line should display \[aq]/Charlie\[aq]
1495 .RB "$" " cat /proc/self/cpuset"
1496 .EE
1497 .in
1498 .\"
1499 .SS Migrating a job to different memory nodes.
1500 To migrate a job (the set of processes attached to a cpuset)
1501 to different CPUs and memory nodes in the system, including moving
1502 the memory pages currently allocated to that job,
1503 perform the following steps.
1504 .P
1505 .PD 0
1506 .IP (1) 5
1507 Let's say we want to move the job in cpuset
1508 .I alpha
1509 (CPUs 4\[en]7 and memory nodes 2\[en]3) to a new cpuset
1510 .I beta
1511 (CPUs 16\[en]19 and memory nodes 8\[en]9).
1512 .IP (2)
1513 First create the new cpuset
1514 .IR beta .
1515 .IP (3)
1516 Then allow CPUs 16\[en]19 and memory nodes 8\[en]9 in
1517 .IR beta .
1518 .IP (4)
1519 Then enable
1520 .I memory_migration
1521 in
1522 .IR beta .
1523 .IP (5)
1524 Then move each process from
1525 .I alpha
1526 to
1527 .IR beta .
1528 .PD
1529 .P
1530 The following sequence of commands accomplishes this.
1531 .P
1532 .in +4n
1533 .EX
1534 .RB "$" " cd /dev/cpuset"
1535 .RB "$" " mkdir beta"
1536 .RB "$" " cd beta"
1537 .RB "$" " /bin/echo 16\-19 > cpuset.cpus"
1538 .RB "$" " /bin/echo 8\-9 > cpuset.mems"
1539 .RB "$" " /bin/echo 1 > cpuset.memory_migrate"
1540 .RB "$" " while read i; do /bin/echo $i; done < ../alpha/tasks > tasks"
1541 .EE
1542 .in
1543 .P
1544 The above should move any processes in
1545 .I alpha
1546 to
1547 .IR beta ,
1548 and any memory held by these processes on memory nodes 2\[en]3 to memory
1549 nodes 8\[en]9, respectively.
1550 .P
1551 Notice that the last step of the above sequence did not do:
1552 .P
1553 .in +4n
1554 .EX
1555 .RB "$" " cp ../alpha/tasks tasks"
1556 .EE
1557 .in
1558 .P
1559 The
1560 .I while
1561 loop, rather than the seemingly easier use of the
1562 .BR cp (1)
1563 command, was necessary because
1564 only one process PID at a time may be written to the
1565 .I tasks
1566 file.
1567 .P
1568 The same effect (writing one PID at a time) as the
1569 .I while
1570 loop can be accomplished more efficiently, in fewer keystrokes and in
1571 syntax that works on any shell, but alas more obscurely, by using the
1572 .B \-u
1573 (unbuffered) option of
1574 .BR sed (1):
1575 .P
1576 .in +4n
1577 .EX
1578 .RB "$" " sed \-un p < ../alpha/tasks > tasks"
1579 .EE
1580 .in
1581 .\" ================== SEE ALSO ==================
1582 .SH SEE ALSO
1583 .BR taskset (1),
1584 .BR get_mempolicy (2),
1585 .BR getcpu (2),
1586 .BR mbind (2),
1587 .BR sched_getaffinity (2),
1588 .BR sched_setaffinity (2),
1589 .BR sched_setscheduler (2),
1590 .BR set_mempolicy (2),
1591 .BR CPU_SET (3),
1592 .BR proc (5),
1593 .BR cgroups (7),
1594 .BR numa (7),
1595 .BR sched (7),
1596 .BR migratepages (8),
1597 .BR numactl (8)
1598 .P
1599 .I Documentation/admin\-guide/cgroup\-v1/cpusets.rst
1600 in the Linux kernel source tree