| 1 |
.\" Copyright 2008, Silicon Graphics, Inc. |
| 2 |
.\" Copyright, the authors of the Linux man-pages project |
| 3 |
.\" |
| 4 |
.\" SPDX-License-Identifier: GPL-2.0-only |
| 5 |
.\" |
| 6 |
.TH cpuset 7 2026-02-08 "Linux man-pages 6.18" |
| 7 |
.SH NAME |
| 8 |
cpuset \- confine processes to processor and memory node subsets |
| 9 |
.SH DESCRIPTION |
| 10 |
The cpuset filesystem is a pseudo-filesystem interface |
| 11 |
to the kernel cpuset mechanism, |
| 12 |
which is used to control the processor placement |
| 13 |
and memory placement of processes. |
| 14 |
It is commonly mounted at |
| 15 |
.IR /dev/cpuset . |
| 16 |
.P |
| 17 |
On systems with kernels compiled with built in support for cpusets, |
| 18 |
all processes are attached to a cpuset, and cpusets are always present. |
| 19 |
If a system supports cpusets, then it will have the entry |
| 20 |
.B nodev cpuset |
| 21 |
in the file |
| 22 |
.IR /proc/filesystems . |
| 23 |
By mounting the cpuset filesystem (see the |
| 24 |
.B EXAMPLES |
| 25 |
section below), |
| 26 |
the administrator can configure the cpusets on a system |
| 27 |
to control the processor and memory placement of processes |
| 28 |
on that system. |
| 29 |
By default, if the cpuset configuration |
| 30 |
on a system is not modified or if the cpuset filesystem |
| 31 |
is not even mounted, then the cpuset mechanism, |
| 32 |
though present, has no effect on the system's behavior. |
| 33 |
.P |
| 34 |
A cpuset defines a list of CPUs and memory nodes. |
| 35 |
.P |
| 36 |
The CPUs of a system include |
| 37 |
all the logical processing units on which a process can execute, |
| 38 |
including, |
| 39 |
if present, |
| 40 |
multiple processor cores within a package |
| 41 |
and Hyper-Threads within a processor core. |
| 42 |
Memory nodes include all distinct banks of main memory; |
| 43 |
small and SMP systems typically have |
| 44 |
just one memory node that contains all the system's main memory, |
| 45 |
while NUMA (non-uniform memory access) systems have multiple memory nodes. |
| 46 |
.P |
| 47 |
Cpusets are represented as directories in a hierarchical pseudo-filesystem, |
| 48 |
where the top directory in the hierarchy |
| 49 |
.RI ( /dev/cpuset ) |
| 50 |
represents the entire system (all online CPUs and memory nodes) |
| 51 |
and any cpuset that is the child (descendant) of another parent cpuset |
| 52 |
contains a subset of that parent's CPUs and memory nodes. |
| 53 |
The directories and files representing cpusets |
| 54 |
have normal filesystem permissions. |
| 55 |
.P |
| 56 |
Every process in the system belongs to exactly one cpuset. |
| 57 |
A process is confined to run |
| 58 |
only on the CPUs in the cpuset it belongs to, |
| 59 |
and to allocate memory |
| 60 |
only on the memory nodes in that cpuset. |
| 61 |
When a process |
| 62 |
.BR fork (2)s, |
| 63 |
the child process is placed in the same cpuset as its parent. |
| 64 |
With sufficient privilege, |
| 65 |
a process may be moved from one cpuset to another |
| 66 |
and the allowed CPUs and memory nodes of an existing cpuset |
| 67 |
may be changed. |
| 68 |
.P |
| 69 |
When the system begins booting, |
| 70 |
a single cpuset is defined |
| 71 |
that includes all CPUs and memory nodes on the system, |
| 72 |
and all processes are in that cpuset. |
| 73 |
During the boot process, |
| 74 |
or later during normal system operation, |
| 75 |
other cpusets may be created, |
| 76 |
as subdirectories of this top cpuset, |
| 77 |
under the control of the system administrator, |
| 78 |
and processes may be placed in these other cpusets. |
| 79 |
.P |
| 80 |
Cpusets are integrated with the |
| 81 |
.BR sched_setaffinity (2) |
| 82 |
scheduling affinity mechanism and the |
| 83 |
.BR mbind (2) |
| 84 |
and |
| 85 |
.BR set_mempolicy (2) |
| 86 |
memory-placement mechanisms in the kernel. |
| 87 |
Neither of these mechanisms let a process |
| 88 |
make use of a CPU or memory node that is not allowed by that process's cpuset. |
| 89 |
If changes to a process's cpuset placement |
| 90 |
conflict with these other mechanisms, |
| 91 |
then cpuset placement is enforced |
| 92 |
even if it means overriding these other mechanisms. |
| 93 |
The kernel accomplishes this overriding |
| 94 |
by silently restricting the CPUs and memory nodes |
| 95 |
requested by these other mechanisms |
| 96 |
to those allowed by the invoking process's cpuset. |
| 97 |
This can result in these other calls returning an error, |
| 98 |
if for example, |
| 99 |
such a call ends up requesting an empty set of CPUs or memory nodes, |
| 100 |
after that request is restricted to the invoking process's cpuset. |
| 101 |
.P |
| 102 |
Typically, |
| 103 |
a cpuset is used to manage |
| 104 |
the CPU and memory-node confinement for a set of cooperating processes, |
| 105 |
such as a batch scheduler job, |
| 106 |
and these other mechanisms are used to manage |
| 107 |
the placement of individual processes or memory regions |
| 108 |
within that set or job. |
| 109 |
.SH FILES |
| 110 |
Each directory below |
| 111 |
.I /dev/cpuset |
| 112 |
represents a cpuset and contains a fixed set of pseudo-files |
| 113 |
describing the state of that cpuset. |
| 114 |
.P |
| 115 |
New cpusets are created using the |
| 116 |
.BR mkdir (2) |
| 117 |
system call or the |
| 118 |
.BR mkdir (1) |
| 119 |
command. |
| 120 |
The properties of a cpuset, |
| 121 |
such as its flags, |
| 122 |
allowed CPUs and memory nodes, |
| 123 |
and attached processes, |
| 124 |
are queried and modified |
| 125 |
by reading or writing to the appropriate file in that cpuset's directory, |
| 126 |
as listed below. |
| 127 |
.P |
| 128 |
The pseudo-files in each cpuset directory |
| 129 |
are automatically created when the cpuset is created, |
| 130 |
as a result of the |
| 131 |
.BR mkdir (2) |
| 132 |
invocation. |
| 133 |
It is not possible to directly add or remove these pseudo-files. |
| 134 |
.P |
| 135 |
A cpuset directory that contains no child cpuset directories, |
| 136 |
and has no attached processes, |
| 137 |
can be removed using |
| 138 |
.BR rmdir (2) |
| 139 |
or |
| 140 |
.BR rmdir (1). |
| 141 |
It is not necessary, or possible, |
| 142 |
to remove the pseudo-files inside the directory before removing it. |
| 143 |
.P |
| 144 |
The pseudo-files in each cpuset directory |
| 145 |
are small text files that may be |
| 146 |
read and written using traditional shell utilities such as |
| 147 |
.BR cat (1), |
| 148 |
and |
| 149 |
.BR echo (1), |
| 150 |
or from a program by using file I/O library functions or system calls, |
| 151 |
such as |
| 152 |
.BR open (2), |
| 153 |
.BR read (2), |
| 154 |
.BR write (2), |
| 155 |
and |
| 156 |
.BR close (2). |
| 157 |
.P |
| 158 |
The pseudo-files in a cpuset directory represent internal kernel state |
| 159 |
and do not have any persistent image on disk. |
| 160 |
Each of these per-cpuset files is listed and described below. |
| 161 |
.\" ====================== tasks ====================== |
| 162 |
.TP |
| 163 |
.I tasks |
| 164 |
List of the process IDs (PIDs) of the processes in that cpuset. |
| 165 |
The list is formatted as a series of |
| 166 |
ASCII decimal numbers, |
| 167 |
each followed by a newline. |
| 168 |
A process may be added to a cpuset |
| 169 |
(automatically removing it from the cpuset that previously contained it) |
| 170 |
by writing its PID to that cpuset's |
| 171 |
.I tasks |
| 172 |
file |
| 173 |
(with or without a trailing newline). |
| 174 |
.IP |
| 175 |
.B Warning: |
| 176 |
only one PID may be written to the |
| 177 |
.I tasks |
| 178 |
file at a time. |
| 179 |
If a string is written that contains more than one PID, |
| 180 |
only the first one will be used. |
| 181 |
.\" =================== notify_on_release =================== |
| 182 |
.TP |
| 183 |
.I notify_on_release |
| 184 |
Flag (0 or 1). |
| 185 |
If set (1), |
| 186 |
that cpuset will receive special handling after it is released, |
| 187 |
that is, |
| 188 |
after all processes cease using it |
| 189 |
(i.e., terminate or are moved to a different cpuset) |
| 190 |
and all child cpuset directories have been removed. |
| 191 |
See the |
| 192 |
.B Notify On Release \" Sx |
| 193 |
section, below. |
| 194 |
.\" ====================== cpus ====================== |
| 195 |
.TP |
| 196 |
.I cpuset.cpus |
| 197 |
List of the physical numbers of the CPUs |
| 198 |
on which processes in that cpuset are allowed to execute. |
| 199 |
See |
| 200 |
.B List Format \" Sx |
| 201 |
below for a description of the format of |
| 202 |
.IR cpus . |
| 203 |
.IP |
| 204 |
The CPUs allowed to a cpuset may be changed by |
| 205 |
writing a new list to its |
| 206 |
.I cpus |
| 207 |
file. |
| 208 |
.\" ==================== cpu_exclusive ==================== |
| 209 |
.TP |
| 210 |
.I cpuset.cpu_exclusive |
| 211 |
Flag (0 or 1). |
| 212 |
If set (1), |
| 213 |
the cpuset has exclusive use of its CPUs |
| 214 |
(no sibling or cousin cpuset may overlap CPUs). |
| 215 |
By default, |
| 216 |
this is off (0). |
| 217 |
Newly created cpusets also initially default this to off (0). |
| 218 |
.IP |
| 219 |
Two cpusets are |
| 220 |
.I sibling |
| 221 |
cpusets if they share the same parent cpuset in the |
| 222 |
.I /dev/cpuset |
| 223 |
hierarchy. |
| 224 |
Two cpusets are |
| 225 |
.I cousin |
| 226 |
cpusets if neither is the ancestor of the other. |
| 227 |
Regardless of the |
| 228 |
.I cpu_exclusive |
| 229 |
setting, |
| 230 |
if one cpuset is the ancestor of another, |
| 231 |
and if both of these cpusets have nonempty |
| 232 |
.IR cpus , |
| 233 |
then their |
| 234 |
.I cpus |
| 235 |
must overlap, |
| 236 |
because the |
| 237 |
.I cpus |
| 238 |
of any cpuset are always a subset of the |
| 239 |
.I cpus |
| 240 |
of its parent cpuset. |
| 241 |
.\" ====================== mems ====================== |
| 242 |
.TP |
| 243 |
.I cpuset.mems |
| 244 |
List of memory nodes on which |
| 245 |
processes in this cpuset |
| 246 |
are allowed to allocate memory. |
| 247 |
See |
| 248 |
.B List Format \" Sx |
| 249 |
below for a description of the format of |
| 250 |
.IR mems . |
| 251 |
.\" ==================== mem_exclusive ==================== |
| 252 |
.TP |
| 253 |
.I cpuset.mem_exclusive |
| 254 |
Flag (0 or 1). |
| 255 |
If set (1), |
| 256 |
the cpuset has exclusive use of its memory nodes |
| 257 |
(no sibling or cousin may overlap). |
| 258 |
Also if set (1), |
| 259 |
the cpuset is a |
| 260 |
.B Hardwall \" Sx |
| 261 |
cpuset (see below). |
| 262 |
By default, |
| 263 |
this is off (0). |
| 264 |
Newly created cpusets also initially default this to off (0). |
| 265 |
.IP |
| 266 |
Regardless of the |
| 267 |
.I mem_exclusive |
| 268 |
setting, |
| 269 |
if one cpuset is the ancestor of another, |
| 270 |
then their memory nodes must overlap, |
| 271 |
because |
| 272 |
the memory nodes of any cpuset |
| 273 |
are always a subset of |
| 274 |
the memory nodes of that cpuset's parent cpuset. |
| 275 |
.\" ==================== mem_hardwall ==================== |
| 276 |
.TP |
| 277 |
.IR cpuset.mem_hardwall " (since Linux 2.6.26)" |
| 278 |
Flag (0 or 1). |
| 279 |
If set (1), |
| 280 |
the cpuset is a |
| 281 |
.B Hardwall |
| 282 |
cpuset (see below). |
| 283 |
Unlike |
| 284 |
.BR mem_exclusive , |
| 285 |
there is no constraint on whether cpusets marked |
| 286 |
.B mem_hardwall |
| 287 |
may have overlapping memory nodes with sibling or cousin cpusets. |
| 288 |
By default, |
| 289 |
this is off (0). |
| 290 |
Newly created cpusets also initially default this to off (0). |
| 291 |
.\" ==================== memory_migrate ==================== |
| 292 |
.TP |
| 293 |
.IR cpuset.memory_migrate " (since Linux 2.6.16)" |
| 294 |
Flag (0 or 1). |
| 295 |
If set (1), |
| 296 |
then memory migration is enabled. |
| 297 |
By default, |
| 298 |
this is off (0). |
| 299 |
See the |
| 300 |
.B Memory Migration \" Sx |
| 301 |
section, below. |
| 302 |
.\" ==================== memory_pressure ==================== |
| 303 |
.TP |
| 304 |
.IR cpuset.memory_pressure " (since Linux 2.6.16)" |
| 305 |
A measure of how much memory pressure the processes in this cpuset are causing. |
| 306 |
See the |
| 307 |
.B Memory Pressure \" Sx |
| 308 |
section, |
| 309 |
below. |
| 310 |
Unless |
| 311 |
.I memory_pressure_enabled |
| 312 |
is enabled, |
| 313 |
always has value zero (0). |
| 314 |
This file is read-only. |
| 315 |
See the |
| 316 |
.B WARNINGS |
| 317 |
section, |
| 318 |
below. |
| 319 |
.\" ================= memory_pressure_enabled ================= |
| 320 |
.TP |
| 321 |
.IR cpuset.memory_pressure_enabled " (since Linux 2.6.16)" |
| 322 |
Flag (0 or 1). |
| 323 |
This file is present only in the root cpuset, |
| 324 |
normally |
| 325 |
.IR /dev/cpuset . |
| 326 |
If set (1), |
| 327 |
the |
| 328 |
.I memory_pressure |
| 329 |
calculations are enabled for all cpusets in the system. |
| 330 |
By default, |
| 331 |
this is off (0). |
| 332 |
See the |
| 333 |
.B Memory Pressure \" Sx |
| 334 |
section, |
| 335 |
below. |
| 336 |
.\" ================== memory_spread_page ================== |
| 337 |
.TP |
| 338 |
.IR cpuset.memory_spread_page " (since Linux 2.6.17)" |
| 339 |
Flag (0 or 1). |
| 340 |
If set (1), |
| 341 |
pages in the kernel page cache |
| 342 |
(filesystem buffers) |
| 343 |
are uniformly spread across the cpuset. |
| 344 |
By default, |
| 345 |
this is off (0) in the top cpuset, |
| 346 |
and inherited from the parent cpuset |
| 347 |
in newly created cpusets. |
| 348 |
See the |
| 349 |
.B Memory Spread \" Sx |
| 350 |
section, |
| 351 |
below. |
| 352 |
.\" ================== memory_spread_slab ================== |
| 353 |
.TP |
| 354 |
.IR cpuset.memory_spread_slab " (since Linux 2.6.17)" |
| 355 |
Flag (0 or 1). |
| 356 |
If set (1), |
| 357 |
the kernel slab caches for file I/O |
| 358 |
(directory and inode structures) |
| 359 |
are uniformly spread across the cpuset. |
| 360 |
By default, |
| 361 |
this is off (0) in the top cpuset, |
| 362 |
and inherited from the parent cpuset |
| 363 |
in newly created cpusets. |
| 364 |
See the |
| 365 |
.B Memory Spread \" Sx |
| 366 |
section, |
| 367 |
below. |
| 368 |
.\" ================== sched_load_balance ================== |
| 369 |
.TP |
| 370 |
.IR cpuset.sched_load_balance " (since Linux 2.6.24)" |
| 371 |
Flag (0 or 1). |
| 372 |
If set (1, the default) |
| 373 |
the kernel will automatically load balance processes in that cpuset |
| 374 |
over the allowed CPUs in that cpuset. |
| 375 |
If cleared (0), |
| 376 |
the kernel will avoid load balancing processes in this cpuset, |
| 377 |
.I unless |
| 378 |
some other cpuset with overlapping CPUs has its |
| 379 |
.I sched_load_balance |
| 380 |
flag set. |
| 381 |
See |
| 382 |
.BR "Scheduler Load Balancing" , \" Sx |
| 383 |
below, |
| 384 |
for further details. |
| 385 |
.\" ================== sched_relax_domain_level ================== |
| 386 |
.TP |
| 387 |
.IR cpuset.sched_relax_domain_level " (since Linux 2.6.26)" |
| 388 |
Integer, |
| 389 |
between \-1 and a small positive value. |
| 390 |
The |
| 391 |
.I sched_relax_domain_level |
| 392 |
controls the width of the range of CPUs over which |
| 393 |
the kernel scheduler performs |
| 394 |
immediate rebalancing of runnable tasks across CPUs. |
| 395 |
If |
| 396 |
.I sched_load_balance |
| 397 |
is disabled, |
| 398 |
then the setting of |
| 399 |
.I sched_relax_domain_level |
| 400 |
does not matter, |
| 401 |
as no such load balancing is done. |
| 402 |
If |
| 403 |
.I sched_load_balance |
| 404 |
is enabled, |
| 405 |
then, |
| 406 |
the higher the value of the |
| 407 |
.IR sched_relax_domain_level , |
| 408 |
the wider the range of CPUs over which immediate load balancing is attempted. |
| 409 |
See |
| 410 |
.BR "Scheduler Relax Domain Level" , \" Sx |
| 411 |
below, for further details. |
| 412 |
.\" ================== proc cpuset ================== |
| 413 |
.P |
| 414 |
In addition to the above pseudo-files in each directory below |
| 415 |
.IR /dev/cpuset , |
| 416 |
each process has a pseudo-file, |
| 417 |
.IR /proc/ pid /cpuset , |
| 418 |
that displays the path of the process's cpuset directory |
| 419 |
relative to the root of the cpuset filesystem. |
| 420 |
.\" ================== proc status ================== |
| 421 |
.P |
| 422 |
Also the |
| 423 |
.IR /proc/ pid /status |
| 424 |
file for each process has four added lines, |
| 425 |
displaying the process's |
| 426 |
.I Cpus_allowed |
| 427 |
(on which CPUs it may be scheduled) and |
| 428 |
.I Mems_allowed |
| 429 |
(on which memory nodes it may obtain memory), |
| 430 |
in the two formats |
| 431 |
.B Mask Format \" Sx |
| 432 |
and |
| 433 |
.B List Format \" Sx |
| 434 |
(see below) |
| 435 |
as shown in the following example: |
| 436 |
.P |
| 437 |
.in +4n |
| 438 |
.EX |
| 439 |
Cpus_allowed: ffffffff,ffffffff,ffffffff,ffffffff |
| 440 |
Cpus_allowed_list: 0\-127 |
| 441 |
Mems_allowed: ffffffff,ffffffff |
| 442 |
Mems_allowed_list: 0\-63 |
| 443 |
.EE |
| 444 |
.in |
| 445 |
.P |
| 446 |
The "allowed" fields were added in Linux 2.6.24; |
| 447 |
the "allowed_list" fields were added in Linux 2.6.26. |
| 448 |
.\" ================== EXTENDED CAPABILITIES ================== |
| 449 |
.SH EXTENDED CAPABILITIES |
| 450 |
In addition to controlling which |
| 451 |
.I cpus |
| 452 |
and |
| 453 |
.I mems |
| 454 |
a process is allowed to use, cpusets provide the following |
| 455 |
extended capabilities. |
| 456 |
.\" ================== Exclusive Cpusets ================== |
| 457 |
.SS Exclusive cpusets |
| 458 |
If a cpuset is marked |
| 459 |
.I cpu_exclusive |
| 460 |
or |
| 461 |
.IR mem_exclusive , |
| 462 |
no other cpuset, other than a direct ancestor or descendant, |
| 463 |
may share any of the same CPUs or memory nodes. |
| 464 |
.P |
| 465 |
A cpuset that is |
| 466 |
.I mem_exclusive |
| 467 |
restricts kernel allocations for |
| 468 |
buffer cache pages and other internal kernel data pages |
| 469 |
commonly shared by the kernel across |
| 470 |
multiple users. |
| 471 |
All cpusets, whether |
| 472 |
.I mem_exclusive |
| 473 |
or not, restrict allocations of memory for user space. |
| 474 |
This enables configuring a |
| 475 |
system so that several independent jobs can share common kernel data, |
| 476 |
while isolating each job's user allocation in |
| 477 |
its own cpuset. |
| 478 |
To do this, construct a large |
| 479 |
.I mem_exclusive |
| 480 |
cpuset to hold all the jobs, and construct child, |
| 481 |
.RI non- mem_exclusive |
| 482 |
cpusets for each individual job. |
| 483 |
Only a small amount of kernel memory, |
| 484 |
such as requests from interrupt handlers, is allowed to be |
| 485 |
placed on memory nodes |
| 486 |
outside even a |
| 487 |
.I mem_exclusive |
| 488 |
cpuset. |
| 489 |
.\" ================== Hardwall ================== |
| 490 |
.SS Hardwall |
| 491 |
A cpuset that has |
| 492 |
.I mem_exclusive |
| 493 |
or |
| 494 |
.I mem_hardwall |
| 495 |
set is a |
| 496 |
.I hardwall |
| 497 |
cpuset. |
| 498 |
A |
| 499 |
.I hardwall |
| 500 |
cpuset restricts kernel allocations for page, buffer, |
| 501 |
and other data commonly shared by the kernel across multiple users. |
| 502 |
All cpusets, whether |
| 503 |
.I hardwall |
| 504 |
or not, restrict allocations of memory for user space. |
| 505 |
.P |
| 506 |
This enables configuring a system so that several independent |
| 507 |
jobs can share common kernel data, such as filesystem pages, |
| 508 |
while isolating each job's user allocation in its own cpuset. |
| 509 |
To do this, construct a large |
| 510 |
.I hardwall |
| 511 |
cpuset to hold |
| 512 |
all the jobs, and construct child cpusets for each individual |
| 513 |
job which are not |
| 514 |
.I hardwall |
| 515 |
cpusets. |
| 516 |
.P |
| 517 |
Only a small amount of kernel memory, such as requests from |
| 518 |
interrupt handlers, is allowed to be taken outside even a |
| 519 |
.I hardwall |
| 520 |
cpuset. |
| 521 |
.\" ================== Notify On Release ================== |
| 522 |
.SS Notify on release |
| 523 |
If the |
| 524 |
.I notify_on_release |
| 525 |
flag is enabled (1) in a cpuset, |
| 526 |
then whenever the last process in the cpuset leaves |
| 527 |
(exits or attaches to some other cpuset) |
| 528 |
and the last child cpuset of that cpuset is removed, |
| 529 |
the kernel will run the command |
| 530 |
.IR /sbin/cpuset_release_agent , |
| 531 |
supplying the pathname (relative to the mount point of the |
| 532 |
cpuset filesystem) of the abandoned cpuset. |
| 533 |
This enables automatic removal of abandoned cpusets. |
| 534 |
.P |
| 535 |
The default value of |
| 536 |
.I notify_on_release |
| 537 |
in the root cpuset at system boot is disabled (0). |
| 538 |
The default value of other cpusets at creation |
| 539 |
is the current value of their parent's |
| 540 |
.I notify_on_release |
| 541 |
setting. |
| 542 |
.P |
| 543 |
The command |
| 544 |
.I /sbin/cpuset_release_agent |
| 545 |
is invoked, with the name |
| 546 |
.RI ( /dev/cpuset |
| 547 |
relative path) |
| 548 |
of the to-be-released cpuset in |
| 549 |
.IR argv[1] . |
| 550 |
.P |
| 551 |
The usual contents of the command |
| 552 |
.I /sbin/cpuset_release_agent |
| 553 |
is simply the shell script: |
| 554 |
.P |
| 555 |
.in +4n |
| 556 |
.EX |
| 557 |
#!/bin/sh |
| 558 |
rmdir /dev/cpuset/$1 |
| 559 |
.EE |
| 560 |
.in |
| 561 |
.P |
| 562 |
As with other flag values below, this flag can |
| 563 |
be changed by writing an ASCII |
| 564 |
number 0 or 1 (with optional trailing newline) |
| 565 |
into the file, to clear or set the flag, respectively. |
| 566 |
.\" ================== Memory Pressure ================== |
| 567 |
.SS Memory pressure |
| 568 |
The |
| 569 |
.I memory_pressure |
| 570 |
of a cpuset provides a simple per-cpuset running average of |
| 571 |
the rate that the processes in a cpuset are attempting to free up in-use |
| 572 |
memory on the nodes of the cpuset to satisfy additional memory requests. |
| 573 |
.P |
| 574 |
This enables batch managers that are monitoring jobs running in dedicated |
| 575 |
cpusets to efficiently detect what level of memory pressure that job |
| 576 |
is causing. |
| 577 |
.P |
| 578 |
This is useful both on tightly managed systems running a wide mix of |
| 579 |
submitted jobs, which may choose to terminate or reprioritize jobs that |
| 580 |
are trying to use more memory than allowed on the nodes assigned them, |
| 581 |
and with tightly coupled, long-running, massively parallel scientific |
| 582 |
computing jobs that will dramatically fail to meet required performance |
| 583 |
goals if they start to use more memory than allowed to them. |
| 584 |
.P |
| 585 |
This mechanism provides a very economical way for the batch manager |
| 586 |
to monitor a cpuset for signs of memory pressure. |
| 587 |
It's up to the batch manager or other user code to decide |
| 588 |
what action to take if it detects signs of memory pressure. |
| 589 |
.P |
| 590 |
Unless memory pressure calculation is enabled by setting the pseudo-file |
| 591 |
.IR /dev/cpuset/cpuset.memory_pressure_enabled , |
| 592 |
it is not computed for any cpuset, and reads from any |
| 593 |
.I memory_pressure |
| 594 |
always return zero, as represented by the ASCII string "0\[rs]n". |
| 595 |
See the |
| 596 |
.B WARNINGS \" Sx |
| 597 |
section, below. |
| 598 |
.P |
| 599 |
A per-cpuset, running average is employed for the following reasons: |
| 600 |
.IP \[bu] 3 |
| 601 |
Because this meter is per-cpuset rather than per-process or per virtual |
| 602 |
memory region, the system load imposed by a batch scheduler monitoring |
| 603 |
this metric is sharply reduced on large systems, because a scan of |
| 604 |
the tasklist can be avoided on each set of queries. |
| 605 |
.IP \[bu] |
| 606 |
Because this meter is a running average rather than an accumulating |
| 607 |
counter, a batch scheduler can detect memory pressure with a |
| 608 |
single read, instead of having to read and accumulate results |
| 609 |
for a period of time. |
| 610 |
.IP \[bu] |
| 611 |
Because this meter is per-cpuset rather than per-process, |
| 612 |
the batch scheduler can obtain the key information\[em]memory |
| 613 |
pressure in a cpuset\[em]with a single read, rather than having to |
| 614 |
query and accumulate results over all the (dynamically changing) |
| 615 |
set of processes in the cpuset. |
| 616 |
.P |
| 617 |
The |
| 618 |
.I memory_pressure |
| 619 |
of a cpuset is calculated using a per-cpuset simple digital filter |
| 620 |
that is kept within the kernel. |
| 621 |
For each cpuset, this filter tracks |
| 622 |
the recent rate at which processes attached to that cpuset enter the |
| 623 |
kernel direct reclaim code. |
| 624 |
.P |
| 625 |
The kernel direct reclaim code is entered whenever a process has to |
| 626 |
satisfy a memory page request by first finding some other page to |
| 627 |
repurpose, due to lack of any readily available already free pages. |
| 628 |
Dirty filesystem pages are repurposed by first writing them |
| 629 |
to disk. |
| 630 |
Unmodified filesystem buffer pages are repurposed |
| 631 |
by simply dropping them, though if that page is needed again, it |
| 632 |
will have to be reread from disk. |
| 633 |
.P |
| 634 |
The |
| 635 |
.I cpuset.memory_pressure |
| 636 |
file provides an integer number representing the recent (half-life of |
| 637 |
10 seconds) rate of entries to the direct reclaim code caused by any |
| 638 |
process in the cpuset, in units of reclaims attempted per second, |
| 639 |
times 1000. |
| 640 |
.\" ================== Memory Spread ================== |
| 641 |
.SS Memory spread |
| 642 |
There are two Boolean flag files per cpuset that control where the |
| 643 |
kernel allocates pages for the filesystem buffers and related |
| 644 |
in-kernel data structures. |
| 645 |
They are called |
| 646 |
.I cpuset.memory_spread_page |
| 647 |
and |
| 648 |
.IR cpuset.memory_spread_slab . |
| 649 |
.P |
| 650 |
If the per-cpuset Boolean flag file |
| 651 |
.I cpuset.memory_spread_page |
| 652 |
is set, then |
| 653 |
the kernel will spread the filesystem buffers (page cache) evenly |
| 654 |
over all the nodes that the faulting process is allowed to use, instead |
| 655 |
of preferring to put those pages on the node where the process is running. |
| 656 |
.P |
| 657 |
If the per-cpuset Boolean flag file |
| 658 |
.I cpuset.memory_spread_slab |
| 659 |
is set, |
| 660 |
then the kernel will spread some filesystem-related slab caches, |
| 661 |
such as those for inodes and directory entries, evenly over all the nodes |
| 662 |
that the faulting process is allowed to use, instead of preferring to |
| 663 |
put those pages on the node where the process is running. |
| 664 |
.P |
| 665 |
The setting of these flags does not affect the data segment |
| 666 |
(see |
| 667 |
.BR brk (2)) |
| 668 |
or stack segment pages of a process. |
| 669 |
.P |
| 670 |
By default, both kinds of memory spreading are off and the kernel |
| 671 |
prefers to allocate memory pages on the node local to where the |
| 672 |
requesting process is running. |
| 673 |
If that node is not allowed by the |
| 674 |
process's NUMA memory policy or cpuset configuration or if there are |
| 675 |
insufficient free memory pages on that node, then the kernel looks |
| 676 |
for the nearest node that is allowed and has sufficient free memory. |
| 677 |
.P |
| 678 |
When new cpusets are created, they inherit the memory spread settings |
| 679 |
of their parent. |
| 680 |
.P |
| 681 |
Setting memory spreading causes allocations for the affected page or |
| 682 |
slab caches to ignore the process's NUMA memory policy and be spread |
| 683 |
instead. |
| 684 |
However, the effect of these changes in memory placement |
| 685 |
caused by cpuset-specified memory spreading is hidden from the |
| 686 |
.BR mbind (2) |
| 687 |
or |
| 688 |
.BR set_mempolicy (2) |
| 689 |
calls. |
| 690 |
These two NUMA memory policy calls always appear to behave as if |
| 691 |
no cpuset-specified memory spreading is in effect, even if it is. |
| 692 |
If cpuset memory spreading is subsequently turned off, the NUMA |
| 693 |
memory policy most recently specified by these calls is automatically |
| 694 |
reapplied. |
| 695 |
.P |
| 696 |
Both |
| 697 |
.I cpuset.memory_spread_page |
| 698 |
and |
| 699 |
.I cpuset.memory_spread_slab |
| 700 |
are Boolean flag files. |
| 701 |
By default, they contain "0", meaning that the feature is off |
| 702 |
for that cpuset. |
| 703 |
If a "1" is written to that file, that turns the named feature on. |
| 704 |
.P |
| 705 |
Cpuset-specified memory spreading behaves similarly to what is known |
| 706 |
(in other contexts) as round-robin or interleave memory placement. |
| 707 |
.P |
| 708 |
Cpuset-specified memory spreading can provide substantial performance |
| 709 |
improvements for jobs that: |
| 710 |
.IP \[bu] 3 |
| 711 |
need to place thread-local data on |
| 712 |
memory nodes close to the CPUs |
| 713 |
which are running the threads |
| 714 |
that most frequently access that data; |
| 715 |
but also |
| 716 |
.IP \[bu] |
| 717 |
need to access large filesystem data sets |
| 718 |
that must to be spread across the several nodes in the job's cpuset |
| 719 |
in order to fit. |
| 720 |
.P |
| 721 |
Without this policy, |
| 722 |
the memory allocation across the nodes in the job's cpuset |
| 723 |
can become very uneven, |
| 724 |
especially for jobs that might have |
| 725 |
just a single thread initializing or reading in the data set. |
| 726 |
.\" ================== Memory Migration ================== |
| 727 |
.SS Memory migration |
| 728 |
Normally, |
| 729 |
under the default setting (disabled) of |
| 730 |
.IR cpuset.memory_migrate , |
| 731 |
once a page is allocated |
| 732 |
(given a physical page of main memory), |
| 733 |
then that page stays on whatever node it was allocated, |
| 734 |
so long as it remains allocated, |
| 735 |
even if the cpuset's memory-placement policy |
| 736 |
.I mems |
| 737 |
subsequently changes. |
| 738 |
.P |
| 739 |
When memory migration is enabled in a cpuset, |
| 740 |
if the |
| 741 |
.I mems |
| 742 |
setting of the cpuset is changed, |
| 743 |
then any memory page in use |
| 744 |
by any process in the cpuset that is |
| 745 |
on a memory node that is no longer allowed |
| 746 |
will be migrated to a memory node that is allowed. |
| 747 |
.P |
| 748 |
Furthermore, |
| 749 |
if a process is moved into a cpuset with |
| 750 |
.I memory_migrate |
| 751 |
enabled, |
| 752 |
any memory pages it uses that were on memory nodes |
| 753 |
allowed in its previous cpuset, |
| 754 |
but which are not allowed in its new cpuset, |
| 755 |
will be migrated to a memory node allowed in the new cpuset. |
| 756 |
.P |
| 757 |
The relative placement of a migrated page within the cpuset |
| 758 |
is preserved during these migration operations if possible. |
| 759 |
For example, |
| 760 |
if the page was on the second valid node of the prior cpuset, |
| 761 |
then the page will be placed on the second valid node of the new cpuset, |
| 762 |
if possible. |
| 763 |
.\" ================== Scheduler Load Balancing ================== |
| 764 |
.SS Scheduler load balancing |
| 765 |
The kernel scheduler automatically load balances processes. |
| 766 |
If one CPU is underutilized, |
| 767 |
the kernel will look for processes on other more |
| 768 |
overloaded CPUs and move those processes to the underutilized CPU, |
| 769 |
within the constraints of such placement mechanisms as cpusets and |
| 770 |
.BR sched_setaffinity (2). |
| 771 |
.P |
| 772 |
The algorithmic cost of load balancing and its impact on key shared |
| 773 |
kernel data structures such as the process list increases more than |
| 774 |
linearly with the number of CPUs being balanced. |
| 775 |
For example, it |
| 776 |
costs more to load balance across one large set of CPUs than it does |
| 777 |
to balance across two smaller sets of CPUs, each of half the size |
| 778 |
of the larger set. |
| 779 |
(The precise relationship between the number of CPUs being balanced |
| 780 |
and the cost of load balancing depends |
| 781 |
on implementation details of the kernel process scheduler, which is |
| 782 |
subject to change over time, as improved kernel scheduler algorithms |
| 783 |
are implemented.) |
| 784 |
.P |
| 785 |
The per-cpuset flag |
| 786 |
.I sched_load_balance |
| 787 |
provides a mechanism to suppress this automatic scheduler load |
| 788 |
balancing in cases where it is not needed and suppressing it would have |
| 789 |
worthwhile performance benefits. |
| 790 |
.P |
| 791 |
By default, load balancing is done across all CPUs, except those |
| 792 |
marked isolated using the kernel boot time "isolcpus=" argument. |
| 793 |
(See |
| 794 |
.BR "Scheduler Relax Domain Level" , \" Sx |
| 795 |
below, to change this default.) |
| 796 |
.P |
| 797 |
This default load balancing across all CPUs is not well suited to |
| 798 |
the following two situations: |
| 799 |
.IP \[bu] 3 |
| 800 |
On large systems, load balancing across many CPUs is expensive. |
| 801 |
If the system is managed using cpusets to place independent jobs |
| 802 |
on separate sets of CPUs, full load balancing is unnecessary. |
| 803 |
.IP \[bu] |
| 804 |
Systems supporting real-time on some CPUs need to minimize |
| 805 |
system overhead on those CPUs, including avoiding process load |
| 806 |
balancing if that is not needed. |
| 807 |
.P |
| 808 |
When the per-cpuset flag |
| 809 |
.I sched_load_balance |
| 810 |
is enabled (the default setting), |
| 811 |
it requests load balancing across |
| 812 |
all the CPUs in that cpuset's allowed CPUs, |
| 813 |
ensuring that load balancing can move a process (not otherwise pinned, |
| 814 |
as by |
| 815 |
.BR sched_setaffinity (2)) |
| 816 |
from any CPU in that cpuset to any other. |
| 817 |
.P |
| 818 |
When the per-cpuset flag |
| 819 |
.I sched_load_balance |
| 820 |
is disabled, then the |
| 821 |
scheduler will avoid load balancing across the CPUs in that cpuset, |
| 822 |
.I except |
| 823 |
in so far as is necessary because some overlapping cpuset |
| 824 |
has |
| 825 |
.I sched_load_balance |
| 826 |
enabled. |
| 827 |
.P |
| 828 |
So, for example, if the top cpuset has the flag |
| 829 |
.I sched_load_balance |
| 830 |
enabled, then the scheduler will load balance across all |
| 831 |
CPUs, and the setting of the |
| 832 |
.I sched_load_balance |
| 833 |
flag in other cpusets has no effect, |
| 834 |
as we're already fully load balancing. |
| 835 |
.P |
| 836 |
Therefore in the above two situations, the flag |
| 837 |
.I sched_load_balance |
| 838 |
should be disabled in the top cpuset, and only some of the smaller, |
| 839 |
child cpusets would have this flag enabled. |
| 840 |
.P |
| 841 |
When doing this, you don't usually want to leave any unpinned processes in |
| 842 |
the top cpuset that might use nontrivial amounts of CPU, as such processes |
| 843 |
may be artificially constrained to some subset of CPUs, depending on |
| 844 |
the particulars of this flag setting in descendant cpusets. |
| 845 |
Even if such a process could use spare CPU cycles in some other CPUs, |
| 846 |
the kernel scheduler might not consider the possibility of |
| 847 |
load balancing that process to the underused CPU. |
| 848 |
.P |
| 849 |
Of course, processes pinned to a particular CPU can be left in a cpuset |
| 850 |
that disables |
| 851 |
.I sched_load_balance |
| 852 |
as those processes aren't going anywhere else anyway. |
| 853 |
.\" ================== Scheduler Relax Domain Level ================== |
| 854 |
.SS Scheduler relax domain level |
| 855 |
The kernel scheduler performs immediate load balancing whenever |
| 856 |
a CPU becomes free or another task becomes runnable. |
| 857 |
This load |
| 858 |
balancing works to ensure that as many CPUs as possible are usefully |
| 859 |
employed running tasks. |
| 860 |
The kernel also performs periodic load |
| 861 |
balancing off the software clock described in |
| 862 |
.BR time (7). |
| 863 |
The setting of |
| 864 |
.I sched_relax_domain_level |
| 865 |
applies only to immediate load balancing. |
| 866 |
Regardless of the |
| 867 |
.I sched_relax_domain_level |
| 868 |
setting, periodic load balancing is attempted over all CPUs |
| 869 |
(unless disabled by turning off |
| 870 |
.IR sched_load_balance .) |
| 871 |
In any case, of course, tasks will be scheduled to run only on |
| 872 |
CPUs allowed by their cpuset, as modified by |
| 873 |
.BR sched_setaffinity (2) |
| 874 |
system calls. |
| 875 |
.P |
| 876 |
On small systems, such as those with just a few CPUs, immediate load |
| 877 |
balancing is useful to improve system interactivity and to minimize |
| 878 |
wasteful idle CPU cycles. |
| 879 |
But on large systems, attempting immediate |
| 880 |
load balancing across a large number of CPUs can be more costly than |
| 881 |
it is worth, depending on the particular performance characteristics |
| 882 |
of the job mix and the hardware. |
| 883 |
.P |
| 884 |
The exact meaning of the small integer values of |
| 885 |
.I sched_relax_domain_level |
| 886 |
will depend on internal |
| 887 |
implementation details of the kernel scheduler code and on the |
| 888 |
non-uniform architecture of the hardware. |
| 889 |
Both of these will evolve |
| 890 |
over time and vary by system architecture and kernel version. |
| 891 |
.P |
| 892 |
As of this writing, when this capability was introduced in Linux |
| 893 |
2.6.26, on certain popular architectures, the positive values of |
| 894 |
.I sched_relax_domain_level |
| 895 |
have the following meanings. |
| 896 |
.P |
| 897 |
.PD 0 |
| 898 |
.TP |
| 899 |
.B 1 |
| 900 |
Perform immediate load balancing across Hyper-Thread |
| 901 |
siblings on the same core. |
| 902 |
.TP |
| 903 |
.B 2 |
| 904 |
Perform immediate load balancing across other cores in the same package. |
| 905 |
.TP |
| 906 |
.B 3 |
| 907 |
Perform immediate load balancing across other CPUs |
| 908 |
on the same node or blade. |
| 909 |
.TP |
| 910 |
.B 4 |
| 911 |
Perform immediate load balancing across over several |
| 912 |
(implementation detail) nodes [On NUMA systems]. |
| 913 |
.TP |
| 914 |
.B 5 |
| 915 |
Perform immediate load balancing across over all CPUs |
| 916 |
in system [On NUMA systems]. |
| 917 |
.PD |
| 918 |
.P |
| 919 |
The |
| 920 |
.I sched_relax_domain_level |
| 921 |
value of zero (0) always means |
| 922 |
don't perform immediate load balancing, |
| 923 |
hence that load balancing is done only periodically, |
| 924 |
not immediately when a CPU becomes available or another task becomes |
| 925 |
runnable. |
| 926 |
.P |
| 927 |
The |
| 928 |
.I sched_relax_domain_level |
| 929 |
value of minus one (\-1) |
| 930 |
always means use the system default value. |
| 931 |
The system default value can vary by architecture and kernel version. |
| 932 |
This system default value can be changed by kernel |
| 933 |
boot-time "relax_domain_level=" argument. |
| 934 |
.P |
| 935 |
In the case of multiple overlapping cpusets which have conflicting |
| 936 |
.I sched_relax_domain_level |
| 937 |
values, then the highest such value |
| 938 |
applies to all CPUs in any of the overlapping cpusets. |
| 939 |
In such cases, |
| 940 |
.B \-1 |
| 941 |
is the lowest value, |
| 942 |
overridden by any other value, |
| 943 |
and |
| 944 |
.B 0 |
| 945 |
is the next lowest value. |
| 946 |
.SH FORMATS |
| 947 |
The following formats are used to represent sets of |
| 948 |
CPUs and memory nodes. |
| 949 |
.\" ================== Mask Format ================== |
| 950 |
.SS Mask format |
| 951 |
The Mask Format is used to |
| 952 |
represent CPU and memory-node bit masks in the |
| 953 |
.IR /proc/ pid /status |
| 954 |
file. |
| 955 |
.P |
| 956 |
This format displays each 32-bit |
| 957 |
word in hexadecimal (using ASCII characters "0" - "9" and "a" - "f"); |
| 958 |
words are filled with leading zeros, if required. |
| 959 |
For masks longer than one word, a comma separator is used between words. |
| 960 |
Words are displayed in big-endian |
| 961 |
order, which has the most significant bit first. |
| 962 |
The hex digits within a word are also in big-endian order. |
| 963 |
.P |
| 964 |
The number of 32-bit words displayed is the minimum number needed to |
| 965 |
display all bits of the bit mask, based on the size of the bit mask. |
| 966 |
.P |
| 967 |
Examples of the |
| 968 |
.BR Mask\~Format : |
| 969 |
.P |
| 970 |
.in +4n |
| 971 |
.EX |
| 972 |
00000001 # just bit 0 set |
| 973 |
40000000,00000000,00000000 # just bit 94 set |
| 974 |
00000001,00000000,00000000 # just bit 64 set |
| 975 |
000000ff,00000000 # bits 32\-39 set |
| 976 |
00000000,000e3862 # 1,5,6,11\-13,17\-19 set |
| 977 |
.EE |
| 978 |
.in |
| 979 |
.P |
| 980 |
A mask with bits 0, 1, 2, 4, 8, 16, 32, and 64 set displays as: |
| 981 |
.P |
| 982 |
.in +4n |
| 983 |
.EX |
| 984 |
00000001,00000001,00010117 |
| 985 |
.EE |
| 986 |
.in |
| 987 |
.P |
| 988 |
The first "1" is for bit 64, the |
| 989 |
second for bit 32, the third for bit 16, the fourth for bit 8, the |
| 990 |
fifth for bit 4, and the "7" is for bits 2, 1, and 0. |
| 991 |
.\" ================== List Format ================== |
| 992 |
.SS List format |
| 993 |
The List Format for |
| 994 |
.I cpus |
| 995 |
and |
| 996 |
.I mems |
| 997 |
is a comma-separated list of CPU or memory-node |
| 998 |
numbers and ranges of numbers, in ASCII decimal. |
| 999 |
.P |
| 1000 |
Examples of the |
| 1001 |
.BR List\~Format : |
| 1002 |
.P |
| 1003 |
.in +4n |
| 1004 |
.EX |
| 1005 |
0\-4,9 # bits 0, 1, 2, 3, 4, and 9 set |
| 1006 |
0\-2,7,12\-14 # bits 0, 1, 2, 7, 12, 13, and 14 set |
| 1007 |
.EE |
| 1008 |
.in |
| 1009 |
.\" ================== RULES ================== |
| 1010 |
.SH RULES |
| 1011 |
The following rules apply to each cpuset: |
| 1012 |
.IP \[bu] 3 |
| 1013 |
Its CPUs and memory nodes must be a (possibly equal) |
| 1014 |
subset of its parent's. |
| 1015 |
.IP \[bu] |
| 1016 |
It can be marked |
| 1017 |
.I cpu_exclusive |
| 1018 |
only if its parent is. |
| 1019 |
.IP \[bu] |
| 1020 |
It can be marked |
| 1021 |
.I mem_exclusive |
| 1022 |
only if its parent is. |
| 1023 |
.IP \[bu] |
| 1024 |
If it is |
| 1025 |
.IR cpu_exclusive , |
| 1026 |
its CPUs may not overlap any sibling. |
| 1027 |
.IP \[bu] |
| 1028 |
If it is |
| 1029 |
.IR mem_exclusive , |
| 1030 |
its memory nodes may not overlap any sibling. |
| 1031 |
.\" ================== PERMISSIONS ================== |
| 1032 |
.SH PERMISSIONS |
| 1033 |
The permissions of a cpuset are determined by the permissions |
| 1034 |
of the directories and pseudo-files in the cpuset filesystem, |
| 1035 |
normally mounted at |
| 1036 |
.IR /dev/cpuset . |
| 1037 |
.P |
| 1038 |
For instance, a process can put itself in some other cpuset (than |
| 1039 |
its current one) if it can write the |
| 1040 |
.I tasks |
| 1041 |
file for that cpuset. |
| 1042 |
This requires execute permission on the encompassing directories |
| 1043 |
and write permission on the |
| 1044 |
.I tasks |
| 1045 |
file. |
| 1046 |
.P |
| 1047 |
An additional constraint is applied to requests to place some |
| 1048 |
other process in a cpuset. |
| 1049 |
One process may not attach another to |
| 1050 |
a cpuset unless it would have permission to send that process |
| 1051 |
a signal (see |
| 1052 |
.BR kill (2)). |
| 1053 |
.P |
| 1054 |
A process may create a child cpuset if it can access and write the |
| 1055 |
parent cpuset directory. |
| 1056 |
It can modify the CPUs or memory nodes |
| 1057 |
in a cpuset if it can access that cpuset's directory (execute |
| 1058 |
permissions on the each of the parent directories) and write the |
| 1059 |
corresponding |
| 1060 |
.I cpus |
| 1061 |
or |
| 1062 |
.I mems |
| 1063 |
file. |
| 1064 |
.P |
| 1065 |
There is one minor difference between the manner in which these |
| 1066 |
permissions are evaluated and the manner in which normal filesystem |
| 1067 |
operation permissions are evaluated. |
| 1068 |
The kernel interprets |
| 1069 |
relative pathnames starting at a process's current working directory. |
| 1070 |
Even if one is operating on a cpuset file, relative pathnames |
| 1071 |
are interpreted relative to the process's current working directory, |
| 1072 |
not relative to the process's current cpuset. |
| 1073 |
The only ways that |
| 1074 |
cpuset paths relative to a process's current cpuset can be used are |
| 1075 |
if either the process's current working directory is its cpuset |
| 1076 |
(it first did a |
| 1077 |
.B cd |
| 1078 |
or |
| 1079 |
.BR chdir (2) |
| 1080 |
to its cpuset directory beneath |
| 1081 |
.IR /dev/cpuset , |
| 1082 |
which is a bit unusual) |
| 1083 |
or if some user code converts the relative cpuset path to a |
| 1084 |
full filesystem path. |
| 1085 |
.P |
| 1086 |
In theory, this means that user code should specify cpusets |
| 1087 |
using absolute pathnames, which requires knowing the mount point of |
| 1088 |
the cpuset filesystem (usually, but not necessarily, |
| 1089 |
.IR /dev/cpuset ). |
| 1090 |
In practice, all user level code that this author is aware of |
| 1091 |
simply assumes that if the cpuset filesystem is mounted, then |
| 1092 |
it is mounted at |
| 1093 |
.IR /dev/cpuset . |
| 1094 |
Furthermore, it is common practice for carefully written |
| 1095 |
user code to verify the presence of the pseudo-file |
| 1096 |
.I /dev/cpuset/tasks |
| 1097 |
in order to verify that the cpuset pseudo-filesystem |
| 1098 |
is currently mounted. |
| 1099 |
.\" ================== WARNINGS ================== |
| 1100 |
.SH WARNINGS |
| 1101 |
.SS Enabling memory_pressure |
| 1102 |
By default, the per-cpuset file |
| 1103 |
.I cpuset.memory_pressure |
| 1104 |
always contains zero (0). |
| 1105 |
Unless this feature is enabled by writing "1" to the pseudo-file |
| 1106 |
.IR /dev/cpuset/cpuset.memory_pressure_enabled , |
| 1107 |
the kernel does |
| 1108 |
not compute per-cpuset |
| 1109 |
.IR memory_pressure . |
| 1110 |
.SS Using the echo command |
| 1111 |
When using the |
| 1112 |
.B echo |
| 1113 |
command at the shell prompt to change the values of cpuset files, |
| 1114 |
beware that the built-in |
| 1115 |
.B echo |
| 1116 |
command in some shells does not display an error message if the |
| 1117 |
.BR write (2) |
| 1118 |
system call fails. |
| 1119 |
.\" Gack! csh(1)'s echo does this |
| 1120 |
For example, if the command: |
| 1121 |
.P |
| 1122 |
.in +4n |
| 1123 |
.EX |
| 1124 |
echo 19 > cpuset.mems |
| 1125 |
.EE |
| 1126 |
.in |
| 1127 |
.P |
| 1128 |
failed because memory node 19 was not allowed (perhaps |
| 1129 |
the current system does not have a memory node 19), then the |
| 1130 |
.B echo |
| 1131 |
command might not display any error. |
| 1132 |
It is better to use the |
| 1133 |
.B /bin/echo |
| 1134 |
external command to change cpuset file settings, as this |
| 1135 |
command will display |
| 1136 |
.BR write (2) |
| 1137 |
errors, as in the example: |
| 1138 |
.P |
| 1139 |
.in +4n |
| 1140 |
.EX |
| 1141 |
/bin/echo 19 > cpuset.mems |
| 1142 |
/bin/echo: write error: Invalid argument |
| 1143 |
.EE |
| 1144 |
.in |
| 1145 |
.\" ================== EXCEPTIONS ================== |
| 1146 |
.SH EXCEPTIONS |
| 1147 |
.SS Memory placement |
| 1148 |
Not all allocations of system memory are constrained by cpusets, |
| 1149 |
for the following reasons. |
| 1150 |
.P |
| 1151 |
If hot-plug functionality is used to remove all the CPUs that are |
| 1152 |
currently assigned to a cpuset, |
| 1153 |
then the kernel will automatically |
| 1154 |
update the |
| 1155 |
.I cpus_allowed |
| 1156 |
of all processes attached to CPUs in that cpuset |
| 1157 |
to allow all CPUs. |
| 1158 |
When memory hot-plug functionality for removing memory nodes is available, |
| 1159 |
a similar exception is expected to apply there as well. |
| 1160 |
In general, |
| 1161 |
the kernel prefers to violate cpuset placement, |
| 1162 |
rather than starving a process that has had all its allowed CPUs or |
| 1163 |
memory nodes taken offline. |
| 1164 |
User code should reconfigure cpusets to refer only to online CPUs |
| 1165 |
and memory nodes when using hot-plug to add or remove such resources. |
| 1166 |
.P |
| 1167 |
A few kernel-critical, |
| 1168 |
internal memory-allocation requests, |
| 1169 |
marked GFP_ATOMIC, |
| 1170 |
must be satisfied immediately. |
| 1171 |
The kernel may drop some request or malfunction |
| 1172 |
if one of these allocations fail. |
| 1173 |
If such a request cannot be satisfied within the current process's cpuset, |
| 1174 |
then we relax the cpuset, |
| 1175 |
and look for memory anywhere we can find it. |
| 1176 |
It's better to violate the cpuset than stress the kernel. |
| 1177 |
.P |
| 1178 |
Allocations of memory requested by kernel drivers |
| 1179 |
while processing an interrupt |
| 1180 |
lack any relevant process context, |
| 1181 |
and are not confined by cpusets. |
| 1182 |
.SS Renaming cpusets |
| 1183 |
You can use the |
| 1184 |
.BR rename (2) |
| 1185 |
system call to rename cpusets. |
| 1186 |
Only simple renaming is supported; |
| 1187 |
that is, |
| 1188 |
changing the name of a cpuset directory is permitted, |
| 1189 |
but moving a directory into a different directory is not permitted. |
| 1190 |
.\" ================== ERRORS ================== |
| 1191 |
.SH ERRORS |
| 1192 |
The Linux kernel implementation of cpusets sets |
| 1193 |
.I errno |
| 1194 |
to specify the reason for a failed system call affecting cpusets. |
| 1195 |
.P |
| 1196 |
The possible |
| 1197 |
.I errno |
| 1198 |
settings and their meaning when set on |
| 1199 |
a failed cpuset call are as listed below. |
| 1200 |
.TP |
| 1201 |
.B E2BIG |
| 1202 |
Attempted a |
| 1203 |
.BR write (2) |
| 1204 |
on a special cpuset file |
| 1205 |
with a length larger than some kernel-determined upper |
| 1206 |
limit on the length of such writes. |
| 1207 |
.TP |
| 1208 |
.B EACCES |
| 1209 |
Attempted to |
| 1210 |
.BR write (2) |
| 1211 |
the process ID (PID) of a process to a cpuset |
| 1212 |
.I tasks |
| 1213 |
file when one lacks permission to move that process. |
| 1214 |
.TP |
| 1215 |
.B EACCES |
| 1216 |
Attempted to add, |
| 1217 |
using |
| 1218 |
.BR write (2), |
| 1219 |
a CPU or memory node to a cpuset, |
| 1220 |
when that CPU or memory node |
| 1221 |
was not already in its parent. |
| 1222 |
.TP |
| 1223 |
.B EACCES |
| 1224 |
Attempted to set, using |
| 1225 |
.BR write (2), |
| 1226 |
.I cpuset.cpu_exclusive |
| 1227 |
or |
| 1228 |
.I cpuset.mem_exclusive |
| 1229 |
on a cpuset whose parent lacks the same setting. |
| 1230 |
.TP |
| 1231 |
.B EACCES |
| 1232 |
Attempted to |
| 1233 |
.BR write (2) |
| 1234 |
a |
| 1235 |
.I cpuset.memory_pressure |
| 1236 |
file. |
| 1237 |
.TP |
| 1238 |
.B EACCES |
| 1239 |
Attempted to create a file in a cpuset directory. |
| 1240 |
.TP |
| 1241 |
.B EBUSY |
| 1242 |
Attempted to remove, using |
| 1243 |
.BR rmdir (2), |
| 1244 |
a cpuset with attached processes. |
| 1245 |
.TP |
| 1246 |
.B EBUSY |
| 1247 |
Attempted to remove, using |
| 1248 |
.BR rmdir (2), |
| 1249 |
a cpuset with child cpusets. |
| 1250 |
.TP |
| 1251 |
.B EBUSY |
| 1252 |
Attempted to remove |
| 1253 |
a CPU or memory node from a cpuset |
| 1254 |
that is also in a child of that cpuset. |
| 1255 |
.TP |
| 1256 |
.B EEXIST |
| 1257 |
Attempted to create, using |
| 1258 |
.BR mkdir (2), |
| 1259 |
a cpuset that already exists. |
| 1260 |
.TP |
| 1261 |
.B EEXIST |
| 1262 |
Attempted to |
| 1263 |
.BR rename (2) |
| 1264 |
a cpuset to a name that already exists. |
| 1265 |
.TP |
| 1266 |
.B EFAULT |
| 1267 |
Attempted to |
| 1268 |
.BR read (2) |
| 1269 |
or |
| 1270 |
.BR write (2) |
| 1271 |
a cpuset file using |
| 1272 |
a buffer that is outside the writing processes accessible address space. |
| 1273 |
.TP |
| 1274 |
.B EINVAL |
| 1275 |
Attempted to change a cpuset, using |
| 1276 |
.BR write (2), |
| 1277 |
in a way that would violate a |
| 1278 |
.I cpu_exclusive |
| 1279 |
or |
| 1280 |
.I mem_exclusive |
| 1281 |
attribute of that cpuset or any of its siblings. |
| 1282 |
.TP |
| 1283 |
.B EINVAL |
| 1284 |
Attempted to |
| 1285 |
.BR write (2) |
| 1286 |
an empty |
| 1287 |
.I cpuset.cpus |
| 1288 |
or |
| 1289 |
.I cpuset.mems |
| 1290 |
list to a cpuset which has attached processes or child cpusets. |
| 1291 |
.TP |
| 1292 |
.B EINVAL |
| 1293 |
Attempted to |
| 1294 |
.BR write (2) |
| 1295 |
a |
| 1296 |
.I cpuset.cpus |
| 1297 |
or |
| 1298 |
.I cpuset.mems |
| 1299 |
list which included a range with the second number smaller than |
| 1300 |
the first number. |
| 1301 |
.TP |
| 1302 |
.B EINVAL |
| 1303 |
Attempted to |
| 1304 |
.BR write (2) |
| 1305 |
a |
| 1306 |
.I cpuset.cpus |
| 1307 |
or |
| 1308 |
.I cpuset.mems |
| 1309 |
list which included an invalid character in the string. |
| 1310 |
.TP |
| 1311 |
.B EINVAL |
| 1312 |
Attempted to |
| 1313 |
.BR write (2) |
| 1314 |
a list to a |
| 1315 |
.I cpuset.cpus |
| 1316 |
file that did not include any online CPUs. |
| 1317 |
.TP |
| 1318 |
.B EINVAL |
| 1319 |
Attempted to |
| 1320 |
.BR write (2) |
| 1321 |
a list to a |
| 1322 |
.I cpuset.mems |
| 1323 |
file that did not include any online memory nodes. |
| 1324 |
.TP |
| 1325 |
.B EINVAL |
| 1326 |
Attempted to |
| 1327 |
.BR write (2) |
| 1328 |
a list to a |
| 1329 |
.I cpuset.mems |
| 1330 |
file that included a node that held no memory. |
| 1331 |
.TP |
| 1332 |
.B EIO |
| 1333 |
Attempted to |
| 1334 |
.BR write (2) |
| 1335 |
a string to a cpuset |
| 1336 |
.I tasks |
| 1337 |
file that |
| 1338 |
does not begin with an ASCII decimal integer. |
| 1339 |
.TP |
| 1340 |
.B EIO |
| 1341 |
Attempted to |
| 1342 |
.BR rename (2) |
| 1343 |
a cpuset into a different directory. |
| 1344 |
.TP |
| 1345 |
.B ENAMETOOLONG |
| 1346 |
Attempted to |
| 1347 |
.BR read (2) |
| 1348 |
a |
| 1349 |
.IR /proc/ pid /cpuset |
| 1350 |
file for a cpuset path that is longer than the kernel page size. |
| 1351 |
.TP |
| 1352 |
.B ENAMETOOLONG |
| 1353 |
Attempted to create, using |
| 1354 |
.BR mkdir (2), |
| 1355 |
a cpuset whose base directory name is longer than 255 characters. |
| 1356 |
.TP |
| 1357 |
.B ENAMETOOLONG |
| 1358 |
Attempted to create, using |
| 1359 |
.BR mkdir (2), |
| 1360 |
a cpuset whose full pathname, |
| 1361 |
including the mount point (typically "/dev/cpuset/") prefix, |
| 1362 |
is longer than 4095 characters. |
| 1363 |
.TP |
| 1364 |
.B ENODEV |
| 1365 |
The cpuset was removed by another process at the same time as a |
| 1366 |
.BR write (2) |
| 1367 |
was attempted on one of the pseudo-files in the cpuset directory. |
| 1368 |
.TP |
| 1369 |
.B ENOENT |
| 1370 |
Attempted to create, using |
| 1371 |
.BR mkdir (2), |
| 1372 |
a cpuset in a parent cpuset that doesn't exist. |
| 1373 |
.TP |
| 1374 |
.B ENOENT |
| 1375 |
Attempted to |
| 1376 |
.BR access (2) |
| 1377 |
or |
| 1378 |
.BR open (2) |
| 1379 |
a nonexistent file in a cpuset directory. |
| 1380 |
.TP |
| 1381 |
.B ENOMEM |
| 1382 |
Insufficient memory is available within the kernel; |
| 1383 |
can occur on a variety of system calls affecting cpusets, |
| 1384 |
but only if the system is extremely short of memory. |
| 1385 |
.TP |
| 1386 |
.B ENOSPC |
| 1387 |
Attempted to |
| 1388 |
.BR write (2) |
| 1389 |
the process ID (PID) |
| 1390 |
of a process to a cpuset |
| 1391 |
.I tasks |
| 1392 |
file when the cpuset had an empty |
| 1393 |
.I cpuset.cpus |
| 1394 |
or empty |
| 1395 |
.I cpuset.mems |
| 1396 |
setting. |
| 1397 |
.TP |
| 1398 |
.B ENOSPC |
| 1399 |
Attempted to |
| 1400 |
.BR write (2) |
| 1401 |
an empty |
| 1402 |
.I cpuset.cpus |
| 1403 |
or |
| 1404 |
.I cpuset.mems |
| 1405 |
setting to a cpuset that |
| 1406 |
has tasks attached. |
| 1407 |
.TP |
| 1408 |
.B ENOTDIR |
| 1409 |
Attempted to |
| 1410 |
.BR rename (2) |
| 1411 |
a nonexistent cpuset. |
| 1412 |
.TP |
| 1413 |
.B EPERM |
| 1414 |
Attempted to remove a file from a cpuset directory. |
| 1415 |
.TP |
| 1416 |
.B ERANGE |
| 1417 |
Specified a |
| 1418 |
.I cpuset.cpus |
| 1419 |
or |
| 1420 |
.I cpuset.mems |
| 1421 |
list to the kernel which included a number too large for the kernel |
| 1422 |
to set in its bit masks. |
| 1423 |
.TP |
| 1424 |
.B ESRCH |
| 1425 |
Attempted to |
| 1426 |
.BR write (2) |
| 1427 |
the process ID (PID) of a nonexistent process to a cpuset |
| 1428 |
.I tasks |
| 1429 |
file. |
| 1430 |
.\" ================== VERSIONS ================== |
| 1431 |
.SH VERSIONS |
| 1432 |
Cpusets appeared in Linux 2.6.12. |
| 1433 |
.\" ================== NOTES ================== |
| 1434 |
.SH NOTES |
| 1435 |
Despite its name, the |
| 1436 |
.I pid |
| 1437 |
parameter is actually a thread ID, |
| 1438 |
and each thread in a threaded group can be attached to a different |
| 1439 |
cpuset. |
| 1440 |
The value returned from a call to |
| 1441 |
.BR gettid (2) |
| 1442 |
can be passed in the argument |
| 1443 |
.IR pid . |
| 1444 |
.\" ================== BUGS ================== |
| 1445 |
.SH BUGS |
| 1446 |
.I cpuset.memory_pressure |
| 1447 |
cpuset files can be opened |
| 1448 |
for writing, creation, or truncation, but then the |
| 1449 |
.BR write (2) |
| 1450 |
fails with |
| 1451 |
.I errno |
| 1452 |
set to |
| 1453 |
.BR EACCES , |
| 1454 |
and the creation and truncation options on |
| 1455 |
.BR open (2) |
| 1456 |
have no effect. |
| 1457 |
.\" ================== EXAMPLES ================== |
| 1458 |
.SH EXAMPLES |
| 1459 |
The following examples demonstrate querying and setting cpuset |
| 1460 |
options using shell commands. |
| 1461 |
.SS Creating and attaching to a cpuset. |
| 1462 |
To create a new cpuset and attach the current command shell to it, |
| 1463 |
the steps are: |
| 1464 |
.P |
| 1465 |
.PD 0 |
| 1466 |
.IP (1) 5 |
| 1467 |
mkdir /dev/cpuset (if not already done) |
| 1468 |
.IP (2) |
| 1469 |
mount \-t cpuset none /dev/cpuset (if not already done) |
| 1470 |
.IP (3) |
| 1471 |
Create the new cpuset using |
| 1472 |
.BR mkdir (1). |
| 1473 |
.IP (4) |
| 1474 |
Assign CPUs and memory nodes to the new cpuset. |
| 1475 |
.IP (5) |
| 1476 |
Attach the shell to the new cpuset. |
| 1477 |
.PD |
| 1478 |
.P |
| 1479 |
For example, the following sequence of commands will set up a cpuset |
| 1480 |
named "Charlie", containing just CPUs 2 and 3, and memory node 1, |
| 1481 |
and then attach the current shell to that cpuset. |
| 1482 |
.P |
| 1483 |
.in +4n |
| 1484 |
.EX |
| 1485 |
.RB "$" " mkdir /dev/cpuset" |
| 1486 |
.RB "$" " mount \-t cpuset cpuset /dev/cpuset" |
| 1487 |
.RB "$" " cd /dev/cpuset" |
| 1488 |
.RB "$" " mkdir Charlie" |
| 1489 |
.RB "$" " cd Charlie" |
| 1490 |
.RB "$" " /bin/echo 2\-3 > cpuset.cpus" |
| 1491 |
.RB "$" " /bin/echo 1 > cpuset.mems" |
| 1492 |
.RB "$" " /bin/echo $$ > tasks" |
| 1493 |
# The current shell is now running in cpuset Charlie |
| 1494 |
# The next line should display \[aq]/Charlie\[aq] |
| 1495 |
.RB "$" " cat /proc/self/cpuset" |
| 1496 |
.EE |
| 1497 |
.in |
| 1498 |
.\" |
| 1499 |
.SS Migrating a job to different memory nodes. |
| 1500 |
To migrate a job (the set of processes attached to a cpuset) |
| 1501 |
to different CPUs and memory nodes in the system, including moving |
| 1502 |
the memory pages currently allocated to that job, |
| 1503 |
perform the following steps. |
| 1504 |
.P |
| 1505 |
.PD 0 |
| 1506 |
.IP (1) 5 |
| 1507 |
Let's say we want to move the job in cpuset |
| 1508 |
.I alpha |
| 1509 |
(CPUs 4\[en]7 and memory nodes 2\[en]3) to a new cpuset |
| 1510 |
.I beta |
| 1511 |
(CPUs 16\[en]19 and memory nodes 8\[en]9). |
| 1512 |
.IP (2) |
| 1513 |
First create the new cpuset |
| 1514 |
.IR beta . |
| 1515 |
.IP (3) |
| 1516 |
Then allow CPUs 16\[en]19 and memory nodes 8\[en]9 in |
| 1517 |
.IR beta . |
| 1518 |
.IP (4) |
| 1519 |
Then enable |
| 1520 |
.I memory_migration |
| 1521 |
in |
| 1522 |
.IR beta . |
| 1523 |
.IP (5) |
| 1524 |
Then move each process from |
| 1525 |
.I alpha |
| 1526 |
to |
| 1527 |
.IR beta . |
| 1528 |
.PD |
| 1529 |
.P |
| 1530 |
The following sequence of commands accomplishes this. |
| 1531 |
.P |
| 1532 |
.in +4n |
| 1533 |
.EX |
| 1534 |
.RB "$" " cd /dev/cpuset" |
| 1535 |
.RB "$" " mkdir beta" |
| 1536 |
.RB "$" " cd beta" |
| 1537 |
.RB "$" " /bin/echo 16\-19 > cpuset.cpus" |
| 1538 |
.RB "$" " /bin/echo 8\-9 > cpuset.mems" |
| 1539 |
.RB "$" " /bin/echo 1 > cpuset.memory_migrate" |
| 1540 |
.RB "$" " while read i; do /bin/echo $i; done < ../alpha/tasks > tasks" |
| 1541 |
.EE |
| 1542 |
.in |
| 1543 |
.P |
| 1544 |
The above should move any processes in |
| 1545 |
.I alpha |
| 1546 |
to |
| 1547 |
.IR beta , |
| 1548 |
and any memory held by these processes on memory nodes 2\[en]3 to memory |
| 1549 |
nodes 8\[en]9, respectively. |
| 1550 |
.P |
| 1551 |
Notice that the last step of the above sequence did not do: |
| 1552 |
.P |
| 1553 |
.in +4n |
| 1554 |
.EX |
| 1555 |
.RB "$" " cp ../alpha/tasks tasks" |
| 1556 |
.EE |
| 1557 |
.in |
| 1558 |
.P |
| 1559 |
The |
| 1560 |
.I while |
| 1561 |
loop, rather than the seemingly easier use of the |
| 1562 |
.BR cp (1) |
| 1563 |
command, was necessary because |
| 1564 |
only one process PID at a time may be written to the |
| 1565 |
.I tasks |
| 1566 |
file. |
| 1567 |
.P |
| 1568 |
The same effect (writing one PID at a time) as the |
| 1569 |
.I while |
| 1570 |
loop can be accomplished more efficiently, in fewer keystrokes and in |
| 1571 |
syntax that works on any shell, but alas more obscurely, by using the |
| 1572 |
.B \-u |
| 1573 |
(unbuffered) option of |
| 1574 |
.BR sed (1): |
| 1575 |
.P |
| 1576 |
.in +4n |
| 1577 |
.EX |
| 1578 |
.RB "$" " sed \-un p < ../alpha/tasks > tasks" |
| 1579 |
.EE |
| 1580 |
.in |
| 1581 |
.\" ================== SEE ALSO ================== |
| 1582 |
.SH SEE ALSO |
| 1583 |
.BR taskset (1), |
| 1584 |
.BR get_mempolicy (2), |
| 1585 |
.BR getcpu (2), |
| 1586 |
.BR mbind (2), |
| 1587 |
.BR sched_getaffinity (2), |
| 1588 |
.BR sched_setaffinity (2), |
| 1589 |
.BR sched_setscheduler (2), |
| 1590 |
.BR set_mempolicy (2), |
| 1591 |
.BR CPU_SET (3), |
| 1592 |
.BR proc (5), |
| 1593 |
.BR cgroups (7), |
| 1594 |
.BR numa (7), |
| 1595 |
.BR sched (7), |
| 1596 |
.BR migratepages (8), |
| 1597 |
.BR numactl (8) |
| 1598 |
.P |
| 1599 |
.I Documentation/admin\-guide/cgroup\-v1/cpusets.rst |
| 1600 |
in the Linux kernel source tree |