| 1 |
.\" Copyright 2016-2017, Michael Kerrisk <[email protected]> |
| 2 |
.\" Copyright, the authors of the Linux man-pages project |
| 3 |
.\" |
| 4 |
.\" SPDX-License-Identifier: Linux-man-pages-copyleft |
| 5 |
.\" |
| 6 |
.TH cgroups 7 2026-02-08 "Linux man-pages 6.18" |
| 7 |
.SH NAME |
| 8 |
cgroups \- Linux control groups |
| 9 |
.SH DESCRIPTION |
| 10 |
Control groups, usually referred to as cgroups, |
| 11 |
are a Linux kernel feature which allow processes to |
| 12 |
be organized into hierarchical groups whose usage of |
| 13 |
various types of resources can then be limited and monitored. |
| 14 |
The kernel's cgroup interface is provided through |
| 15 |
a pseudo-filesystem called cgroupfs. |
| 16 |
Grouping is implemented in the core cgroup kernel code, |
| 17 |
while resource tracking and limits are implemented in |
| 18 |
a set of per-resource-type subsystems (memory, CPU, and so on). |
| 19 |
.\" |
| 20 |
.SS Terminology |
| 21 |
A |
| 22 |
.I cgroup |
| 23 |
is a collection of processes that are bound to a set of |
| 24 |
limits or parameters defined via the cgroup filesystem. |
| 25 |
.P |
| 26 |
A |
| 27 |
.I subsystem |
| 28 |
is a kernel component that modifies the behavior of |
| 29 |
the processes in a cgroup. |
| 30 |
Various subsystems have been implemented, making it possible to do things |
| 31 |
such as limiting the amount of CPU time and memory available to a cgroup, |
| 32 |
accounting for the CPU time used by a cgroup, |
| 33 |
and freezing and resuming execution of the processes in a cgroup. |
| 34 |
Subsystems are sometimes also known as |
| 35 |
.I resource controllers |
| 36 |
(or simply, controllers). |
| 37 |
.P |
| 38 |
The cgroups for a controller are arranged in a |
| 39 |
.IR hierarchy . |
| 40 |
This hierarchy is defined by creating, removing, and |
| 41 |
renaming subdirectories within the cgroup filesystem. |
| 42 |
At each level of the hierarchy, attributes (e.g., limits) can be defined. |
| 43 |
The limits, control, and accounting provided by cgroups generally have |
| 44 |
effect throughout the subhierarchy underneath the cgroup where the |
| 45 |
attributes are defined. |
| 46 |
Thus, for example, the limits placed on |
| 47 |
a cgroup at a higher level in the hierarchy cannot be exceeded |
| 48 |
by descendant cgroups. |
| 49 |
.\" |
| 50 |
.SS Cgroups version 1 and version 2 |
| 51 |
The initial release of the cgroups implementation was in Linux 2.6.24. |
| 52 |
Over time, various cgroup controllers have been added |
| 53 |
to allow the management of various types of resources. |
| 54 |
However, the development of these controllers was largely uncoordinated, |
| 55 |
with the result that many inconsistencies arose between controllers |
| 56 |
and management of the cgroup hierarchies became rather complex. |
| 57 |
A longer description of these problems can be found in the kernel |
| 58 |
source file |
| 59 |
.I Documentation/admin\-guide/cgroup\-v2.rst |
| 60 |
(or |
| 61 |
.I Documentation/cgroup\-v2.rst |
| 62 |
in Linux 4.17 and earlier). |
| 63 |
.P |
| 64 |
Because of the problems with the initial cgroups implementation |
| 65 |
(cgroups version 1), |
| 66 |
starting in Linux 3.10, work began on a new, |
| 67 |
orthogonal implementation to remedy these problems. |
| 68 |
Initially marked experimental, and hidden behind the |
| 69 |
.I "\-o\ __DEVEL__sane_behavior" |
| 70 |
mount option, the new version (cgroups version 2) |
| 71 |
was eventually made official with the release of Linux 4.5. |
| 72 |
Differences between the two versions are described in the text below. |
| 73 |
The file |
| 74 |
.IR cgroup.sane_behavior , |
| 75 |
present in cgroups v1, is a relic of this mount option. |
| 76 |
The file always reports "0" and is only retained for backward compatibility. |
| 77 |
.P |
| 78 |
Although cgroups v2 is intended as a replacement for cgroups v1, |
| 79 |
the older system continues to exist |
| 80 |
(and for compatibility reasons is unlikely to be removed). |
| 81 |
Currently, cgroups v2 implements only a subset of the controllers |
| 82 |
available in cgroups v1. |
| 83 |
The two systems are implemented so that both v1 controllers and |
| 84 |
v2 controllers can be mounted on the same system. |
| 85 |
Thus, for example, it is possible to use those controllers |
| 86 |
that are supported under version 2, |
| 87 |
while also using version 1 controllers |
| 88 |
where version 2 does not yet support those controllers. |
| 89 |
The only restriction here is that a controller can't be simultaneously |
| 90 |
employed in both a cgroups v1 hierarchy and in the cgroups v2 hierarchy. |
| 91 |
.\" |
| 92 |
.SH CGROUPS VERSION 1 |
| 93 |
Under cgroups v1, each controller may be mounted against a separate |
| 94 |
cgroup filesystem that provides its own hierarchical organization of the |
| 95 |
processes on the system. |
| 96 |
It is also possible to comount multiple (or even all) cgroups v1 controllers |
| 97 |
against the same cgroup filesystem, meaning that the comounted controllers |
| 98 |
manage the same hierarchical organization of processes. |
| 99 |
.P |
| 100 |
For each mounted hierarchy, |
| 101 |
the directory tree mirrors the control group hierarchy. |
| 102 |
Each control group is represented by a directory, with each of its child |
| 103 |
control cgroups represented as a child directory. |
| 104 |
For instance, |
| 105 |
.I /user/joe/1.session |
| 106 |
represents control group |
| 107 |
.IR 1.session , |
| 108 |
which is a child of cgroup |
| 109 |
.IR joe , |
| 110 |
which is a child of |
| 111 |
.IR /user . |
| 112 |
Under each cgroup directory is a set of files which can be read or |
| 113 |
written to, reflecting resource limits and a few general cgroup |
| 114 |
properties. |
| 115 |
.\" |
| 116 |
.SS Tasks (threads) versus processes |
| 117 |
In cgroups v1, a distinction is drawn between |
| 118 |
.I processes |
| 119 |
and |
| 120 |
.IR tasks . |
| 121 |
In this view, a process can consist of multiple tasks |
| 122 |
(more commonly called threads, from a user-space perspective, |
| 123 |
and called such in the remainder of this man page). |
| 124 |
In cgroups v1, it is possible to independently manipulate |
| 125 |
the cgroup memberships of the threads in a process. |
| 126 |
.P |
| 127 |
The cgroups v1 ability to split threads across different cgroups |
| 128 |
caused problems in some cases. |
| 129 |
For example, it made no sense for the |
| 130 |
.I memory |
| 131 |
controller, |
| 132 |
since all of the threads of a process share a single address space. |
| 133 |
Because of these problems, |
| 134 |
the ability to independently manipulate the cgroup memberships |
| 135 |
of the threads in a process was removed in the initial cgroups v2 |
| 136 |
implementation, and subsequently restored in a more limited form |
| 137 |
(see the discussion of "thread mode" below). |
| 138 |
.\" |
| 139 |
.SS Mounting v1 controllers |
| 140 |
The use of cgroups requires a kernel built with the |
| 141 |
.B CONFIG_CGROUP |
| 142 |
option. |
| 143 |
In addition, each of the v1 controllers has an associated |
| 144 |
configuration option that must be set in order to employ that controller. |
| 145 |
.P |
| 146 |
In order to use a v1 controller, |
| 147 |
it must be mounted against a cgroup filesystem. |
| 148 |
The usual place for such mounts is under a |
| 149 |
.BR tmpfs (5) |
| 150 |
filesystem mounted at |
| 151 |
.IR /sys/fs/cgroup . |
| 152 |
Thus, one might mount the |
| 153 |
.I cpu |
| 154 |
controller as follows: |
| 155 |
.P |
| 156 |
.in +4n |
| 157 |
.EX |
| 158 |
mount \-t cgroup \-o cpu none /sys/fs/cgroup/cpu |
| 159 |
.EE |
| 160 |
.in |
| 161 |
.P |
| 162 |
It is possible to comount multiple controllers against the same hierarchy. |
| 163 |
For example, here the |
| 164 |
.I cpu |
| 165 |
and |
| 166 |
.I cpuacct |
| 167 |
controllers are comounted against a single hierarchy: |
| 168 |
.P |
| 169 |
.in +4n |
| 170 |
.EX |
| 171 |
mount \-t cgroup \-o cpu,cpuacct none /sys/fs/cgroup/cpu,cpuacct |
| 172 |
.EE |
| 173 |
.in |
| 174 |
.P |
| 175 |
Comounting controllers has the effect that a process is in the same cgroup for |
| 176 |
all of the comounted controllers. |
| 177 |
Separately mounting controllers allows a process to |
| 178 |
be in cgroup |
| 179 |
.I /foo1 |
| 180 |
for one controller while being in |
| 181 |
.I /foo2/foo3 |
| 182 |
for another. |
| 183 |
.P |
| 184 |
It is possible to comount all v1 controllers against the same hierarchy: |
| 185 |
.P |
| 186 |
.in +4n |
| 187 |
.EX |
| 188 |
mount \-t cgroup \-o all cgroup /sys/fs/cgroup |
| 189 |
.EE |
| 190 |
.in |
| 191 |
.P |
| 192 |
(One can achieve the same result by omitting |
| 193 |
.IR "\-o all" , |
| 194 |
since it is the default if no controllers are explicitly specified.) |
| 195 |
.P |
| 196 |
It is not possible to mount the same controller |
| 197 |
against multiple cgroup hierarchies. |
| 198 |
For example, it is not possible to mount both the |
| 199 |
.I cpu |
| 200 |
and |
| 201 |
.I cpuacct |
| 202 |
controllers against one hierarchy, and to mount the |
| 203 |
.I cpu |
| 204 |
controller alone against another hierarchy. |
| 205 |
It is possible to create multiple mounts with exactly |
| 206 |
the same set of comounted controllers. |
| 207 |
However, in this case all that results is multiple mount points |
| 208 |
providing a view of the same hierarchy. |
| 209 |
.P |
| 210 |
Note that on many systems, the v1 controllers are automatically mounted under |
| 211 |
.IR /sys/fs/cgroup ; |
| 212 |
in particular, |
| 213 |
.BR systemd (1) |
| 214 |
automatically creates such mounts. |
| 215 |
.\" |
| 216 |
.SS Unmounting v1 controllers |
| 217 |
A mounted cgroup filesystem can be unmounted using the |
| 218 |
.BR umount (8) |
| 219 |
command, as in the following example: |
| 220 |
.P |
| 221 |
.in +4n |
| 222 |
.EX |
| 223 |
umount /sys/fs/cgroup/pids |
| 224 |
.EE |
| 225 |
.in |
| 226 |
.P |
| 227 |
.IR "But note well" : |
| 228 |
a cgroup filesystem is unmounted only if it is not busy, |
| 229 |
that is, it has no child cgroups. |
| 230 |
If this is not the case, then the only effect of the |
| 231 |
.BR umount (8) |
| 232 |
is to make the mount invisible. |
| 233 |
Thus, to ensure that the mount is really removed, |
| 234 |
one must first remove all child cgroups, |
| 235 |
which in turn can be done only after all member processes |
| 236 |
have been moved from those cgroups to the root cgroup. |
| 237 |
.\" |
| 238 |
.SS Cgroups version 1 controllers |
| 239 |
Each of the cgroups version 1 controllers is governed |
| 240 |
by a kernel configuration option (listed below). |
| 241 |
Additionally, the availability of the cgroups feature is governed by the |
| 242 |
.B CONFIG_CGROUPS |
| 243 |
kernel configuration option. |
| 244 |
.TP |
| 245 |
.IR cpu " (since Linux 2.6.24; " \f[B]CONFIG_CGROUP_SCHED\f[] ) |
| 246 |
Cgroups can be guaranteed a minimum number of "CPU shares" |
| 247 |
when a system is busy. |
| 248 |
This does not limit a cgroup's CPU usage if the CPUs are not busy. |
| 249 |
For further information, see |
| 250 |
.IR Documentation/scheduler/sched\-design\-CFS.rst . |
| 251 |
.IP |
| 252 |
In Linux 3.2, |
| 253 |
this controller was extended to provide CPU "bandwidth" control. |
| 254 |
If the kernel is configured with |
| 255 |
.BR CONFIG_CFS_BANDWIDTH , |
| 256 |
then within each scheduling period |
| 257 |
(defined via a file in the cgroup directory), it is possible to define |
| 258 |
an upper limit on the CPU time allocated to the processes in a cgroup. |
| 259 |
This upper limit applies even if there is no other competition for the CPU. |
| 260 |
Further information can be found in the kernel source file |
| 261 |
.IR Documentation/scheduler/sched\-bwc.rst . |
| 262 |
.TP |
| 263 |
.IR cpuacct " (since Linux 2.6.24; " \f[B]CONFIG_CGROUP_CPUACCT\f[] ) |
| 264 |
This provides accounting for CPU usage by groups of processes. |
| 265 |
.IP |
| 266 |
Further information can be found in the kernel source file |
| 267 |
.IR Documentation/admin\-guide/cgroup\-v1/cpuacct.rst . |
| 268 |
.TP |
| 269 |
.IR cpuset " (since Linux 2.6.24; " \f[B]CONFIG_CPUSETS\f[] ) |
| 270 |
This cgroup can be used to bind the processes in a cgroup to |
| 271 |
a specified set of CPUs and NUMA nodes. |
| 272 |
.IP |
| 273 |
Further information can be found in the kernel source file |
| 274 |
.IR Documentation/admin\-guide/cgroup\-v1/cpusets.rst . |
| 275 |
. |
| 276 |
.TP |
| 277 |
.IR memory " (since Linux 2.6.25; " \f[B]CONFIG_MEMCG\f[] ) |
| 278 |
The memory controller supports reporting and limiting of process memory, kernel |
| 279 |
memory, and swap used by cgroups. |
| 280 |
.IP |
| 281 |
Further information can be found in the kernel source file |
| 282 |
.IR Documentation/admin\-guide/cgroup\-v1/memory.rst . |
| 283 |
.TP |
| 284 |
.IR devices " (since Linux 2.6.26; " \f[B]CONFIG_CGROUP_DEVICE\f[] ) |
| 285 |
This supports controlling which processes may create (mknod) devices as |
| 286 |
well as open them for reading or writing. |
| 287 |
The policies may be specified as allow-lists and deny-lists. |
| 288 |
Hierarchy is enforced, so new rules must not |
| 289 |
violate existing rules for the target or ancestor cgroups. |
| 290 |
.IP |
| 291 |
Further information can be found in the kernel source file |
| 292 |
.IR Documentation/admin\-guide/cgroup\-v1/devices.rst . |
| 293 |
.TP |
| 294 |
.IR freezer " (since Linux 2.6.28; " \f[B]CONFIG_CGROUP_FREEZER\f[] ) |
| 295 |
The |
| 296 |
.I freezer |
| 297 |
cgroup can suspend and restore (resume) all processes in a cgroup. |
| 298 |
Freezing a cgroup |
| 299 |
.I /A |
| 300 |
also causes its children, for example, processes in |
| 301 |
.IR /A/B , |
| 302 |
to be frozen. |
| 303 |
.IP |
| 304 |
Further information can be found in the kernel source file |
| 305 |
.IR Documentation/admin\-guide/cgroup\-v1/freezer\-subsystem.rst . |
| 306 |
.TP |
| 307 |
.IR net_cls " (since Linux 2.6.29; " \f[B]CONFIG_CGROUP_NET_CLASSID\f[] ) |
| 308 |
This places a classid, specified for the cgroup, on network packets |
| 309 |
created by a cgroup. |
| 310 |
These classids can then be used in firewall rules, |
| 311 |
as well as used to shape traffic using |
| 312 |
.BR tc (8). |
| 313 |
This applies only to packets |
| 314 |
leaving the cgroup, not to traffic arriving at the cgroup. |
| 315 |
.IP |
| 316 |
Further information can be found in the kernel source file |
| 317 |
.IR Documentation/admin\-guide/cgroup\-v1/net_cls.rst . |
| 318 |
.TP |
| 319 |
.IR blkio " (since Linux 2.6.33; " \f[B]CONFIG_BLK_CGROUP\f[] ) |
| 320 |
The |
| 321 |
.I blkio |
| 322 |
cgroup controls and limits access to specified block devices by |
| 323 |
applying IO control in the form of throttling and upper limits against leaf |
| 324 |
nodes and intermediate nodes in the storage hierarchy. |
| 325 |
.IP |
| 326 |
Two policies are available. |
| 327 |
The first is a proportional-weight time-based division |
| 328 |
of disk implemented with CFQ. |
| 329 |
This is in effect for leaf nodes using CFQ. |
| 330 |
The second is a throttling policy which specifies |
| 331 |
upper I/O rate limits on a device. |
| 332 |
.IP |
| 333 |
Further information can be found in the kernel source file |
| 334 |
.IR Documentation/admin\-guide/cgroup\-v1/blkio\-controller.rst . |
| 335 |
.TP |
| 336 |
.IR perf_event " (since Linux 2.6.39; " \f[B]CONFIG_CGROUP_PERF\f[] ) |
| 337 |
This controller allows |
| 338 |
.I perf |
| 339 |
monitoring of the set of processes grouped in a cgroup. |
| 340 |
.IP |
| 341 |
Further information can be found in the kernel source files |
| 342 |
.TP |
| 343 |
.IR net_prio " (since Linux 3.3; " \f[B]CONFIG_CGROUP_NET_PRIO\f[] ) |
| 344 |
This allows priorities to be specified, per network interface, for cgroups. |
| 345 |
.IP |
| 346 |
Further information can be found in the kernel source file |
| 347 |
.IR Documentation/admin\-guide/cgroup\-v1/net_prio.rst . |
| 348 |
.TP |
| 349 |
.IR hugetlb " (since Linux 3.5; " \f[B]CONFIG_CGROUP_HUGETLB\f[] ) |
| 350 |
This supports limiting the use of huge pages by cgroups. |
| 351 |
.IP |
| 352 |
Further information can be found in the kernel source file |
| 353 |
.IR Documentation/admin\-guide/cgroup\-v1/hugetlb.rst . |
| 354 |
.TP |
| 355 |
.IR pids " (since Linux 4.3; " \f[B]CONFIG_CGROUP_PIDS\f[] ) |
| 356 |
This controller permits limiting the number of process that may be created |
| 357 |
in a cgroup (and its descendants). |
| 358 |
.IP |
| 359 |
Further information can be found in the kernel source file |
| 360 |
.IR Documentation/admin\-guide/cgroup\-v1/pids.rst . |
| 361 |
.TP |
| 362 |
.IR rdma " (since Linux 4.11; " \f[B]CONFIG_CGROUP_RDMA\f[] ) |
| 363 |
The RDMA controller permits limiting the use of |
| 364 |
RDMA/IB-specific resources per cgroup. |
| 365 |
.IP |
| 366 |
Further information can be found in the kernel source file |
| 367 |
.IR Documentation/admin\-guide/cgroup\-v1/rdma.rst . |
| 368 |
.\" |
| 369 |
.SS Creating cgroups and moving processes |
| 370 |
A cgroup filesystem initially contains a single root cgroup, '/', |
| 371 |
which all processes belong to. |
| 372 |
A new cgroup is created by creating a directory in the cgroup filesystem: |
| 373 |
.P |
| 374 |
.in +4n |
| 375 |
.EX |
| 376 |
mkdir /sys/fs/cgroup/cpu/cg1 |
| 377 |
.EE |
| 378 |
.in |
| 379 |
.P |
| 380 |
This creates a new empty cgroup. |
| 381 |
.P |
| 382 |
A process may be moved to this cgroup by writing its PID into the cgroup's |
| 383 |
.I cgroup.procs |
| 384 |
file: |
| 385 |
.P |
| 386 |
.in +4n |
| 387 |
.EX |
| 388 |
echo $$ > /sys/fs/cgroup/cpu/cg1/cgroup.procs |
| 389 |
.EE |
| 390 |
.in |
| 391 |
.P |
| 392 |
Only one PID at a time should be written to this file. |
| 393 |
.P |
| 394 |
Writing the value 0 to a |
| 395 |
.I cgroup.procs |
| 396 |
file causes the writing process to be moved to the corresponding cgroup. |
| 397 |
.P |
| 398 |
When writing a PID into the |
| 399 |
.IR cgroup.procs , |
| 400 |
all threads in the process are moved into the new cgroup at once. |
| 401 |
.P |
| 402 |
Within a hierarchy, a process can be a member of exactly one cgroup. |
| 403 |
Writing a process's PID to a |
| 404 |
.I cgroup.procs |
| 405 |
file automatically removes it from the cgroup of |
| 406 |
which it was previously a member. |
| 407 |
.P |
| 408 |
The |
| 409 |
.I cgroup.procs |
| 410 |
file can be read to obtain a list of the processes that are |
| 411 |
members of a cgroup. |
| 412 |
The returned list of PIDs is not guaranteed to be in order. |
| 413 |
Nor is it guaranteed to be free of duplicates. |
| 414 |
(For example, a PID may be recycled while reading from the list.) |
| 415 |
.P |
| 416 |
In cgroups v1, an individual thread can be moved to |
| 417 |
another cgroup by writing its thread ID |
| 418 |
(i.e., the kernel thread ID returned by |
| 419 |
.BR clone (2) |
| 420 |
and |
| 421 |
.BR gettid (2)) |
| 422 |
to the |
| 423 |
.I tasks |
| 424 |
file in a cgroup directory. |
| 425 |
This file can be read to discover the set of threads |
| 426 |
that are members of the cgroup. |
| 427 |
.\" |
| 428 |
.SS Removing cgroups |
| 429 |
To remove a cgroup, |
| 430 |
it must first have no child cgroups and contain no (nonzombie) processes. |
| 431 |
So long as that is the case, one can simply |
| 432 |
remove the corresponding directory pathname. |
| 433 |
Note that files in a cgroup directory cannot and need not be |
| 434 |
removed. |
| 435 |
.\" |
| 436 |
.SS Cgroups v1 release notification |
| 437 |
Two files can be used to determine whether the kernel provides |
| 438 |
notifications when a cgroup becomes empty. |
| 439 |
A cgroup is considered to be empty when it contains no child |
| 440 |
cgroups and no member processes. |
| 441 |
.P |
| 442 |
A special file in the root directory of each cgroup hierarchy, |
| 443 |
.IR release_agent , |
| 444 |
can be used to register the pathname of a program that may be invoked when |
| 445 |
a cgroup in the hierarchy becomes empty. |
| 446 |
The pathname of the newly empty cgroup (relative to the cgroup mount point) |
| 447 |
is provided as the sole command-line argument when the |
| 448 |
.I release_agent |
| 449 |
program is invoked. |
| 450 |
The |
| 451 |
.I release_agent |
| 452 |
program might remove the cgroup directory, |
| 453 |
or perhaps repopulate it with a process. |
| 454 |
.P |
| 455 |
The default value of the |
| 456 |
.I release_agent |
| 457 |
file is empty, meaning that no release agent is invoked. |
| 458 |
.P |
| 459 |
The content of the |
| 460 |
.I release_agent |
| 461 |
file can also be specified via a mount option when the |
| 462 |
cgroup filesystem is mounted: |
| 463 |
.P |
| 464 |
.in +4n |
| 465 |
.EX |
| 466 |
mount \-o release_agent=pathname ... |
| 467 |
.EE |
| 468 |
.in |
| 469 |
.P |
| 470 |
Whether or not the |
| 471 |
.I release_agent |
| 472 |
program is invoked when a particular cgroup becomes empty is determined |
| 473 |
by the value in the |
| 474 |
.I notify_on_release |
| 475 |
file in the corresponding cgroup directory. |
| 476 |
If this file contains the value 0, then the |
| 477 |
.I release_agent |
| 478 |
program is not invoked. |
| 479 |
If it contains the value 1, the |
| 480 |
.I release_agent |
| 481 |
program is invoked. |
| 482 |
The default value for this file in the root cgroup is 0. |
| 483 |
At the time when a new cgroup is created, |
| 484 |
the value in this file is inherited from the corresponding file |
| 485 |
in the parent cgroup. |
| 486 |
.\" |
| 487 |
.SS Cgroup v1 named hierarchies |
| 488 |
In cgroups v1, |
| 489 |
it is possible to mount a cgroup hierarchy that has no attached controllers: |
| 490 |
.P |
| 491 |
.in +4n |
| 492 |
.EX |
| 493 |
mount \-t cgroup \-o none,name=somename none /some/mount/point |
| 494 |
.EE |
| 495 |
.in |
| 496 |
.P |
| 497 |
Multiple instances of such hierarchies can be mounted; |
| 498 |
each hierarchy must have a unique name. |
| 499 |
The only purpose of such hierarchies is to track processes. |
| 500 |
(See the discussion of release notification below.) |
| 501 |
An example of this is the |
| 502 |
.I name=systemd |
| 503 |
cgroup hierarchy that is used by |
| 504 |
.BR systemd (1) |
| 505 |
to track services and user sessions. |
| 506 |
.P |
| 507 |
Since Linux 5.0, the |
| 508 |
.I cgroup_no_v1 |
| 509 |
kernel boot option (described below) can be used to disable cgroup v1 |
| 510 |
named hierarchies, by specifying |
| 511 |
.IR cgroup_no_v1=named . |
| 512 |
.\" |
| 513 |
.SH CGROUPS VERSION 2 |
| 514 |
In cgroups v2, |
| 515 |
all mounted controllers reside in a single unified hierarchy. |
| 516 |
While (different) controllers may be simultaneously |
| 517 |
mounted under the v1 and v2 hierarchies, |
| 518 |
it is not possible to mount the same controller simultaneously |
| 519 |
under both the v1 and the v2 hierarchies. |
| 520 |
.P |
| 521 |
The new behaviors in cgroups v2 are summarized here, |
| 522 |
and in some cases elaborated in the following subsections. |
| 523 |
.IP \[bu] 3 |
| 524 |
Cgroups v2 provides a unified hierarchy against |
| 525 |
which all controllers are mounted. |
| 526 |
.IP \[bu] |
| 527 |
"Internal" processes are not permitted. |
| 528 |
With the exception of the root cgroup, processes may reside |
| 529 |
only in leaf nodes (cgroups that do not themselves contain child cgroups). |
| 530 |
The details are somewhat more subtle than this, and are described below. |
| 531 |
.IP \[bu] |
| 532 |
Active cgroups must be specified via the files |
| 533 |
.I cgroup.controllers |
| 534 |
and |
| 535 |
.IR cgroup.subtree_control . |
| 536 |
.IP \[bu] |
| 537 |
The |
| 538 |
.I tasks |
| 539 |
file has been removed. |
| 540 |
In addition, the |
| 541 |
.I cgroup.clone_children |
| 542 |
file that is employed by the |
| 543 |
.I cpuset |
| 544 |
controller has been removed. |
| 545 |
.IP \[bu] |
| 546 |
An improved mechanism for notification of empty cgroups is provided by the |
| 547 |
.I cgroup.events |
| 548 |
file. |
| 549 |
.P |
| 550 |
For more changes, see the |
| 551 |
.I Documentation/admin\-guide/cgroup\-v2.rst |
| 552 |
file in the kernel source. |
| 553 |
. |
| 554 |
.P |
| 555 |
Some of the new behaviors listed above saw subsequent modification with |
| 556 |
the addition in Linux 4.14 of "thread mode" (described below). |
| 557 |
.\" |
| 558 |
.SS Cgroups v2 unified hierarchy |
| 559 |
In cgroups v1, the ability to mount different controllers |
| 560 |
against different hierarchies was intended to allow great flexibility |
| 561 |
for application design. |
| 562 |
In practice, though, |
| 563 |
the flexibility turned out to be less useful than expected, |
| 564 |
and in many cases added complexity. |
| 565 |
Therefore, in cgroups v2, |
| 566 |
all available controllers are mounted against a single hierarchy. |
| 567 |
The available controllers are automatically mounted, |
| 568 |
meaning that it is not necessary (or possible) to specify the controllers |
| 569 |
when mounting the cgroup v2 filesystem using a command such as the following: |
| 570 |
.P |
| 571 |
.in +4n |
| 572 |
.EX |
| 573 |
mount \-t cgroup2 none /mnt/cgroup2 |
| 574 |
.EE |
| 575 |
.in |
| 576 |
.P |
| 577 |
A cgroup v2 controller is available only if it is not currently in use |
| 578 |
via a mount against a cgroup v1 hierarchy. |
| 579 |
Or, to put things another way, it is not possible to employ |
| 580 |
the same controller against both a v1 hierarchy and the unified v2 hierarchy. |
| 581 |
This means that it may be necessary first to unmount a v1 controller |
| 582 |
(as described above) before that controller is available in v2. |
| 583 |
Since |
| 584 |
.BR systemd (1) |
| 585 |
makes heavy use of some v1 controllers by default, |
| 586 |
it can in some cases be simpler to boot the system with |
| 587 |
selected v1 controllers disabled. |
| 588 |
To do this, specify the |
| 589 |
.I cgroup_no_v1=list |
| 590 |
option on the kernel boot command line; |
| 591 |
.I list |
| 592 |
is a comma-separated list of the names of the controllers to disable, |
| 593 |
or the word |
| 594 |
.I all |
| 595 |
to disable all v1 controllers. |
| 596 |
(This situation is correctly handled by |
| 597 |
.BR systemd (1), |
| 598 |
which falls back to operating without the specified controllers.) |
| 599 |
.P |
| 600 |
Note that on many modern systems, |
| 601 |
.BR systemd (1) |
| 602 |
automatically mounts the |
| 603 |
.I cgroup2 |
| 604 |
filesystem at |
| 605 |
.I /sys/fs/cgroup/unified |
| 606 |
during the boot process. |
| 607 |
.\" |
| 608 |
.SS Cgroups v2 mount options |
| 609 |
The following options |
| 610 |
.RI ( mount\~\-o ) |
| 611 |
can be specified when mounting the group v2 filesystem: |
| 612 |
.TP |
| 613 |
.IR nsdelegate " (since Linux 4.15)" |
| 614 |
Treat cgroup namespaces as delegation boundaries. |
| 615 |
For details, see below. |
| 616 |
.TP |
| 617 |
.IR memory_localevents " (since Linux 5.2)" |
| 618 |
.\" commit 9852ae3fe5293264f01c49f2571ef7688f7823ce |
| 619 |
The |
| 620 |
.I memory.events |
| 621 |
should show statistics only for the cgroup itself, |
| 622 |
and not for any descendant cgroups. |
| 623 |
This was the behavior before Linux 5.2. |
| 624 |
Since Linux 5.2, |
| 625 |
the default behavior is to include statistics for descendant cgroups in |
| 626 |
.IR memory.events , |
| 627 |
and this mount option can be used to revert to the legacy behavior. |
| 628 |
This option is system wide and can be set on mount or |
| 629 |
modified through remount only from the initial mount namespace; |
| 630 |
it is silently ignored in noninitial namespaces. |
| 631 |
.\" |
| 632 |
.SS Cgroups v2 controllers |
| 633 |
The following controllers, documented in the kernel source file |
| 634 |
.IR Documentation/admin\-guide/cgroup\-v2.rst , |
| 635 |
are supported in cgroups version 2: |
| 636 |
.TP |
| 637 |
.IR cpu " (since Linux 4.15)" |
| 638 |
This is the successor to the version 1 |
| 639 |
.I cpu |
| 640 |
and |
| 641 |
.I cpuacct |
| 642 |
controllers. |
| 643 |
.TP |
| 644 |
.IR cpuset " (since Linux 5.0)" |
| 645 |
This is the successor of the version 1 |
| 646 |
.I cpuset |
| 647 |
controller. |
| 648 |
.TP |
| 649 |
.IR freezer " (since Linux 5.2)" |
| 650 |
.\" commit 76f969e8948d82e78e1bc4beb6b9465908e74873 |
| 651 |
This is the successor of the version 1 |
| 652 |
.I freezer |
| 653 |
controller. |
| 654 |
.TP |
| 655 |
.IR hugetlb " (since Linux 5.6)" |
| 656 |
This is the successor of the version 1 |
| 657 |
.I hugetlb |
| 658 |
controller. |
| 659 |
.TP |
| 660 |
.IR io " (since Linux 4.5)" |
| 661 |
This is the successor of the version 1 |
| 662 |
.I blkio |
| 663 |
controller. |
| 664 |
.TP |
| 665 |
.IR memory " (since Linux 4.5)" |
| 666 |
This is the successor of the version 1 |
| 667 |
.I memory |
| 668 |
controller. |
| 669 |
.TP |
| 670 |
.IR perf_event " (since Linux 4.11)" |
| 671 |
This is the same as the version 1 |
| 672 |
.I perf_event |
| 673 |
controller. |
| 674 |
.TP |
| 675 |
.IR pids " (since Linux 4.5)" |
| 676 |
This is the same as the version 1 |
| 677 |
.I pids |
| 678 |
controller. |
| 679 |
.TP |
| 680 |
.IR rdma " (since Linux 4.11)" |
| 681 |
This is the same as the version 1 |
| 682 |
.I rdma |
| 683 |
controller. |
| 684 |
.P |
| 685 |
There is no direct equivalent of the |
| 686 |
.I net_cls |
| 687 |
and |
| 688 |
.I net_prio |
| 689 |
controllers from cgroups version 1. |
| 690 |
Instead, support has been added to |
| 691 |
.BR iptables (8) |
| 692 |
to allow eBPF filters that hook on cgroup v2 pathnames to make decisions |
| 693 |
about network traffic on a per-cgroup basis. |
| 694 |
.P |
| 695 |
The v2 |
| 696 |
.I devices |
| 697 |
controller provides no interface files; |
| 698 |
instead, device control is gated by attaching an eBPF |
| 699 |
.RB ( BPF_CGROUP_DEVICE ) |
| 700 |
program to a v2 cgroup. |
| 701 |
.\" |
| 702 |
.SS Cgroups v2 subtree control |
| 703 |
Each cgroup in the v2 hierarchy contains the following two files: |
| 704 |
.TP |
| 705 |
.I cgroup.controllers |
| 706 |
This read-only file exposes a list of the controllers that are |
| 707 |
.I available |
| 708 |
in this cgroup. |
| 709 |
The contents of this file match the contents of the |
| 710 |
.I cgroup.subtree_control |
| 711 |
file in the parent cgroup. |
| 712 |
.TP |
| 713 |
.I cgroup.subtree_control |
| 714 |
This is a list of controllers that are |
| 715 |
.I active |
| 716 |
.RI ( enabled ) |
| 717 |
in the cgroup. |
| 718 |
The set of controllers in this file is a subset of the set in the |
| 719 |
.I cgroup.controllers |
| 720 |
of this cgroup. |
| 721 |
The set of active controllers is modified by writing strings to this file |
| 722 |
containing space-delimited controller names, |
| 723 |
each preceded by '+' (to enable a controller) |
| 724 |
or '\-' (to disable a controller), as in the following example: |
| 725 |
.IP |
| 726 |
.in +4n |
| 727 |
.EX |
| 728 |
echo \[aq]+pids \-memory\[aq] > x/y/cgroup.subtree_control |
| 729 |
.EE |
| 730 |
.in |
| 731 |
.IP |
| 732 |
An attempt to enable a controller |
| 733 |
that is not present in |
| 734 |
.I cgroup.controllers |
| 735 |
leads to an |
| 736 |
.B ENOENT |
| 737 |
error when writing to the |
| 738 |
.I cgroup.subtree_control |
| 739 |
file. |
| 740 |
.P |
| 741 |
Because the list of controllers in |
| 742 |
.I cgroup.subtree_control |
| 743 |
is a subset of those |
| 744 |
.IR cgroup.controllers , |
| 745 |
a controller that has been disabled in one cgroup in the hierarchy |
| 746 |
can never be re-enabled in the subtree below that cgroup. |
| 747 |
.P |
| 748 |
A cgroup's |
| 749 |
.I cgroup.subtree_control |
| 750 |
file determines the set of controllers that are exercised in the |
| 751 |
.I child |
| 752 |
cgroups. |
| 753 |
When a controller (e.g., |
| 754 |
.IR pids ) |
| 755 |
is present in the |
| 756 |
.I cgroup.subtree_control |
| 757 |
file of a parent cgroup, |
| 758 |
then the corresponding controller-interface files (e.g., |
| 759 |
.IR pids.max ) |
| 760 |
are automatically created in the children of that cgroup |
| 761 |
and can be used to exert resource control in the child cgroups. |
| 762 |
.\" |
| 763 |
.SS Cgroups v2 \[dq]no internal processes\[dq] rule |
| 764 |
Cgroups v2 enforces a so-called "no internal processes" rule. |
| 765 |
Roughly speaking, this rule means that, |
| 766 |
with the exception of the root cgroup, processes may reside |
| 767 |
only in leaf nodes (cgroups that do not themselves contain child cgroups). |
| 768 |
This avoids the need to decide how to partition resources between |
| 769 |
processes which are members of cgroup A and processes in child cgroups of A. |
| 770 |
.P |
| 771 |
For instance, if cgroup |
| 772 |
.I /cg1/cg2 |
| 773 |
exists, then a process may reside in |
| 774 |
.IR /cg1/cg2 , |
| 775 |
but not in |
| 776 |
.IR /cg1 . |
| 777 |
This is to avoid an ambiguity in cgroups v1 |
| 778 |
with respect to the delegation of resources between processes in |
| 779 |
.I /cg1 |
| 780 |
and its child cgroups. |
| 781 |
The recommended approach in cgroups v2 is to create a subdirectory called |
| 782 |
.I leaf |
| 783 |
for any nonleaf cgroup which should contain processes, but no child cgroups. |
| 784 |
Thus, processes which previously would have gone into |
| 785 |
.I /cg1 |
| 786 |
would now go into |
| 787 |
.IR /cg1/leaf . |
| 788 |
This has the advantage of making explicit |
| 789 |
the relationship between processes in |
| 790 |
.I /cg1/leaf |
| 791 |
and |
| 792 |
.IR /cg1 's |
| 793 |
other children. |
| 794 |
.P |
| 795 |
The "no internal processes" rule is in fact more subtle than stated above. |
| 796 |
More precisely, the rule is that a (nonroot) cgroup can't both |
| 797 |
(1) have member processes, and |
| 798 |
(2) distribute resources into child cgroups\[em]that is, have a nonempty |
| 799 |
.I cgroup.subtree_control |
| 800 |
file. |
| 801 |
Thus, it |
| 802 |
.I is |
| 803 |
possible for a cgroup to have both member processes and child cgroups, |
| 804 |
but before controllers can be enabled for that cgroup, |
| 805 |
the member processes must be moved out of the cgroup |
| 806 |
(e.g., perhaps into the child cgroups). |
| 807 |
.P |
| 808 |
With the Linux 4.14 addition of "thread mode" (described below), |
| 809 |
the "no internal processes" rule has been relaxed in some cases. |
| 810 |
.\" |
| 811 |
.SS Cgroups v2 cgroup.events file |
| 812 |
Each nonroot cgroup in the v2 hierarchy contains a read-only file, |
| 813 |
.IR cgroup.events , |
| 814 |
whose contents are key-value pairs |
| 815 |
(delimited by newline characters, with the key and value separated by spaces) |
| 816 |
providing state information about the cgroup: |
| 817 |
.P |
| 818 |
.in +4n |
| 819 |
.EX |
| 820 |
.RB $ " cat mygrp/cgroup.events" |
| 821 |
populated 1 |
| 822 |
frozen 0 |
| 823 |
.EE |
| 824 |
.in |
| 825 |
.P |
| 826 |
The following keys may appear in this file: |
| 827 |
.TP |
| 828 |
.I populated |
| 829 |
The value of this key is either 1, |
| 830 |
if this cgroup or any of its descendants has member processes, |
| 831 |
or otherwise 0. |
| 832 |
.TP |
| 833 |
.IR frozen " (since Linux 5.2)" |
| 834 |
.\" commit 76f969e8948d82e78e1bc4beb6b9465908e7487 |
| 835 |
The value of this key is 1 if this cgroup is currently frozen, |
| 836 |
or 0 if it is not. |
| 837 |
.P |
| 838 |
The |
| 839 |
.I cgroup.events |
| 840 |
file can be monitored, in order to receive notification when the value of |
| 841 |
one of its keys changes. |
| 842 |
Such monitoring can be done using |
| 843 |
.BR inotify (7), |
| 844 |
which notifies changes as |
| 845 |
.B IN_MODIFY |
| 846 |
events, or |
| 847 |
.BR poll (2), |
| 848 |
which notifies changes by returning the |
| 849 |
.B POLLPRI |
| 850 |
and |
| 851 |
.B POLLERR |
| 852 |
bits in the |
| 853 |
.I revents |
| 854 |
field. |
| 855 |
.\" |
| 856 |
.SS Cgroup v2 release notification |
| 857 |
Cgroups v2 provides a new mechanism for obtaining notification |
| 858 |
when a cgroup becomes empty. |
| 859 |
The cgroups v1 |
| 860 |
.I release_agent |
| 861 |
and |
| 862 |
.I notify_on_release |
| 863 |
files are removed, and replaced by the |
| 864 |
.I populated |
| 865 |
key in the |
| 866 |
.I cgroup.events |
| 867 |
file. |
| 868 |
This key either has the value 0, |
| 869 |
meaning that the cgroup (and its descendants) |
| 870 |
contain no (nonzombie) member processes, |
| 871 |
or 1, meaning that the cgroup (or one of its descendants) |
| 872 |
contains member processes. |
| 873 |
.P |
| 874 |
The cgroups v2 release-notification mechanism |
| 875 |
offers the following advantages over the cgroups v1 |
| 876 |
.I release_agent |
| 877 |
mechanism: |
| 878 |
.IP \[bu] 3 |
| 879 |
It allows for cheaper notification, |
| 880 |
since a single process can monitor multiple |
| 881 |
.I cgroup.events |
| 882 |
files (using the techniques described earlier). |
| 883 |
By contrast, the cgroups v1 mechanism requires the expense of creating |
| 884 |
a process for each notification. |
| 885 |
.IP \[bu] |
| 886 |
Notification for different cgroup subhierarchies can be delegated |
| 887 |
to different processes. |
| 888 |
By contrast, the cgroups v1 mechanism allows only one release agent |
| 889 |
for an entire hierarchy. |
| 890 |
.\" |
| 891 |
.SS Cgroups v2 cgroup.stat file |
| 892 |
.\" commit ec39225cca42c05ac36853d11d28f877fde5c42e |
| 893 |
Each cgroup in the v2 hierarchy contains a read-only |
| 894 |
.I cgroup.stat |
| 895 |
file (first introduced in Linux 4.14) |
| 896 |
that consists of lines containing key-value pairs. |
| 897 |
The following keys currently appear in this file: |
| 898 |
.TP |
| 899 |
.I nr_descendants |
| 900 |
This is the total number of visible (i.e., living) descendant cgroups |
| 901 |
underneath this cgroup. |
| 902 |
.TP |
| 903 |
.I nr_dying_descendants |
| 904 |
This is the total number of dying descendant cgroups |
| 905 |
underneath this cgroup. |
| 906 |
A cgroup enters the dying state after being deleted. |
| 907 |
It remains in that state for an undefined period |
| 908 |
(which will depend on system load) |
| 909 |
while resources are freed before the cgroup is destroyed. |
| 910 |
Note that the presence of some cgroups in the dying state is normal, |
| 911 |
and is not indicative of any problem. |
| 912 |
.IP |
| 913 |
A process can't be made a member of a dying cgroup, |
| 914 |
and a dying cgroup can't be brought back to life. |
| 915 |
.\" |
| 916 |
.SS Limiting the number of descendant cgroups |
| 917 |
Each cgroup in the v2 hierarchy contains the following files, |
| 918 |
which can be used to view and set limits on the number |
| 919 |
of descendant cgroups under that cgroup: |
| 920 |
.TP |
| 921 |
.IR cgroup.max.depth " (since Linux 4.14)" |
| 922 |
.\" commit 1a926e0bbab83bae8207d05a533173425e0496d1 |
| 923 |
This file defines a limit on the depth of nesting of descendant cgroups. |
| 924 |
A value of 0 in this file means that no descendant cgroups can be created. |
| 925 |
An attempt to create a descendant whose nesting level exceeds |
| 926 |
the limit fails |
| 927 |
.RI ( mkdir (2) |
| 928 |
fails with the error |
| 929 |
.BR EAGAIN ). |
| 930 |
.IP |
| 931 |
Writing the string |
| 932 |
.I \[dq]max\[dq] |
| 933 |
to this file means that no limit is imposed. |
| 934 |
The default value in this file is |
| 935 |
.IR \[dq]max\[dq] . |
| 936 |
.TP |
| 937 |
.IR cgroup.max.descendants " (since Linux 4.14)" |
| 938 |
.\" commit 1a926e0bbab83bae8207d05a533173425e0496d1 |
| 939 |
This file defines a limit on the number of live descendant cgroups that |
| 940 |
this cgroup may have. |
| 941 |
An attempt to create more descendants than allowed by the limit fails |
| 942 |
.RI ( mkdir (2) |
| 943 |
fails with the error |
| 944 |
.BR EAGAIN ). |
| 945 |
.IP |
| 946 |
Writing the string |
| 947 |
.I \[dq]max\[dq] |
| 948 |
to this file means that no limit is imposed. |
| 949 |
The default value in this file is |
| 950 |
.IR \[dq]max\[dq] . |
| 951 |
.\" |
| 952 |
.SH CGROUPS DELEGATION: DELEGATING A HIERARCHY TO A LESS PRIVILEGED USER |
| 953 |
In the context of cgroups, |
| 954 |
delegation means passing management of some subtree |
| 955 |
of the cgroup hierarchy to a nonprivileged user. |
| 956 |
Cgroups v1 provides support for delegation based on file permissions |
| 957 |
in the cgroup hierarchy but with less strict containment rules than v2 |
| 958 |
(as noted below). |
| 959 |
Cgroups v2 supports delegation with containment by explicit design. |
| 960 |
The focus of the discussion in this section is on delegation in cgroups v2, |
| 961 |
with some differences for cgroups v1 noted along the way. |
| 962 |
.P |
| 963 |
Some terminology is required in order to describe delegation. |
| 964 |
A |
| 965 |
.I delegater |
| 966 |
is a privileged user (i.e., root) who owns a parent cgroup. |
| 967 |
A |
| 968 |
.I delegatee |
| 969 |
is a nonprivileged user who will be granted the permissions needed |
| 970 |
to manage some subhierarchy under that parent cgroup, |
| 971 |
known as the |
| 972 |
.IR "delegated subtree" . |
| 973 |
.P |
| 974 |
To perform delegation, |
| 975 |
the delegater makes certain directories and files writable by the delegatee, |
| 976 |
typically by changing the ownership of the objects to be the user ID |
| 977 |
of the delegatee. |
| 978 |
Assuming that we want to delegate the hierarchy rooted at (say) |
| 979 |
.I /dlgt_grp |
| 980 |
and that there are not yet any child cgroups under that cgroup, |
| 981 |
the ownership of the following is changed to the user ID of the delegatee: |
| 982 |
.TP |
| 983 |
.I /dlgt_grp |
| 984 |
Changing the ownership of the root of the subtree means that any new |
| 985 |
cgroups created under the subtree (and the files they contain) |
| 986 |
will also be owned by the delegatee. |
| 987 |
.TP |
| 988 |
.I /dlgt_grp/cgroup.procs |
| 989 |
Changing the ownership of this file means that the delegatee |
| 990 |
can move processes into the root of the delegated subtree. |
| 991 |
.TP |
| 992 |
.IR /dlgt_grp/cgroup.subtree_control " (cgroups v2 only)" |
| 993 |
Changing the ownership of this file means that the delegatee |
| 994 |
can enable controllers (that are present in |
| 995 |
.IR /dlgt_grp/cgroup.controllers ) |
| 996 |
in order to further redistribute resources at lower levels in the subtree. |
| 997 |
(As an alternative to changing the ownership of this file, |
| 998 |
the delegater might instead add selected controllers to this file.) |
| 999 |
.TP |
| 1000 |
.IR /dlgt_grp/cgroup.threads " (cgroups v2 only)" |
| 1001 |
Changing the ownership of this file is necessary if a threaded subtree |
| 1002 |
is being delegated (see the description of "thread mode", below). |
| 1003 |
This permits the delegatee to write thread IDs to the file. |
| 1004 |
(The ownership of this file can also be changed when delegating |
| 1005 |
a domain subtree, but currently this serves no purpose, |
| 1006 |
since, as described below, it is not possible to move a thread between |
| 1007 |
domain cgroups by writing its thread ID to the |
| 1008 |
.I cgroup.threads |
| 1009 |
file.) |
| 1010 |
.IP |
| 1011 |
In cgroups v1, the corresponding file that should instead be delegated is the |
| 1012 |
.I tasks |
| 1013 |
file. |
| 1014 |
.P |
| 1015 |
The delegater should |
| 1016 |
.I not |
| 1017 |
change the ownership of any of the controller interfaces files (e.g., |
| 1018 |
.IR pids.max , |
| 1019 |
.IR memory.high ) |
| 1020 |
in |
| 1021 |
.IR dlgt_grp . |
| 1022 |
Those files are used from the next level above the delegated subtree |
| 1023 |
in order to distribute resources into the subtree, |
| 1024 |
and the delegatee should not have permission to change |
| 1025 |
the resources that are distributed into the delegated subtree. |
| 1026 |
.P |
| 1027 |
See also the discussion of the |
| 1028 |
.I /sys/kernel/cgroup/delegate |
| 1029 |
file in NOTES for information about further delegatable files in cgroups v2. |
| 1030 |
.P |
| 1031 |
After the aforementioned steps have been performed, |
| 1032 |
the delegatee can create child cgroups within the delegated subtree |
| 1033 |
(the cgroup subdirectories and the files they contain |
| 1034 |
will be owned by the delegatee) |
| 1035 |
and move processes between cgroups in the subtree. |
| 1036 |
If some controllers are present in |
| 1037 |
.IR dlgt_grp/cgroup.subtree_control , |
| 1038 |
or the ownership of that file was passed to the delegatee, |
| 1039 |
the delegatee can also control the further redistribution |
| 1040 |
of the corresponding resources into the delegated subtree. |
| 1041 |
.\" |
| 1042 |
.SS Cgroups v2 delegation: nsdelegate and cgroup namespaces |
| 1043 |
Starting with Linux 4.13, |
| 1044 |
.\" commit 5136f6365ce3eace5a926e10f16ed2a233db5ba9 |
| 1045 |
there is a second way to perform cgroup delegation in the cgroups v2 hierarchy. |
| 1046 |
This is done by mounting or remounting the cgroup v2 filesystem with the |
| 1047 |
.I nsdelegate |
| 1048 |
mount option. |
| 1049 |
For example, if the cgroup v2 filesystem has already been mounted, |
| 1050 |
we can remount it with the |
| 1051 |
.I nsdelegate |
| 1052 |
option as follows: |
| 1053 |
.P |
| 1054 |
.in +4n |
| 1055 |
.EX |
| 1056 |
mount \-t cgroup2 \-o remount,nsdelegate \[rs] |
| 1057 |
none /sys/fs/cgroup/unified |
| 1058 |
.EE |
| 1059 |
.in |
| 1060 |
.\" |
| 1061 |
.\" Alternatively, we could boot the kernel with the options: |
| 1062 |
.\" |
| 1063 |
.\" cgroup_no_v1=all systemd.legacy_systemd_cgroup_controller |
| 1064 |
.\" |
| 1065 |
.\" The effect of the latter option is to prevent systemd from employing |
| 1066 |
.\" its "hybrid" cgroup mode, where it tries to make use of cgroups v2. |
| 1067 |
.P |
| 1068 |
The effect of this mount option is to cause cgroup namespaces |
| 1069 |
to automatically become delegation boundaries. |
| 1070 |
More specifically, |
| 1071 |
the following restrictions apply for processes inside the cgroup namespace: |
| 1072 |
.IP \[bu] 3 |
| 1073 |
Writes to controller interface files in the root directory of the namespace |
| 1074 |
will fail with the error |
| 1075 |
.BR EPERM . |
| 1076 |
Processes inside the cgroup namespace can still write to delegatable |
| 1077 |
files in the root directory of the cgroup namespace such as |
| 1078 |
.I cgroup.procs |
| 1079 |
and |
| 1080 |
.IR cgroup.subtree_control , |
| 1081 |
and can create subhierarchy underneath the root directory. |
| 1082 |
.IP \[bu] |
| 1083 |
Attempts to migrate processes across the namespace boundary are denied |
| 1084 |
(with the error |
| 1085 |
.BR ENOENT ). |
| 1086 |
Processes inside the cgroup namespace can still |
| 1087 |
(subject to the containment rules described below) |
| 1088 |
move processes between cgroups |
| 1089 |
.I within |
| 1090 |
the subhierarchy under the namespace root. |
| 1091 |
.P |
| 1092 |
The ability to define cgroup namespaces as delegation boundaries |
| 1093 |
makes cgroup namespaces more useful. |
| 1094 |
To understand why, suppose that we already have one cgroup hierarchy |
| 1095 |
that has been delegated to a nonprivileged user, |
| 1096 |
.IR cecilia , |
| 1097 |
using the older delegation technique described above. |
| 1098 |
Suppose further that |
| 1099 |
.I cecilia |
| 1100 |
wanted to further delegate a subhierarchy |
| 1101 |
under the existing delegated hierarchy. |
| 1102 |
(For example, the delegated hierarchy might be associated with |
| 1103 |
an unprivileged container run by |
| 1104 |
.IR cecilia .) |
| 1105 |
Even if a cgroup namespace was employed, |
| 1106 |
because both hierarchies are owned by the unprivileged user |
| 1107 |
.IR cecilia , |
| 1108 |
the following illegitimate actions could be performed: |
| 1109 |
.IP \[bu] 3 |
| 1110 |
A process in the inferior hierarchy could change the |
| 1111 |
resource controller settings in the root directory of that hierarchy. |
| 1112 |
(These resource controller settings are intended to allow control to |
| 1113 |
be exercised from the |
| 1114 |
.I parent |
| 1115 |
cgroup; |
| 1116 |
a process inside the child cgroup should not be allowed to modify them.) |
| 1117 |
.IP \[bu] |
| 1118 |
A process inside the inferior hierarchy could move processes |
| 1119 |
into and out of the inferior hierarchy if the cgroups in the |
| 1120 |
superior hierarchy were somehow visible. |
| 1121 |
.P |
| 1122 |
Employing the |
| 1123 |
.I nsdelegate |
| 1124 |
mount option prevents both of these possibilities. |
| 1125 |
.P |
| 1126 |
The |
| 1127 |
.I nsdelegate |
| 1128 |
mount option only has an effect when performed in |
| 1129 |
the initial mount namespace; |
| 1130 |
in other mount namespaces, the option is silently ignored. |
| 1131 |
.P |
| 1132 |
.IR Note : |
| 1133 |
On some systems, |
| 1134 |
.BR systemd (1) |
| 1135 |
automatically mounts the cgroup v2 filesystem. |
| 1136 |
In order to experiment with the |
| 1137 |
.I nsdelegate |
| 1138 |
operation, it may be useful to boot the kernel with |
| 1139 |
the following command-line options: |
| 1140 |
.P |
| 1141 |
.in +4n |
| 1142 |
.EX |
| 1143 |
cgroup_no_v1=all systemd.legacy_systemd_cgroup_controller |
| 1144 |
.EE |
| 1145 |
.in |
| 1146 |
.P |
| 1147 |
These options cause the kernel to boot with the cgroups v1 controllers |
| 1148 |
disabled (meaning that the controllers are available in the v2 hierarchy), |
| 1149 |
and tells |
| 1150 |
.BR systemd (1) |
| 1151 |
not to mount and use the cgroup v2 hierarchy, |
| 1152 |
so that the v2 hierarchy can be manually mounted |
| 1153 |
with the desired options after boot-up. |
| 1154 |
.\" |
| 1155 |
.SS Cgroup delegation containment rules |
| 1156 |
Some delegation |
| 1157 |
.I containment rules |
| 1158 |
ensure that the delegatee can move processes between cgroups within the |
| 1159 |
delegated subtree, |
| 1160 |
but can't move processes from outside the delegated subtree into |
| 1161 |
the subtree or vice versa. |
| 1162 |
A nonprivileged process (i.e., the delegatee) can write the PID of |
| 1163 |
a "target" process into a |
| 1164 |
.I cgroup.procs |
| 1165 |
file only if all of the following are true: |
| 1166 |
.IP \[bu] 3 |
| 1167 |
The writer has write permission on the |
| 1168 |
.I cgroup.procs |
| 1169 |
file in the destination cgroup. |
| 1170 |
.IP \[bu] |
| 1171 |
The writer has write permission on the |
| 1172 |
.I cgroup.procs |
| 1173 |
file in the nearest common ancestor of the source and destination cgroups. |
| 1174 |
Note that in some cases, |
| 1175 |
the nearest common ancestor may be the source or destination cgroup itself. |
| 1176 |
This requirement is not enforced for cgroups v1 hierarchies, |
| 1177 |
with the consequence that containment in v1 is less strict than in v2. |
| 1178 |
(For example, in cgroups v1 the user that owns two distinct |
| 1179 |
delegated subhierarchies can move a process between the hierarchies.) |
| 1180 |
.IP \[bu] |
| 1181 |
If the cgroup v2 filesystem was mounted with the |
| 1182 |
.I nsdelegate |
| 1183 |
option, the writer must be able to see the source and destination cgroups |
| 1184 |
from its cgroup namespace. |
| 1185 |
.IP \[bu] |
| 1186 |
In cgroups v1: |
| 1187 |
the effective UID of the writer (i.e., the delegatee) matches the |
| 1188 |
real user ID or the saved set-user-ID of the target process. |
| 1189 |
Before Linux 4.11, |
| 1190 |
.\" commit 576dd464505fc53d501bb94569db76f220104d28 |
| 1191 |
this requirement also applied in cgroups v2 |
| 1192 |
(This was a historical requirement inherited from cgroups v1 |
| 1193 |
that was later deemed unnecessary, |
| 1194 |
since the other rules suffice for containment in cgroups v2.) |
| 1195 |
.P |
| 1196 |
.IR Note : |
| 1197 |
one consequence of these delegation containment rules is that the |
| 1198 |
unprivileged delegatee can't place the first process into |
| 1199 |
the delegated subtree; |
| 1200 |
instead, the delegater must place the first process |
| 1201 |
(a process owned by the delegatee) into the delegated subtree. |
| 1202 |
.\" |
| 1203 |
.SH CGROUPS VERSION 2 THREAD MODE |
| 1204 |
Among the restrictions imposed by cgroups v2 that were not present |
| 1205 |
in cgroups v1 are the following: |
| 1206 |
.IP \[bu] 3 |
| 1207 |
.IR "No thread-granularity control" : |
| 1208 |
all of the threads of a process must be in the same cgroup. |
| 1209 |
.IP \[bu] |
| 1210 |
.IR "No internal processes" : |
| 1211 |
a cgroup can't both have member processes and |
| 1212 |
exercise controllers on child cgroups. |
| 1213 |
.P |
| 1214 |
Both of these restrictions were added because |
| 1215 |
the lack of these restrictions had caused problems |
| 1216 |
in cgroups v1. |
| 1217 |
In particular, the cgroups v1 ability to allow thread-level granularity |
| 1218 |
for cgroup membership made no sense for some controllers. |
| 1219 |
(A notable example was the |
| 1220 |
.I memory |
| 1221 |
controller: since threads share an address space, |
| 1222 |
it made no sense to split threads across different |
| 1223 |
.I memory |
| 1224 |
cgroups.) |
| 1225 |
.P |
| 1226 |
Notwithstanding the initial design decision in cgroups v2, |
| 1227 |
there were use cases for certain controllers, notably the |
| 1228 |
.I cpu |
| 1229 |
controller, |
| 1230 |
for which thread-level granularity of control was meaningful and useful. |
| 1231 |
To accommodate such use cases, Linux 4.14 added |
| 1232 |
.I "thread mode" |
| 1233 |
for cgroups v2. |
| 1234 |
.P |
| 1235 |
Thread mode allows the following: |
| 1236 |
.IP \[bu] 3 |
| 1237 |
The creation of |
| 1238 |
.I threaded subtrees |
| 1239 |
in which the threads of a process may |
| 1240 |
be spread across cgroups inside the tree. |
| 1241 |
(A threaded subtree may contain multiple multithreaded processes.) |
| 1242 |
.IP \[bu] |
| 1243 |
The concept of |
| 1244 |
.IR "threaded controllers" , |
| 1245 |
which can distribute resources across the cgroups in a threaded subtree. |
| 1246 |
.IP \[bu] |
| 1247 |
A relaxation of the "no internal processes rule", |
| 1248 |
so that, within a threaded subtree, |
| 1249 |
a cgroup can both contain member threads and |
| 1250 |
exercise resource control over child cgroups. |
| 1251 |
.P |
| 1252 |
With the addition of thread mode, |
| 1253 |
each nonroot cgroup now contains a new file, |
| 1254 |
.IR cgroup.type , |
| 1255 |
that exposes, and in some circumstances can be used to change, |
| 1256 |
the "type" of a cgroup. |
| 1257 |
This file contains one of the following type values: |
| 1258 |
.TP |
| 1259 |
.I domain |
| 1260 |
This is a normal v2 cgroup that provides process-granularity control. |
| 1261 |
If a process is a member of this cgroup, |
| 1262 |
then all threads of the process are (by definition) in the same cgroup. |
| 1263 |
This is the default cgroup type, |
| 1264 |
and provides the same behavior that was provided for |
| 1265 |
cgroups in the initial cgroups v2 implementation. |
| 1266 |
.TP |
| 1267 |
.I threaded |
| 1268 |
This cgroup is a member of a threaded subtree. |
| 1269 |
Threads can be added to this cgroup, |
| 1270 |
and controllers can be enabled for the cgroup. |
| 1271 |
.TP |
| 1272 |
.I domain threaded |
| 1273 |
This is a domain cgroup that serves as the root of a threaded subtree. |
| 1274 |
This cgroup type is also known as "threaded root". |
| 1275 |
.TP |
| 1276 |
.I domain invalid |
| 1277 |
This is a cgroup inside a threaded subtree |
| 1278 |
that is in an "invalid" state. |
| 1279 |
Processes can't be added to the cgroup, |
| 1280 |
and controllers can't be enabled for the cgroup. |
| 1281 |
The only thing that can be done with this cgroup (other than deleting it) |
| 1282 |
is to convert it to a |
| 1283 |
.I threaded |
| 1284 |
cgroup by writing the string |
| 1285 |
.I \[dq]threaded\[dq] |
| 1286 |
to the |
| 1287 |
.I cgroup.type |
| 1288 |
file. |
| 1289 |
.IP |
| 1290 |
The rationale for the existence of this "interim" type |
| 1291 |
during the creation of a threaded subtree |
| 1292 |
(rather than the kernel simply immediately converting all cgroups |
| 1293 |
under the threaded root to the type |
| 1294 |
.IR threaded ) |
| 1295 |
is to allow for |
| 1296 |
possible future extensions to the thread mode model |
| 1297 |
.\" |
| 1298 |
.SS Threaded versus domain controllers |
| 1299 |
With the addition of threads mode, |
| 1300 |
cgroups v2 now distinguishes two types of resource controllers: |
| 1301 |
.IP \[bu] 3 |
| 1302 |
.I Threaded |
| 1303 |
.\" In the kernel source, look for ".threaded[ \t]*= true" in |
| 1304 |
.\" initializations of struct cgroup_subsys |
| 1305 |
controllers: these controllers support thread-granularity for |
| 1306 |
resource control and can be enabled inside threaded subtrees, |
| 1307 |
with the result that the corresponding controller-interface files |
| 1308 |
appear inside the cgroups in the threaded subtree. |
| 1309 |
As at Linux 4.19, the following controllers are threaded: |
| 1310 |
.IR cpu , |
| 1311 |
.IR perf_event , |
| 1312 |
and |
| 1313 |
.IR pids . |
| 1314 |
.IP \[bu] |
| 1315 |
.I Domain |
| 1316 |
controllers: these controllers support only process granularity |
| 1317 |
for resource control. |
| 1318 |
From the perspective of a domain controller, |
| 1319 |
all threads of a process are always in the same cgroup. |
| 1320 |
Domain controllers can't be enabled inside a threaded subtree. |
| 1321 |
.\" |
| 1322 |
.SS Creating a threaded subtree |
| 1323 |
There are two pathways that lead to the creation of a threaded subtree. |
| 1324 |
The first pathway proceeds as follows: |
| 1325 |
.IP (1) 5 |
| 1326 |
We write the string |
| 1327 |
.I \[dq]threaded\[dq] |
| 1328 |
to the |
| 1329 |
.I cgroup.type |
| 1330 |
file of a cgroup |
| 1331 |
.I y/z |
| 1332 |
that currently has the type |
| 1333 |
.IR domain . |
| 1334 |
This has the following effects: |
| 1335 |
.RS |
| 1336 |
.IP \[bu] 3 |
| 1337 |
The type of the cgroup |
| 1338 |
.I y/z |
| 1339 |
becomes |
| 1340 |
.IR threaded . |
| 1341 |
.IP \[bu] |
| 1342 |
The type of the parent cgroup, |
| 1343 |
.IR y , |
| 1344 |
becomes |
| 1345 |
.IR "domain threaded" . |
| 1346 |
The parent cgroup is the root of a threaded subtree |
| 1347 |
(also known as the "threaded root"). |
| 1348 |
.IP \[bu] |
| 1349 |
All other cgroups under |
| 1350 |
.I y |
| 1351 |
that were not already of type |
| 1352 |
.I threaded |
| 1353 |
(because they were inside already existing threaded subtrees |
| 1354 |
under the new threaded root) |
| 1355 |
are converted to type |
| 1356 |
.IR "domain invalid" . |
| 1357 |
Any subsequently created cgroups under |
| 1358 |
.I y |
| 1359 |
will also have the type |
| 1360 |
.IR "domain invalid" . |
| 1361 |
.RE |
| 1362 |
.IP (2) |
| 1363 |
We write the string |
| 1364 |
.I \[dq]threaded\[dq] |
| 1365 |
to each of the |
| 1366 |
.I domain invalid |
| 1367 |
cgroups under |
| 1368 |
.IR y , |
| 1369 |
in order to convert them to the type |
| 1370 |
.IR threaded . |
| 1371 |
As a consequence of this step, all threads under the threaded root |
| 1372 |
now have the type |
| 1373 |
.I threaded |
| 1374 |
and the threaded subtree is now fully usable. |
| 1375 |
The requirement to write |
| 1376 |
.I \[dq]threaded\[dq] |
| 1377 |
to each of these cgroups is somewhat cumbersome, |
| 1378 |
but allows for possible future extensions to the thread-mode model. |
| 1379 |
.P |
| 1380 |
The second way of creating a threaded subtree is as follows: |
| 1381 |
.IP (1) 5 |
| 1382 |
In an existing cgroup, |
| 1383 |
.IR z , |
| 1384 |
that currently has the type |
| 1385 |
.IR domain , |
| 1386 |
we (1.1) enable one or more threaded controllers and |
| 1387 |
(1.2) make a process a member of |
| 1388 |
.IR z . |
| 1389 |
(These two steps can be done in either order.) |
| 1390 |
This has the following consequences: |
| 1391 |
.RS |
| 1392 |
.IP \[bu] 3 |
| 1393 |
The type of |
| 1394 |
.I z |
| 1395 |
becomes |
| 1396 |
.IR "domain threaded" . |
| 1397 |
.IP \[bu] |
| 1398 |
All of the descendant cgroups of |
| 1399 |
.I z |
| 1400 |
that were not already of type |
| 1401 |
.I threaded |
| 1402 |
are converted to type |
| 1403 |
.IR "domain invalid" . |
| 1404 |
.RE |
| 1405 |
.IP (2) |
| 1406 |
As before, we make the threaded subtree usable by writing the string |
| 1407 |
.I \[dq]threaded\[dq] |
| 1408 |
to each of the |
| 1409 |
.I domain invalid |
| 1410 |
cgroups under |
| 1411 |
.IR z , |
| 1412 |
in order to convert them to the type |
| 1413 |
.IR threaded . |
| 1414 |
.P |
| 1415 |
One of the consequences of the above pathways to creating a threaded subtree |
| 1416 |
is that the threaded root cgroup can be a parent only to |
| 1417 |
.I threaded |
| 1418 |
(and |
| 1419 |
.IR "domain invalid" ) |
| 1420 |
cgroups. |
| 1421 |
The threaded root cgroup can't be a parent of a |
| 1422 |
.I domain |
| 1423 |
cgroup, and a |
| 1424 |
.I threaded |
| 1425 |
cgroup |
| 1426 |
can't have a sibling that is a |
| 1427 |
.I domain |
| 1428 |
cgroup. |
| 1429 |
.\" |
| 1430 |
.SS Using a threaded subtree |
| 1431 |
Within a threaded subtree, threaded controllers can be enabled |
| 1432 |
in each subgroup whose type has been changed to |
| 1433 |
.IR threaded ; |
| 1434 |
upon doing so, the corresponding controller interface files |
| 1435 |
appear in the children of that cgroup. |
| 1436 |
.P |
| 1437 |
A process can be moved into a threaded subtree by writing its PID to the |
| 1438 |
.I cgroup.procs |
| 1439 |
file in one of the cgroups inside the tree. |
| 1440 |
This has the effect of making all of the threads |
| 1441 |
in the process members of the corresponding cgroup |
| 1442 |
and makes the process a member of the threaded subtree. |
| 1443 |
The threads of the process can then be spread across |
| 1444 |
the threaded subtree by writing their thread IDs (see |
| 1445 |
.BR gettid (2)) |
| 1446 |
to the |
| 1447 |
.I cgroup.threads |
| 1448 |
files in different cgroups inside the subtree. |
| 1449 |
The threads of a process must all reside in the same threaded subtree. |
| 1450 |
.P |
| 1451 |
As with writing to |
| 1452 |
.IR cgroup.procs , |
| 1453 |
some containment rules apply when writing to the |
| 1454 |
.I cgroup.threads |
| 1455 |
file: |
| 1456 |
.IP \[bu] 3 |
| 1457 |
The writer must have write permission on the |
| 1458 |
cgroup.threads |
| 1459 |
file in the destination cgroup. |
| 1460 |
.IP \[bu] |
| 1461 |
The writer must have write permission on the |
| 1462 |
.I cgroup.procs |
| 1463 |
file in the common ancestor of the source and destination cgroups. |
| 1464 |
(In some cases, |
| 1465 |
the common ancestor may be the source or destination cgroup itself.) |
| 1466 |
.IP \[bu] |
| 1467 |
The source and destination cgroups must be in the same threaded subtree. |
| 1468 |
(Outside a threaded subtree, an attempt to move a thread by writing |
| 1469 |
its thread ID to the |
| 1470 |
.I cgroup.threads |
| 1471 |
file in a different |
| 1472 |
.I domain |
| 1473 |
cgroup fails with the error |
| 1474 |
.BR EOPNOTSUPP .) |
| 1475 |
.P |
| 1476 |
The |
| 1477 |
.I cgroup.threads |
| 1478 |
file is present in each cgroup (including |
| 1479 |
.I domain |
| 1480 |
cgroups) and can be read in order to discover the set of threads |
| 1481 |
that is present in the cgroup. |
| 1482 |
The set of thread IDs obtained when reading this file |
| 1483 |
is not guaranteed to be ordered or free of duplicates. |
| 1484 |
.P |
| 1485 |
The |
| 1486 |
.I cgroup.procs |
| 1487 |
file in the threaded root shows the PIDs of all processes |
| 1488 |
that are members of the threaded subtree. |
| 1489 |
The |
| 1490 |
.I cgroup.procs |
| 1491 |
files in the other cgroups in the subtree are not readable. |
| 1492 |
.P |
| 1493 |
Domain controllers can't be enabled in a threaded subtree; |
| 1494 |
no controller-interface files appear inside the cgroups underneath the |
| 1495 |
threaded root. |
| 1496 |
From the point of view of a domain controller, |
| 1497 |
threaded subtrees are invisible: |
| 1498 |
a multithreaded process inside a threaded subtree appears to a domain |
| 1499 |
controller as a process that resides in the threaded root cgroup. |
| 1500 |
.P |
| 1501 |
Within a threaded subtree, the "no internal processes" rule does not apply: |
| 1502 |
a cgroup can both contain member processes (or thread) |
| 1503 |
and exercise controllers on child cgroups. |
| 1504 |
.\" |
| 1505 |
.SS Rules for writing to cgroup.type and creating threaded subtrees |
| 1506 |
A number of rules apply when writing to the |
| 1507 |
.I cgroup.type |
| 1508 |
file: |
| 1509 |
.IP \[bu] 3 |
| 1510 |
Only the string |
| 1511 |
.I \[dq]threaded\[dq] |
| 1512 |
may be written. |
| 1513 |
In other words, the only explicit transition that is possible is to convert a |
| 1514 |
.I domain |
| 1515 |
cgroup to type |
| 1516 |
.IR threaded . |
| 1517 |
.IP \[bu] |
| 1518 |
The effect of writing |
| 1519 |
.I \[dq]threaded\[dq] |
| 1520 |
depends on the current value in |
| 1521 |
.IR cgroup.type , |
| 1522 |
as follows: |
| 1523 |
.RS |
| 1524 |
.IP \[bu] 3 |
| 1525 |
.I domain |
| 1526 |
or |
| 1527 |
.IR "domain threaded" : |
| 1528 |
start the creation of a threaded subtree |
| 1529 |
(whose root is the parent of this cgroup) via |
| 1530 |
the first of the pathways described above; |
| 1531 |
.IP \[bu] |
| 1532 |
.IR "domain\ invalid" : |
| 1533 |
convert this cgroup (which is inside a threaded subtree) to a usable (i.e., |
| 1534 |
.IR threaded ) |
| 1535 |
state; |
| 1536 |
.IP \[bu] |
| 1537 |
.IR threaded : |
| 1538 |
no effect (a "no-op"). |
| 1539 |
.RE |
| 1540 |
.IP \[bu] |
| 1541 |
We can't write to a |
| 1542 |
.I cgroup.type |
| 1543 |
file if the parent's type is |
| 1544 |
.IR "domain invalid" . |
| 1545 |
In other words, the cgroups of a threaded subtree must be converted to the |
| 1546 |
.I threaded |
| 1547 |
state in a top-down manner. |
| 1548 |
.P |
| 1549 |
There are also some constraints that must be satisfied |
| 1550 |
in order to create a threaded subtree rooted at the cgroup |
| 1551 |
.IR x : |
| 1552 |
.IP \[bu] 3 |
| 1553 |
There can be no member processes in the descendant cgroups of |
| 1554 |
.IR x . |
| 1555 |
(The cgroup |
| 1556 |
.I x |
| 1557 |
can itself have member processes.) |
| 1558 |
.IP \[bu] |
| 1559 |
No domain controllers may be enabled in |
| 1560 |
.IR x 's |
| 1561 |
.I cgroup.subtree_control |
| 1562 |
file. |
| 1563 |
.P |
| 1564 |
If any of the above constraints is violated, then an attempt to write |
| 1565 |
.I \[dq]threaded\[dq] |
| 1566 |
to a |
| 1567 |
.I cgroup.type |
| 1568 |
file fails with the error |
| 1569 |
.BR ENOTSUP . |
| 1570 |
.\" |
| 1571 |
.SS The \[dq]domain threaded\[dq] cgroup type |
| 1572 |
According to the pathways described above, |
| 1573 |
the type of a cgroup can change to |
| 1574 |
.I domain threaded |
| 1575 |
in either of the following cases: |
| 1576 |
.IP \[bu] 3 |
| 1577 |
The string |
| 1578 |
.I \[dq]threaded\[dq] |
| 1579 |
is written to a child cgroup. |
| 1580 |
.IP \[bu] |
| 1581 |
A threaded controller is enabled inside the cgroup and |
| 1582 |
a process is made a member of the cgroup. |
| 1583 |
.P |
| 1584 |
A |
| 1585 |
.I domain threaded |
| 1586 |
cgroup, |
| 1587 |
.IR x , |
| 1588 |
can revert to the type |
| 1589 |
.I domain |
| 1590 |
if the above conditions no longer hold true\[em]that is, if all |
| 1591 |
.I threaded |
| 1592 |
child cgroups of |
| 1593 |
.I x |
| 1594 |
are removed and either |
| 1595 |
.I x |
| 1596 |
no longer has threaded controllers enabled or |
| 1597 |
no longer has member processes. |
| 1598 |
.P |
| 1599 |
When a |
| 1600 |
.I domain threaded |
| 1601 |
cgroup |
| 1602 |
.I x |
| 1603 |
reverts to the type |
| 1604 |
.IR domain : |
| 1605 |
.IP \[bu] 3 |
| 1606 |
All |
| 1607 |
.I domain invalid |
| 1608 |
descendants of |
| 1609 |
.I x |
| 1610 |
that are not in lower-level threaded subtrees revert to the type |
| 1611 |
.IR domain . |
| 1612 |
.IP \[bu] |
| 1613 |
The root cgroups in any lower-level threaded subtrees revert to the type |
| 1614 |
.IR "domain threaded" . |
| 1615 |
.\" |
| 1616 |
.SS Exceptions for the root cgroup |
| 1617 |
The root cgroup of the v2 hierarchy is treated exceptionally: |
| 1618 |
it can be the parent of both |
| 1619 |
.I domain |
| 1620 |
and |
| 1621 |
.I threaded |
| 1622 |
cgroups. |
| 1623 |
If the string |
| 1624 |
.I \[dq]threaded\[dq] |
| 1625 |
is written to the |
| 1626 |
.I cgroup.type |
| 1627 |
file of one of the children of the root cgroup, then |
| 1628 |
.IP \[bu] 3 |
| 1629 |
The type of that cgroup becomes |
| 1630 |
.IR threaded . |
| 1631 |
.IP \[bu] |
| 1632 |
The type of any descendants of that cgroup that |
| 1633 |
are not part of lower-level threaded subtrees changes to |
| 1634 |
.IR "domain invalid" . |
| 1635 |
.P |
| 1636 |
Note that in this case, there is no cgroup whose type becomes |
| 1637 |
.IR "domain threaded" . |
| 1638 |
(Notionally, the root cgroup can be considered as the threaded root |
| 1639 |
for the cgroup whose type was changed to |
| 1640 |
.IR threaded .) |
| 1641 |
.P |
| 1642 |
The aim of this exceptional treatment for the root cgroup is to |
| 1643 |
allow a threaded cgroup that employs the |
| 1644 |
.I cpu |
| 1645 |
controller to be placed as high as possible in the hierarchy, |
| 1646 |
so as to minimize the (small) cost of traversing the cgroup hierarchy. |
| 1647 |
.\" |
| 1648 |
.SS The cgroups v2 \[dq]cpu\[dq] controller and realtime threads |
| 1649 |
As at Linux 4.19, the cgroups v2 |
| 1650 |
.I cpu |
| 1651 |
controller does not support control of realtime threads |
| 1652 |
(specifically threads scheduled under any of the policies |
| 1653 |
.BR SCHED_FIFO , |
| 1654 |
.BR SCHED_RR , |
| 1655 |
described |
| 1656 |
.BR SCHED_DEADLINE ; |
| 1657 |
see |
| 1658 |
.BR sched (7)). |
| 1659 |
Therefore, the |
| 1660 |
.I cpu |
| 1661 |
controller can be enabled in the root cgroup only |
| 1662 |
if all realtime threads are in the root cgroup. |
| 1663 |
(If there are realtime threads in nonroot cgroups, then a |
| 1664 |
.BR write (2) |
| 1665 |
of the string |
| 1666 |
.I \[dq]+cpu\[dq] |
| 1667 |
to the |
| 1668 |
.I cgroup.subtree_control |
| 1669 |
file fails with the error |
| 1670 |
.BR EINVAL .) |
| 1671 |
.P |
| 1672 |
On some systems, |
| 1673 |
.BR systemd (1) |
| 1674 |
places certain realtime threads in nonroot cgroups in the v2 hierarchy. |
| 1675 |
On such systems, |
| 1676 |
these threads must first be moved to the root cgroup before the |
| 1677 |
.I cpu |
| 1678 |
controller can be enabled. |
| 1679 |
.\" |
| 1680 |
.SH ERRORS |
| 1681 |
The following errors can occur for |
| 1682 |
.BR mount (2): |
| 1683 |
.TP |
| 1684 |
.B EBUSY |
| 1685 |
An attempt to mount a cgroup version 1 filesystem specified neither the |
| 1686 |
.I name= |
| 1687 |
option (to mount a named hierarchy) nor a controller name (or |
| 1688 |
.IR all ). |
| 1689 |
.SH NOTES |
| 1690 |
A child process created via |
| 1691 |
.BR fork (2) |
| 1692 |
inherits its parent's cgroup memberships. |
| 1693 |
A process's cgroup memberships are preserved across |
| 1694 |
.BR execve (2). |
| 1695 |
.P |
| 1696 |
The |
| 1697 |
.BR clone3 (2) |
| 1698 |
.B CLONE_INTO_CGROUP |
| 1699 |
flag can be used to create a child process that begins its life in |
| 1700 |
a different version 2 cgroup from the parent process. |
| 1701 |
.\" |
| 1702 |
.SS /proc files |
| 1703 |
.TP |
| 1704 |
.IR /proc/cgroups " (since Linux 2.6.24)" |
| 1705 |
This file contains information about the controllers |
| 1706 |
that are compiled into the kernel. |
| 1707 |
An example of the contents of this file (reformatted for readability) |
| 1708 |
is the following: |
| 1709 |
.IP |
| 1710 |
.in +4n |
| 1711 |
.EX |
| 1712 |
#subsys_name hierarchy num_cgroups enabled |
| 1713 |
cpuset 4 1 1 |
| 1714 |
cpu 8 1 1 |
| 1715 |
cpuacct 8 1 1 |
| 1716 |
blkio 6 1 1 |
| 1717 |
memory 3 1 1 |
| 1718 |
devices 10 84 1 |
| 1719 |
freezer 7 1 1 |
| 1720 |
net_cls 9 1 1 |
| 1721 |
perf_event 5 1 1 |
| 1722 |
net_prio 9 1 1 |
| 1723 |
hugetlb 0 1 0 |
| 1724 |
pids 2 1 1 |
| 1725 |
.EE |
| 1726 |
.in |
| 1727 |
.IP |
| 1728 |
The fields in this file are, from left to right: |
| 1729 |
.RS |
| 1730 |
.IP [1] 5 |
| 1731 |
The name of the controller. |
| 1732 |
.IP [2] |
| 1733 |
The unique ID of the cgroup hierarchy on which this controller is mounted. |
| 1734 |
If multiple cgroups v1 controllers are bound to the same hierarchy, |
| 1735 |
then each will show the same hierarchy ID in this field. |
| 1736 |
The value in this field will be 0 if: |
| 1737 |
.RS |
| 1738 |
.IP \[bu] 3 |
| 1739 |
the controller is not mounted on a cgroups v1 hierarchy; |
| 1740 |
.IP \[bu] |
| 1741 |
the controller is bound to the cgroups v2 single unified hierarchy; or |
| 1742 |
.IP \[bu] |
| 1743 |
the controller is disabled (see below). |
| 1744 |
.RE |
| 1745 |
.IP [3] |
| 1746 |
The number of control groups in this hierarchy using this controller. |
| 1747 |
.IP [4] |
| 1748 |
This field contains the value 1 if this controller is enabled, |
| 1749 |
or 0 if it has been disabled (via the |
| 1750 |
.I cgroup_disable |
| 1751 |
kernel command-line boot parameter). |
| 1752 |
.RE |
| 1753 |
.TP |
| 1754 |
.IR /proc/ pid /cgroup " (since Linux 2.6.24)" |
| 1755 |
This file describes control groups to which the process |
| 1756 |
with the corresponding PID belongs. |
| 1757 |
The displayed information differs for |
| 1758 |
cgroups version 1 and version 2 hierarchies. |
| 1759 |
.IP |
| 1760 |
For each cgroup hierarchy of which the process is a member, |
| 1761 |
there is one entry containing three colon-separated fields: |
| 1762 |
.IP |
| 1763 |
.in +4n |
| 1764 |
.EX |
| 1765 |
hierarchy\-ID:controller\-list:cgroup\-path |
| 1766 |
.EE |
| 1767 |
.in |
| 1768 |
.IP |
| 1769 |
For example: |
| 1770 |
.IP |
| 1771 |
.in +4n |
| 1772 |
.EX |
| 1773 |
5:cpuacct,cpu,cpuset:/daemons |
| 1774 |
.EE |
| 1775 |
.in |
| 1776 |
.IP |
| 1777 |
The colon-separated fields are, from left to right: |
| 1778 |
.RS |
| 1779 |
.IP [1] 5 |
| 1780 |
For cgroups version 1 hierarchies, |
| 1781 |
this field contains a unique hierarchy ID number |
| 1782 |
that can be matched to a hierarchy ID in |
| 1783 |
.IR /proc/cgroups . |
| 1784 |
For the cgroups version 2 hierarchy, this field contains the value 0. |
| 1785 |
.IP [2] |
| 1786 |
For cgroups version 1 hierarchies, |
| 1787 |
this field contains a comma-separated list of the controllers |
| 1788 |
bound to the hierarchy. |
| 1789 |
For the cgroups version 2 hierarchy, this field is empty. |
| 1790 |
.IP [3] |
| 1791 |
This field contains the pathname of the control group in the hierarchy |
| 1792 |
to which the process belongs. |
| 1793 |
This pathname is relative to the mount point of the hierarchy. |
| 1794 |
.RE |
| 1795 |
.\" |
| 1796 |
.SS /sys/kernel/cgroup files |
| 1797 |
.TP |
| 1798 |
.IR /sys/kernel/cgroup/delegate " (since Linux 4.15)" |
| 1799 |
.\" commit 01ee6cfb1483fe57c9cbd8e73817dfbf9bacffd3 |
| 1800 |
This file exports a list of the cgroups v2 files |
| 1801 |
(one per line) that are delegatable |
| 1802 |
(i.e., whose ownership should be changed to the user ID of the delegatee). |
| 1803 |
In the future, the set of delegatable files may change or grow, |
| 1804 |
and this file provides a way for the kernel to inform |
| 1805 |
user-space applications of which files must be delegated. |
| 1806 |
As at Linux 4.15, one sees the following when inspecting this file: |
| 1807 |
.IP |
| 1808 |
.in +4n |
| 1809 |
.EX |
| 1810 |
.RB $ " cat /sys/kernel/cgroup/delegate" |
| 1811 |
cgroup.procs |
| 1812 |
cgroup.subtree_control |
| 1813 |
cgroup.threads |
| 1814 |
.EE |
| 1815 |
.in |
| 1816 |
.TP |
| 1817 |
.IR /sys/kernel/cgroup/features " (since Linux 4.15)" |
| 1818 |
.\" commit 5f2e673405b742be64e7c3604ed4ed3ac14f35ce |
| 1819 |
Over time, the set of cgroups v2 features that are provided by the |
| 1820 |
kernel may change or grow, |
| 1821 |
or some features may not be enabled by default. |
| 1822 |
This file provides a way for user-space applications to discover what |
| 1823 |
features the running kernel supports and has enabled. |
| 1824 |
Features are listed one per line: |
| 1825 |
.IP |
| 1826 |
.in +4n |
| 1827 |
.EX |
| 1828 |
.RB $ " cat /sys/kernel/cgroup/features" |
| 1829 |
nsdelegate |
| 1830 |
memory_localevents |
| 1831 |
.EE |
| 1832 |
.in |
| 1833 |
.IP |
| 1834 |
The entries that can appear in this file are: |
| 1835 |
.RS |
| 1836 |
.TP |
| 1837 |
.IR memory_localevents " (since Linux 5.2)" |
| 1838 |
The kernel supports the |
| 1839 |
.I memory_localevents |
| 1840 |
mount option. |
| 1841 |
.TP |
| 1842 |
.IR nsdelegate " (since Linux 4.15)" |
| 1843 |
The kernel supports the |
| 1844 |
.I nsdelegate |
| 1845 |
mount option. |
| 1846 |
.TP |
| 1847 |
.IR memory_recursiveprot " (since Linux 5.7)" |
| 1848 |
.\" commit 8a931f801340c2be10552c7b5622d5f4852f3a36 |
| 1849 |
The kernel supports the |
| 1850 |
.I memory_recursiveprot |
| 1851 |
mount option. |
| 1852 |
.RE |
| 1853 |
.SH SEE ALSO |
| 1854 |
.BR prlimit (1), |
| 1855 |
.BR systemd (1), |
| 1856 |
.BR systemd\-cgls (1), |
| 1857 |
.BR systemd\-cgtop (1), |
| 1858 |
.BR clone (2), |
| 1859 |
.BR ioprio_set (2), |
| 1860 |
.BR perf_event_open (2), |
| 1861 |
.BR setrlimit (2), |
| 1862 |
.BR cgroup_namespaces (7), |
| 1863 |
.BR cpuset (7), |
| 1864 |
.BR namespaces (7), |
| 1865 |
.BR sched (7), |
| 1866 |
.BR user_namespaces (7) |
| 1867 |
.P |
| 1868 |
The kernel source file |
| 1869 |
.IR Documentation/admin\-guide/cgroup\-v2.rst . |