Debian Codegraph @

upstream/debian-unstable/man7/cgroups.7

raw
1 .\" Copyright 2016-2017, Michael Kerrisk <[email protected]>
2 .\" Copyright, the authors of the Linux man-pages project
3 .\"
4 .\" SPDX-License-Identifier: Linux-man-pages-copyleft
5 .\"
6 .TH cgroups 7 2026-02-08 "Linux man-pages 6.18"
7 .SH NAME
8 cgroups \- Linux control groups
9 .SH DESCRIPTION
10 Control groups, usually referred to as cgroups,
11 are a Linux kernel feature which allow processes to
12 be organized into hierarchical groups whose usage of
13 various types of resources can then be limited and monitored.
14 The kernel's cgroup interface is provided through
15 a pseudo-filesystem called cgroupfs.
16 Grouping is implemented in the core cgroup kernel code,
17 while resource tracking and limits are implemented in
18 a set of per-resource-type subsystems (memory, CPU, and so on).
19 .\"
20 .SS Terminology
21 A
22 .I cgroup
23 is a collection of processes that are bound to a set of
24 limits or parameters defined via the cgroup filesystem.
25 .P
26 A
27 .I subsystem
28 is a kernel component that modifies the behavior of
29 the processes in a cgroup.
30 Various subsystems have been implemented, making it possible to do things
31 such as limiting the amount of CPU time and memory available to a cgroup,
32 accounting for the CPU time used by a cgroup,
33 and freezing and resuming execution of the processes in a cgroup.
34 Subsystems are sometimes also known as
35 .I resource controllers
36 (or simply, controllers).
37 .P
38 The cgroups for a controller are arranged in a
39 .IR hierarchy .
40 This hierarchy is defined by creating, removing, and
41 renaming subdirectories within the cgroup filesystem.
42 At each level of the hierarchy, attributes (e.g., limits) can be defined.
43 The limits, control, and accounting provided by cgroups generally have
44 effect throughout the subhierarchy underneath the cgroup where the
45 attributes are defined.
46 Thus, for example, the limits placed on
47 a cgroup at a higher level in the hierarchy cannot be exceeded
48 by descendant cgroups.
49 .\"
50 .SS Cgroups version 1 and version 2
51 The initial release of the cgroups implementation was in Linux 2.6.24.
52 Over time, various cgroup controllers have been added
53 to allow the management of various types of resources.
54 However, the development of these controllers was largely uncoordinated,
55 with the result that many inconsistencies arose between controllers
56 and management of the cgroup hierarchies became rather complex.
57 A longer description of these problems can be found in the kernel
58 source file
59 .I Documentation/admin\-guide/cgroup\-v2.rst
60 (or
61 .I Documentation/cgroup\-v2.rst
62 in Linux 4.17 and earlier).
63 .P
64 Because of the problems with the initial cgroups implementation
65 (cgroups version 1),
66 starting in Linux 3.10, work began on a new,
67 orthogonal implementation to remedy these problems.
68 Initially marked experimental, and hidden behind the
69 .I "\-o\ __DEVEL__sane_behavior"
70 mount option, the new version (cgroups version 2)
71 was eventually made official with the release of Linux 4.5.
72 Differences between the two versions are described in the text below.
73 The file
74 .IR cgroup.sane_behavior ,
75 present in cgroups v1, is a relic of this mount option.
76 The file always reports "0" and is only retained for backward compatibility.
77 .P
78 Although cgroups v2 is intended as a replacement for cgroups v1,
79 the older system continues to exist
80 (and for compatibility reasons is unlikely to be removed).
81 Currently, cgroups v2 implements only a subset of the controllers
82 available in cgroups v1.
83 The two systems are implemented so that both v1 controllers and
84 v2 controllers can be mounted on the same system.
85 Thus, for example, it is possible to use those controllers
86 that are supported under version 2,
87 while also using version 1 controllers
88 where version 2 does not yet support those controllers.
89 The only restriction here is that a controller can't be simultaneously
90 employed in both a cgroups v1 hierarchy and in the cgroups v2 hierarchy.
91 .\"
92 .SH CGROUPS VERSION 1
93 Under cgroups v1, each controller may be mounted against a separate
94 cgroup filesystem that provides its own hierarchical organization of the
95 processes on the system.
96 It is also possible to comount multiple (or even all) cgroups v1 controllers
97 against the same cgroup filesystem, meaning that the comounted controllers
98 manage the same hierarchical organization of processes.
99 .P
100 For each mounted hierarchy,
101 the directory tree mirrors the control group hierarchy.
102 Each control group is represented by a directory, with each of its child
103 control cgroups represented as a child directory.
104 For instance,
105 .I /user/joe/1.session
106 represents control group
107 .IR 1.session ,
108 which is a child of cgroup
109 .IR joe ,
110 which is a child of
111 .IR /user .
112 Under each cgroup directory is a set of files which can be read or
113 written to, reflecting resource limits and a few general cgroup
114 properties.
115 .\"
116 .SS Tasks (threads) versus processes
117 In cgroups v1, a distinction is drawn between
118 .I processes
119 and
120 .IR tasks .
121 In this view, a process can consist of multiple tasks
122 (more commonly called threads, from a user-space perspective,
123 and called such in the remainder of this man page).
124 In cgroups v1, it is possible to independently manipulate
125 the cgroup memberships of the threads in a process.
126 .P
127 The cgroups v1 ability to split threads across different cgroups
128 caused problems in some cases.
129 For example, it made no sense for the
130 .I memory
131 controller,
132 since all of the threads of a process share a single address space.
133 Because of these problems,
134 the ability to independently manipulate the cgroup memberships
135 of the threads in a process was removed in the initial cgroups v2
136 implementation, and subsequently restored in a more limited form
137 (see the discussion of "thread mode" below).
138 .\"
139 .SS Mounting v1 controllers
140 The use of cgroups requires a kernel built with the
141 .B CONFIG_CGROUP
142 option.
143 In addition, each of the v1 controllers has an associated
144 configuration option that must be set in order to employ that controller.
145 .P
146 In order to use a v1 controller,
147 it must be mounted against a cgroup filesystem.
148 The usual place for such mounts is under a
149 .BR tmpfs (5)
150 filesystem mounted at
151 .IR /sys/fs/cgroup .
152 Thus, one might mount the
153 .I cpu
154 controller as follows:
155 .P
156 .in +4n
157 .EX
158 mount \-t cgroup \-o cpu none /sys/fs/cgroup/cpu
159 .EE
160 .in
161 .P
162 It is possible to comount multiple controllers against the same hierarchy.
163 For example, here the
164 .I cpu
165 and
166 .I cpuacct
167 controllers are comounted against a single hierarchy:
168 .P
169 .in +4n
170 .EX
171 mount \-t cgroup \-o cpu,cpuacct none /sys/fs/cgroup/cpu,cpuacct
172 .EE
173 .in
174 .P
175 Comounting controllers has the effect that a process is in the same cgroup for
176 all of the comounted controllers.
177 Separately mounting controllers allows a process to
178 be in cgroup
179 .I /foo1
180 for one controller while being in
181 .I /foo2/foo3
182 for another.
183 .P
184 It is possible to comount all v1 controllers against the same hierarchy:
185 .P
186 .in +4n
187 .EX
188 mount \-t cgroup \-o all cgroup /sys/fs/cgroup
189 .EE
190 .in
191 .P
192 (One can achieve the same result by omitting
193 .IR "\-o all" ,
194 since it is the default if no controllers are explicitly specified.)
195 .P
196 It is not possible to mount the same controller
197 against multiple cgroup hierarchies.
198 For example, it is not possible to mount both the
199 .I cpu
200 and
201 .I cpuacct
202 controllers against one hierarchy, and to mount the
203 .I cpu
204 controller alone against another hierarchy.
205 It is possible to create multiple mounts with exactly
206 the same set of comounted controllers.
207 However, in this case all that results is multiple mount points
208 providing a view of the same hierarchy.
209 .P
210 Note that on many systems, the v1 controllers are automatically mounted under
211 .IR /sys/fs/cgroup ;
212 in particular,
213 .BR systemd (1)
214 automatically creates such mounts.
215 .\"
216 .SS Unmounting v1 controllers
217 A mounted cgroup filesystem can be unmounted using the
218 .BR umount (8)
219 command, as in the following example:
220 .P
221 .in +4n
222 .EX
223 umount /sys/fs/cgroup/pids
224 .EE
225 .in
226 .P
227 .IR "But note well" :
228 a cgroup filesystem is unmounted only if it is not busy,
229 that is, it has no child cgroups.
230 If this is not the case, then the only effect of the
231 .BR umount (8)
232 is to make the mount invisible.
233 Thus, to ensure that the mount is really removed,
234 one must first remove all child cgroups,
235 which in turn can be done only after all member processes
236 have been moved from those cgroups to the root cgroup.
237 .\"
238 .SS Cgroups version 1 controllers
239 Each of the cgroups version 1 controllers is governed
240 by a kernel configuration option (listed below).
241 Additionally, the availability of the cgroups feature is governed by the
242 .B CONFIG_CGROUPS
243 kernel configuration option.
244 .TP
245 .IR cpu " (since Linux 2.6.24; " \f[B]CONFIG_CGROUP_SCHED\f[] )
246 Cgroups can be guaranteed a minimum number of "CPU shares"
247 when a system is busy.
248 This does not limit a cgroup's CPU usage if the CPUs are not busy.
249 For further information, see
250 .IR Documentation/scheduler/sched\-design\-CFS.rst .
251 .IP
252 In Linux 3.2,
253 this controller was extended to provide CPU "bandwidth" control.
254 If the kernel is configured with
255 .BR CONFIG_CFS_BANDWIDTH ,
256 then within each scheduling period
257 (defined via a file in the cgroup directory), it is possible to define
258 an upper limit on the CPU time allocated to the processes in a cgroup.
259 This upper limit applies even if there is no other competition for the CPU.
260 Further information can be found in the kernel source file
261 .IR Documentation/scheduler/sched\-bwc.rst .
262 .TP
263 .IR cpuacct " (since Linux 2.6.24; " \f[B]CONFIG_CGROUP_CPUACCT\f[] )
264 This provides accounting for CPU usage by groups of processes.
265 .IP
266 Further information can be found in the kernel source file
267 .IR Documentation/admin\-guide/cgroup\-v1/cpuacct.rst .
268 .TP
269 .IR cpuset " (since Linux 2.6.24; " \f[B]CONFIG_CPUSETS\f[] )
270 This cgroup can be used to bind the processes in a cgroup to
271 a specified set of CPUs and NUMA nodes.
272 .IP
273 Further information can be found in the kernel source file
274 .IR Documentation/admin\-guide/cgroup\-v1/cpusets.rst .
275 .
276 .TP
277 .IR memory " (since Linux 2.6.25; " \f[B]CONFIG_MEMCG\f[] )
278 The memory controller supports reporting and limiting of process memory, kernel
279 memory, and swap used by cgroups.
280 .IP
281 Further information can be found in the kernel source file
282 .IR Documentation/admin\-guide/cgroup\-v1/memory.rst .
283 .TP
284 .IR devices " (since Linux 2.6.26; " \f[B]CONFIG_CGROUP_DEVICE\f[] )
285 This supports controlling which processes may create (mknod) devices as
286 well as open them for reading or writing.
287 The policies may be specified as allow-lists and deny-lists.
288 Hierarchy is enforced, so new rules must not
289 violate existing rules for the target or ancestor cgroups.
290 .IP
291 Further information can be found in the kernel source file
292 .IR Documentation/admin\-guide/cgroup\-v1/devices.rst .
293 .TP
294 .IR freezer " (since Linux 2.6.28; " \f[B]CONFIG_CGROUP_FREEZER\f[] )
295 The
296 .I freezer
297 cgroup can suspend and restore (resume) all processes in a cgroup.
298 Freezing a cgroup
299 .I /A
300 also causes its children, for example, processes in
301 .IR /A/B ,
302 to be frozen.
303 .IP
304 Further information can be found in the kernel source file
305 .IR Documentation/admin\-guide/cgroup\-v1/freezer\-subsystem.rst .
306 .TP
307 .IR net_cls " (since Linux 2.6.29; " \f[B]CONFIG_CGROUP_NET_CLASSID\f[] )
308 This places a classid, specified for the cgroup, on network packets
309 created by a cgroup.
310 These classids can then be used in firewall rules,
311 as well as used to shape traffic using
312 .BR tc (8).
313 This applies only to packets
314 leaving the cgroup, not to traffic arriving at the cgroup.
315 .IP
316 Further information can be found in the kernel source file
317 .IR Documentation/admin\-guide/cgroup\-v1/net_cls.rst .
318 .TP
319 .IR blkio " (since Linux 2.6.33; " \f[B]CONFIG_BLK_CGROUP\f[] )
320 The
321 .I blkio
322 cgroup controls and limits access to specified block devices by
323 applying IO control in the form of throttling and upper limits against leaf
324 nodes and intermediate nodes in the storage hierarchy.
325 .IP
326 Two policies are available.
327 The first is a proportional-weight time-based division
328 of disk implemented with CFQ.
329 This is in effect for leaf nodes using CFQ.
330 The second is a throttling policy which specifies
331 upper I/O rate limits on a device.
332 .IP
333 Further information can be found in the kernel source file
334 .IR Documentation/admin\-guide/cgroup\-v1/blkio\-controller.rst .
335 .TP
336 .IR perf_event " (since Linux 2.6.39; " \f[B]CONFIG_CGROUP_PERF\f[] )
337 This controller allows
338 .I perf
339 monitoring of the set of processes grouped in a cgroup.
340 .IP
341 Further information can be found in the kernel source files
342 .TP
343 .IR net_prio " (since Linux 3.3; " \f[B]CONFIG_CGROUP_NET_PRIO\f[] )
344 This allows priorities to be specified, per network interface, for cgroups.
345 .IP
346 Further information can be found in the kernel source file
347 .IR Documentation/admin\-guide/cgroup\-v1/net_prio.rst .
348 .TP
349 .IR hugetlb " (since Linux 3.5; " \f[B]CONFIG_CGROUP_HUGETLB\f[] )
350 This supports limiting the use of huge pages by cgroups.
351 .IP
352 Further information can be found in the kernel source file
353 .IR Documentation/admin\-guide/cgroup\-v1/hugetlb.rst .
354 .TP
355 .IR pids " (since Linux 4.3; " \f[B]CONFIG_CGROUP_PIDS\f[] )
356 This controller permits limiting the number of process that may be created
357 in a cgroup (and its descendants).
358 .IP
359 Further information can be found in the kernel source file
360 .IR Documentation/admin\-guide/cgroup\-v1/pids.rst .
361 .TP
362 .IR rdma " (since Linux 4.11; " \f[B]CONFIG_CGROUP_RDMA\f[] )
363 The RDMA controller permits limiting the use of
364 RDMA/IB-specific resources per cgroup.
365 .IP
366 Further information can be found in the kernel source file
367 .IR Documentation/admin\-guide/cgroup\-v1/rdma.rst .
368 .\"
369 .SS Creating cgroups and moving processes
370 A cgroup filesystem initially contains a single root cgroup, '/',
371 which all processes belong to.
372 A new cgroup is created by creating a directory in the cgroup filesystem:
373 .P
374 .in +4n
375 .EX
376 mkdir /sys/fs/cgroup/cpu/cg1
377 .EE
378 .in
379 .P
380 This creates a new empty cgroup.
381 .P
382 A process may be moved to this cgroup by writing its PID into the cgroup's
383 .I cgroup.procs
384 file:
385 .P
386 .in +4n
387 .EX
388 echo $$ > /sys/fs/cgroup/cpu/cg1/cgroup.procs
389 .EE
390 .in
391 .P
392 Only one PID at a time should be written to this file.
393 .P
394 Writing the value 0 to a
395 .I cgroup.procs
396 file causes the writing process to be moved to the corresponding cgroup.
397 .P
398 When writing a PID into the
399 .IR cgroup.procs ,
400 all threads in the process are moved into the new cgroup at once.
401 .P
402 Within a hierarchy, a process can be a member of exactly one cgroup.
403 Writing a process's PID to a
404 .I cgroup.procs
405 file automatically removes it from the cgroup of
406 which it was previously a member.
407 .P
408 The
409 .I cgroup.procs
410 file can be read to obtain a list of the processes that are
411 members of a cgroup.
412 The returned list of PIDs is not guaranteed to be in order.
413 Nor is it guaranteed to be free of duplicates.
414 (For example, a PID may be recycled while reading from the list.)
415 .P
416 In cgroups v1, an individual thread can be moved to
417 another cgroup by writing its thread ID
418 (i.e., the kernel thread ID returned by
419 .BR clone (2)
420 and
421 .BR gettid (2))
422 to the
423 .I tasks
424 file in a cgroup directory.
425 This file can be read to discover the set of threads
426 that are members of the cgroup.
427 .\"
428 .SS Removing cgroups
429 To remove a cgroup,
430 it must first have no child cgroups and contain no (nonzombie) processes.
431 So long as that is the case, one can simply
432 remove the corresponding directory pathname.
433 Note that files in a cgroup directory cannot and need not be
434 removed.
435 .\"
436 .SS Cgroups v1 release notification
437 Two files can be used to determine whether the kernel provides
438 notifications when a cgroup becomes empty.
439 A cgroup is considered to be empty when it contains no child
440 cgroups and no member processes.
441 .P
442 A special file in the root directory of each cgroup hierarchy,
443 .IR release_agent ,
444 can be used to register the pathname of a program that may be invoked when
445 a cgroup in the hierarchy becomes empty.
446 The pathname of the newly empty cgroup (relative to the cgroup mount point)
447 is provided as the sole command-line argument when the
448 .I release_agent
449 program is invoked.
450 The
451 .I release_agent
452 program might remove the cgroup directory,
453 or perhaps repopulate it with a process.
454 .P
455 The default value of the
456 .I release_agent
457 file is empty, meaning that no release agent is invoked.
458 .P
459 The content of the
460 .I release_agent
461 file can also be specified via a mount option when the
462 cgroup filesystem is mounted:
463 .P
464 .in +4n
465 .EX
466 mount \-o release_agent=pathname ...
467 .EE
468 .in
469 .P
470 Whether or not the
471 .I release_agent
472 program is invoked when a particular cgroup becomes empty is determined
473 by the value in the
474 .I notify_on_release
475 file in the corresponding cgroup directory.
476 If this file contains the value 0, then the
477 .I release_agent
478 program is not invoked.
479 If it contains the value 1, the
480 .I release_agent
481 program is invoked.
482 The default value for this file in the root cgroup is 0.
483 At the time when a new cgroup is created,
484 the value in this file is inherited from the corresponding file
485 in the parent cgroup.
486 .\"
487 .SS Cgroup v1 named hierarchies
488 In cgroups v1,
489 it is possible to mount a cgroup hierarchy that has no attached controllers:
490 .P
491 .in +4n
492 .EX
493 mount \-t cgroup \-o none,name=somename none /some/mount/point
494 .EE
495 .in
496 .P
497 Multiple instances of such hierarchies can be mounted;
498 each hierarchy must have a unique name.
499 The only purpose of such hierarchies is to track processes.
500 (See the discussion of release notification below.)
501 An example of this is the
502 .I name=systemd
503 cgroup hierarchy that is used by
504 .BR systemd (1)
505 to track services and user sessions.
506 .P
507 Since Linux 5.0, the
508 .I cgroup_no_v1
509 kernel boot option (described below) can be used to disable cgroup v1
510 named hierarchies, by specifying
511 .IR cgroup_no_v1=named .
512 .\"
513 .SH CGROUPS VERSION 2
514 In cgroups v2,
515 all mounted controllers reside in a single unified hierarchy.
516 While (different) controllers may be simultaneously
517 mounted under the v1 and v2 hierarchies,
518 it is not possible to mount the same controller simultaneously
519 under both the v1 and the v2 hierarchies.
520 .P
521 The new behaviors in cgroups v2 are summarized here,
522 and in some cases elaborated in the following subsections.
523 .IP \[bu] 3
524 Cgroups v2 provides a unified hierarchy against
525 which all controllers are mounted.
526 .IP \[bu]
527 "Internal" processes are not permitted.
528 With the exception of the root cgroup, processes may reside
529 only in leaf nodes (cgroups that do not themselves contain child cgroups).
530 The details are somewhat more subtle than this, and are described below.
531 .IP \[bu]
532 Active cgroups must be specified via the files
533 .I cgroup.controllers
534 and
535 .IR cgroup.subtree_control .
536 .IP \[bu]
537 The
538 .I tasks
539 file has been removed.
540 In addition, the
541 .I cgroup.clone_children
542 file that is employed by the
543 .I cpuset
544 controller has been removed.
545 .IP \[bu]
546 An improved mechanism for notification of empty cgroups is provided by the
547 .I cgroup.events
548 file.
549 .P
550 For more changes, see the
551 .I Documentation/admin\-guide/cgroup\-v2.rst
552 file in the kernel source.
553 .
554 .P
555 Some of the new behaviors listed above saw subsequent modification with
556 the addition in Linux 4.14 of "thread mode" (described below).
557 .\"
558 .SS Cgroups v2 unified hierarchy
559 In cgroups v1, the ability to mount different controllers
560 against different hierarchies was intended to allow great flexibility
561 for application design.
562 In practice, though,
563 the flexibility turned out to be less useful than expected,
564 and in many cases added complexity.
565 Therefore, in cgroups v2,
566 all available controllers are mounted against a single hierarchy.
567 The available controllers are automatically mounted,
568 meaning that it is not necessary (or possible) to specify the controllers
569 when mounting the cgroup v2 filesystem using a command such as the following:
570 .P
571 .in +4n
572 .EX
573 mount \-t cgroup2 none /mnt/cgroup2
574 .EE
575 .in
576 .P
577 A cgroup v2 controller is available only if it is not currently in use
578 via a mount against a cgroup v1 hierarchy.
579 Or, to put things another way, it is not possible to employ
580 the same controller against both a v1 hierarchy and the unified v2 hierarchy.
581 This means that it may be necessary first to unmount a v1 controller
582 (as described above) before that controller is available in v2.
583 Since
584 .BR systemd (1)
585 makes heavy use of some v1 controllers by default,
586 it can in some cases be simpler to boot the system with
587 selected v1 controllers disabled.
588 To do this, specify the
589 .I cgroup_no_v1=list
590 option on the kernel boot command line;
591 .I list
592 is a comma-separated list of the names of the controllers to disable,
593 or the word
594 .I all
595 to disable all v1 controllers.
596 (This situation is correctly handled by
597 .BR systemd (1),
598 which falls back to operating without the specified controllers.)
599 .P
600 Note that on many modern systems,
601 .BR systemd (1)
602 automatically mounts the
603 .I cgroup2
604 filesystem at
605 .I /sys/fs/cgroup/unified
606 during the boot process.
607 .\"
608 .SS Cgroups v2 mount options
609 The following options
610 .RI ( mount\~\-o )
611 can be specified when mounting the group v2 filesystem:
612 .TP
613 .IR nsdelegate " (since Linux 4.15)"
614 Treat cgroup namespaces as delegation boundaries.
615 For details, see below.
616 .TP
617 .IR memory_localevents " (since Linux 5.2)"
618 .\" commit 9852ae3fe5293264f01c49f2571ef7688f7823ce
619 The
620 .I memory.events
621 should show statistics only for the cgroup itself,
622 and not for any descendant cgroups.
623 This was the behavior before Linux 5.2.
624 Since Linux 5.2,
625 the default behavior is to include statistics for descendant cgroups in
626 .IR memory.events ,
627 and this mount option can be used to revert to the legacy behavior.
628 This option is system wide and can be set on mount or
629 modified through remount only from the initial mount namespace;
630 it is silently ignored in noninitial namespaces.
631 .\"
632 .SS Cgroups v2 controllers
633 The following controllers, documented in the kernel source file
634 .IR Documentation/admin\-guide/cgroup\-v2.rst ,
635 are supported in cgroups version 2:
636 .TP
637 .IR cpu " (since Linux 4.15)"
638 This is the successor to the version 1
639 .I cpu
640 and
641 .I cpuacct
642 controllers.
643 .TP
644 .IR cpuset " (since Linux 5.0)"
645 This is the successor of the version 1
646 .I cpuset
647 controller.
648 .TP
649 .IR freezer " (since Linux 5.2)"
650 .\" commit 76f969e8948d82e78e1bc4beb6b9465908e74873
651 This is the successor of the version 1
652 .I freezer
653 controller.
654 .TP
655 .IR hugetlb " (since Linux 5.6)"
656 This is the successor of the version 1
657 .I hugetlb
658 controller.
659 .TP
660 .IR io " (since Linux 4.5)"
661 This is the successor of the version 1
662 .I blkio
663 controller.
664 .TP
665 .IR memory " (since Linux 4.5)"
666 This is the successor of the version 1
667 .I memory
668 controller.
669 .TP
670 .IR perf_event " (since Linux 4.11)"
671 This is the same as the version 1
672 .I perf_event
673 controller.
674 .TP
675 .IR pids " (since Linux 4.5)"
676 This is the same as the version 1
677 .I pids
678 controller.
679 .TP
680 .IR rdma " (since Linux 4.11)"
681 This is the same as the version 1
682 .I rdma
683 controller.
684 .P
685 There is no direct equivalent of the
686 .I net_cls
687 and
688 .I net_prio
689 controllers from cgroups version 1.
690 Instead, support has been added to
691 .BR iptables (8)
692 to allow eBPF filters that hook on cgroup v2 pathnames to make decisions
693 about network traffic on a per-cgroup basis.
694 .P
695 The v2
696 .I devices
697 controller provides no interface files;
698 instead, device control is gated by attaching an eBPF
699 .RB ( BPF_CGROUP_DEVICE )
700 program to a v2 cgroup.
701 .\"
702 .SS Cgroups v2 subtree control
703 Each cgroup in the v2 hierarchy contains the following two files:
704 .TP
705 .I cgroup.controllers
706 This read-only file exposes a list of the controllers that are
707 .I available
708 in this cgroup.
709 The contents of this file match the contents of the
710 .I cgroup.subtree_control
711 file in the parent cgroup.
712 .TP
713 .I cgroup.subtree_control
714 This is a list of controllers that are
715 .I active
716 .RI ( enabled )
717 in the cgroup.
718 The set of controllers in this file is a subset of the set in the
719 .I cgroup.controllers
720 of this cgroup.
721 The set of active controllers is modified by writing strings to this file
722 containing space-delimited controller names,
723 each preceded by '+' (to enable a controller)
724 or '\-' (to disable a controller), as in the following example:
725 .IP
726 .in +4n
727 .EX
728 echo \[aq]+pids \-memory\[aq] > x/y/cgroup.subtree_control
729 .EE
730 .in
731 .IP
732 An attempt to enable a controller
733 that is not present in
734 .I cgroup.controllers
735 leads to an
736 .B ENOENT
737 error when writing to the
738 .I cgroup.subtree_control
739 file.
740 .P
741 Because the list of controllers in
742 .I cgroup.subtree_control
743 is a subset of those
744 .IR cgroup.controllers ,
745 a controller that has been disabled in one cgroup in the hierarchy
746 can never be re-enabled in the subtree below that cgroup.
747 .P
748 A cgroup's
749 .I cgroup.subtree_control
750 file determines the set of controllers that are exercised in the
751 .I child
752 cgroups.
753 When a controller (e.g.,
754 .IR pids )
755 is present in the
756 .I cgroup.subtree_control
757 file of a parent cgroup,
758 then the corresponding controller-interface files (e.g.,
759 .IR pids.max )
760 are automatically created in the children of that cgroup
761 and can be used to exert resource control in the child cgroups.
762 .\"
763 .SS Cgroups v2 \[dq]no internal processes\[dq] rule
764 Cgroups v2 enforces a so-called "no internal processes" rule.
765 Roughly speaking, this rule means that,
766 with the exception of the root cgroup, processes may reside
767 only in leaf nodes (cgroups that do not themselves contain child cgroups).
768 This avoids the need to decide how to partition resources between
769 processes which are members of cgroup A and processes in child cgroups of A.
770 .P
771 For instance, if cgroup
772 .I /cg1/cg2
773 exists, then a process may reside in
774 .IR /cg1/cg2 ,
775 but not in
776 .IR /cg1 .
777 This is to avoid an ambiguity in cgroups v1
778 with respect to the delegation of resources between processes in
779 .I /cg1
780 and its child cgroups.
781 The recommended approach in cgroups v2 is to create a subdirectory called
782 .I leaf
783 for any nonleaf cgroup which should contain processes, but no child cgroups.
784 Thus, processes which previously would have gone into
785 .I /cg1
786 would now go into
787 .IR /cg1/leaf .
788 This has the advantage of making explicit
789 the relationship between processes in
790 .I /cg1/leaf
791 and
792 .IR /cg1 's
793 other children.
794 .P
795 The "no internal processes" rule is in fact more subtle than stated above.
796 More precisely, the rule is that a (nonroot) cgroup can't both
797 (1) have member processes, and
798 (2) distribute resources into child cgroups\[em]that is, have a nonempty
799 .I cgroup.subtree_control
800 file.
801 Thus, it
802 .I is
803 possible for a cgroup to have both member processes and child cgroups,
804 but before controllers can be enabled for that cgroup,
805 the member processes must be moved out of the cgroup
806 (e.g., perhaps into the child cgroups).
807 .P
808 With the Linux 4.14 addition of "thread mode" (described below),
809 the "no internal processes" rule has been relaxed in some cases.
810 .\"
811 .SS Cgroups v2 cgroup.events file
812 Each nonroot cgroup in the v2 hierarchy contains a read-only file,
813 .IR cgroup.events ,
814 whose contents are key-value pairs
815 (delimited by newline characters, with the key and value separated by spaces)
816 providing state information about the cgroup:
817 .P
818 .in +4n
819 .EX
820 .RB $ " cat mygrp/cgroup.events"
821 populated 1
822 frozen 0
823 .EE
824 .in
825 .P
826 The following keys may appear in this file:
827 .TP
828 .I populated
829 The value of this key is either 1,
830 if this cgroup or any of its descendants has member processes,
831 or otherwise 0.
832 .TP
833 .IR frozen " (since Linux 5.2)"
834 .\" commit 76f969e8948d82e78e1bc4beb6b9465908e7487
835 The value of this key is 1 if this cgroup is currently frozen,
836 or 0 if it is not.
837 .P
838 The
839 .I cgroup.events
840 file can be monitored, in order to receive notification when the value of
841 one of its keys changes.
842 Such monitoring can be done using
843 .BR inotify (7),
844 which notifies changes as
845 .B IN_MODIFY
846 events, or
847 .BR poll (2),
848 which notifies changes by returning the
849 .B POLLPRI
850 and
851 .B POLLERR
852 bits in the
853 .I revents
854 field.
855 .\"
856 .SS Cgroup v2 release notification
857 Cgroups v2 provides a new mechanism for obtaining notification
858 when a cgroup becomes empty.
859 The cgroups v1
860 .I release_agent
861 and
862 .I notify_on_release
863 files are removed, and replaced by the
864 .I populated
865 key in the
866 .I cgroup.events
867 file.
868 This key either has the value 0,
869 meaning that the cgroup (and its descendants)
870 contain no (nonzombie) member processes,
871 or 1, meaning that the cgroup (or one of its descendants)
872 contains member processes.
873 .P
874 The cgroups v2 release-notification mechanism
875 offers the following advantages over the cgroups v1
876 .I release_agent
877 mechanism:
878 .IP \[bu] 3
879 It allows for cheaper notification,
880 since a single process can monitor multiple
881 .I cgroup.events
882 files (using the techniques described earlier).
883 By contrast, the cgroups v1 mechanism requires the expense of creating
884 a process for each notification.
885 .IP \[bu]
886 Notification for different cgroup subhierarchies can be delegated
887 to different processes.
888 By contrast, the cgroups v1 mechanism allows only one release agent
889 for an entire hierarchy.
890 .\"
891 .SS Cgroups v2 cgroup.stat file
892 .\" commit ec39225cca42c05ac36853d11d28f877fde5c42e
893 Each cgroup in the v2 hierarchy contains a read-only
894 .I cgroup.stat
895 file (first introduced in Linux 4.14)
896 that consists of lines containing key-value pairs.
897 The following keys currently appear in this file:
898 .TP
899 .I nr_descendants
900 This is the total number of visible (i.e., living) descendant cgroups
901 underneath this cgroup.
902 .TP
903 .I nr_dying_descendants
904 This is the total number of dying descendant cgroups
905 underneath this cgroup.
906 A cgroup enters the dying state after being deleted.
907 It remains in that state for an undefined period
908 (which will depend on system load)
909 while resources are freed before the cgroup is destroyed.
910 Note that the presence of some cgroups in the dying state is normal,
911 and is not indicative of any problem.
912 .IP
913 A process can't be made a member of a dying cgroup,
914 and a dying cgroup can't be brought back to life.
915 .\"
916 .SS Limiting the number of descendant cgroups
917 Each cgroup in the v2 hierarchy contains the following files,
918 which can be used to view and set limits on the number
919 of descendant cgroups under that cgroup:
920 .TP
921 .IR cgroup.max.depth " (since Linux 4.14)"
922 .\" commit 1a926e0bbab83bae8207d05a533173425e0496d1
923 This file defines a limit on the depth of nesting of descendant cgroups.
924 A value of 0 in this file means that no descendant cgroups can be created.
925 An attempt to create a descendant whose nesting level exceeds
926 the limit fails
927 .RI ( mkdir (2)
928 fails with the error
929 .BR EAGAIN ).
930 .IP
931 Writing the string
932 .I \[dq]max\[dq]
933 to this file means that no limit is imposed.
934 The default value in this file is
935 .IR \[dq]max\[dq] .
936 .TP
937 .IR cgroup.max.descendants " (since Linux 4.14)"
938 .\" commit 1a926e0bbab83bae8207d05a533173425e0496d1
939 This file defines a limit on the number of live descendant cgroups that
940 this cgroup may have.
941 An attempt to create more descendants than allowed by the limit fails
942 .RI ( mkdir (2)
943 fails with the error
944 .BR EAGAIN ).
945 .IP
946 Writing the string
947 .I \[dq]max\[dq]
948 to this file means that no limit is imposed.
949 The default value in this file is
950 .IR \[dq]max\[dq] .
951 .\"
952 .SH CGROUPS DELEGATION: DELEGATING A HIERARCHY TO A LESS PRIVILEGED USER
953 In the context of cgroups,
954 delegation means passing management of some subtree
955 of the cgroup hierarchy to a nonprivileged user.
956 Cgroups v1 provides support for delegation based on file permissions
957 in the cgroup hierarchy but with less strict containment rules than v2
958 (as noted below).
959 Cgroups v2 supports delegation with containment by explicit design.
960 The focus of the discussion in this section is on delegation in cgroups v2,
961 with some differences for cgroups v1 noted along the way.
962 .P
963 Some terminology is required in order to describe delegation.
964 A
965 .I delegater
966 is a privileged user (i.e., root) who owns a parent cgroup.
967 A
968 .I delegatee
969 is a nonprivileged user who will be granted the permissions needed
970 to manage some subhierarchy under that parent cgroup,
971 known as the
972 .IR "delegated subtree" .
973 .P
974 To perform delegation,
975 the delegater makes certain directories and files writable by the delegatee,
976 typically by changing the ownership of the objects to be the user ID
977 of the delegatee.
978 Assuming that we want to delegate the hierarchy rooted at (say)
979 .I /dlgt_grp
980 and that there are not yet any child cgroups under that cgroup,
981 the ownership of the following is changed to the user ID of the delegatee:
982 .TP
983 .I /dlgt_grp
984 Changing the ownership of the root of the subtree means that any new
985 cgroups created under the subtree (and the files they contain)
986 will also be owned by the delegatee.
987 .TP
988 .I /dlgt_grp/cgroup.procs
989 Changing the ownership of this file means that the delegatee
990 can move processes into the root of the delegated subtree.
991 .TP
992 .IR /dlgt_grp/cgroup.subtree_control " (cgroups v2 only)"
993 Changing the ownership of this file means that the delegatee
994 can enable controllers (that are present in
995 .IR /dlgt_grp/cgroup.controllers )
996 in order to further redistribute resources at lower levels in the subtree.
997 (As an alternative to changing the ownership of this file,
998 the delegater might instead add selected controllers to this file.)
999 .TP
1000 .IR /dlgt_grp/cgroup.threads " (cgroups v2 only)"
1001 Changing the ownership of this file is necessary if a threaded subtree
1002 is being delegated (see the description of "thread mode", below).
1003 This permits the delegatee to write thread IDs to the file.
1004 (The ownership of this file can also be changed when delegating
1005 a domain subtree, but currently this serves no purpose,
1006 since, as described below, it is not possible to move a thread between
1007 domain cgroups by writing its thread ID to the
1008 .I cgroup.threads
1009 file.)
1010 .IP
1011 In cgroups v1, the corresponding file that should instead be delegated is the
1012 .I tasks
1013 file.
1014 .P
1015 The delegater should
1016 .I not
1017 change the ownership of any of the controller interfaces files (e.g.,
1018 .IR pids.max ,
1019 .IR memory.high )
1020 in
1021 .IR dlgt_grp .
1022 Those files are used from the next level above the delegated subtree
1023 in order to distribute resources into the subtree,
1024 and the delegatee should not have permission to change
1025 the resources that are distributed into the delegated subtree.
1026 .P
1027 See also the discussion of the
1028 .I /sys/kernel/cgroup/delegate
1029 file in NOTES for information about further delegatable files in cgroups v2.
1030 .P
1031 After the aforementioned steps have been performed,
1032 the delegatee can create child cgroups within the delegated subtree
1033 (the cgroup subdirectories and the files they contain
1034 will be owned by the delegatee)
1035 and move processes between cgroups in the subtree.
1036 If some controllers are present in
1037 .IR dlgt_grp/cgroup.subtree_control ,
1038 or the ownership of that file was passed to the delegatee,
1039 the delegatee can also control the further redistribution
1040 of the corresponding resources into the delegated subtree.
1041 .\"
1042 .SS Cgroups v2 delegation: nsdelegate and cgroup namespaces
1043 Starting with Linux 4.13,
1044 .\" commit 5136f6365ce3eace5a926e10f16ed2a233db5ba9
1045 there is a second way to perform cgroup delegation in the cgroups v2 hierarchy.
1046 This is done by mounting or remounting the cgroup v2 filesystem with the
1047 .I nsdelegate
1048 mount option.
1049 For example, if the cgroup v2 filesystem has already been mounted,
1050 we can remount it with the
1051 .I nsdelegate
1052 option as follows:
1053 .P
1054 .in +4n
1055 .EX
1056 mount \-t cgroup2 \-o remount,nsdelegate \[rs]
1057 none /sys/fs/cgroup/unified
1058 .EE
1059 .in
1060 .\"
1061 .\" Alternatively, we could boot the kernel with the options:
1062 .\"
1063 .\" cgroup_no_v1=all systemd.legacy_systemd_cgroup_controller
1064 .\"
1065 .\" The effect of the latter option is to prevent systemd from employing
1066 .\" its "hybrid" cgroup mode, where it tries to make use of cgroups v2.
1067 .P
1068 The effect of this mount option is to cause cgroup namespaces
1069 to automatically become delegation boundaries.
1070 More specifically,
1071 the following restrictions apply for processes inside the cgroup namespace:
1072 .IP \[bu] 3
1073 Writes to controller interface files in the root directory of the namespace
1074 will fail with the error
1075 .BR EPERM .
1076 Processes inside the cgroup namespace can still write to delegatable
1077 files in the root directory of the cgroup namespace such as
1078 .I cgroup.procs
1079 and
1080 .IR cgroup.subtree_control ,
1081 and can create subhierarchy underneath the root directory.
1082 .IP \[bu]
1083 Attempts to migrate processes across the namespace boundary are denied
1084 (with the error
1085 .BR ENOENT ).
1086 Processes inside the cgroup namespace can still
1087 (subject to the containment rules described below)
1088 move processes between cgroups
1089 .I within
1090 the subhierarchy under the namespace root.
1091 .P
1092 The ability to define cgroup namespaces as delegation boundaries
1093 makes cgroup namespaces more useful.
1094 To understand why, suppose that we already have one cgroup hierarchy
1095 that has been delegated to a nonprivileged user,
1096 .IR cecilia ,
1097 using the older delegation technique described above.
1098 Suppose further that
1099 .I cecilia
1100 wanted to further delegate a subhierarchy
1101 under the existing delegated hierarchy.
1102 (For example, the delegated hierarchy might be associated with
1103 an unprivileged container run by
1104 .IR cecilia .)
1105 Even if a cgroup namespace was employed,
1106 because both hierarchies are owned by the unprivileged user
1107 .IR cecilia ,
1108 the following illegitimate actions could be performed:
1109 .IP \[bu] 3
1110 A process in the inferior hierarchy could change the
1111 resource controller settings in the root directory of that hierarchy.
1112 (These resource controller settings are intended to allow control to
1113 be exercised from the
1114 .I parent
1115 cgroup;
1116 a process inside the child cgroup should not be allowed to modify them.)
1117 .IP \[bu]
1118 A process inside the inferior hierarchy could move processes
1119 into and out of the inferior hierarchy if the cgroups in the
1120 superior hierarchy were somehow visible.
1121 .P
1122 Employing the
1123 .I nsdelegate
1124 mount option prevents both of these possibilities.
1125 .P
1126 The
1127 .I nsdelegate
1128 mount option only has an effect when performed in
1129 the initial mount namespace;
1130 in other mount namespaces, the option is silently ignored.
1131 .P
1132 .IR Note :
1133 On some systems,
1134 .BR systemd (1)
1135 automatically mounts the cgroup v2 filesystem.
1136 In order to experiment with the
1137 .I nsdelegate
1138 operation, it may be useful to boot the kernel with
1139 the following command-line options:
1140 .P
1141 .in +4n
1142 .EX
1143 cgroup_no_v1=all systemd.legacy_systemd_cgroup_controller
1144 .EE
1145 .in
1146 .P
1147 These options cause the kernel to boot with the cgroups v1 controllers
1148 disabled (meaning that the controllers are available in the v2 hierarchy),
1149 and tells
1150 .BR systemd (1)
1151 not to mount and use the cgroup v2 hierarchy,
1152 so that the v2 hierarchy can be manually mounted
1153 with the desired options after boot-up.
1154 .\"
1155 .SS Cgroup delegation containment rules
1156 Some delegation
1157 .I containment rules
1158 ensure that the delegatee can move processes between cgroups within the
1159 delegated subtree,
1160 but can't move processes from outside the delegated subtree into
1161 the subtree or vice versa.
1162 A nonprivileged process (i.e., the delegatee) can write the PID of
1163 a "target" process into a
1164 .I cgroup.procs
1165 file only if all of the following are true:
1166 .IP \[bu] 3
1167 The writer has write permission on the
1168 .I cgroup.procs
1169 file in the destination cgroup.
1170 .IP \[bu]
1171 The writer has write permission on the
1172 .I cgroup.procs
1173 file in the nearest common ancestor of the source and destination cgroups.
1174 Note that in some cases,
1175 the nearest common ancestor may be the source or destination cgroup itself.
1176 This requirement is not enforced for cgroups v1 hierarchies,
1177 with the consequence that containment in v1 is less strict than in v2.
1178 (For example, in cgroups v1 the user that owns two distinct
1179 delegated subhierarchies can move a process between the hierarchies.)
1180 .IP \[bu]
1181 If the cgroup v2 filesystem was mounted with the
1182 .I nsdelegate
1183 option, the writer must be able to see the source and destination cgroups
1184 from its cgroup namespace.
1185 .IP \[bu]
1186 In cgroups v1:
1187 the effective UID of the writer (i.e., the delegatee) matches the
1188 real user ID or the saved set-user-ID of the target process.
1189 Before Linux 4.11,
1190 .\" commit 576dd464505fc53d501bb94569db76f220104d28
1191 this requirement also applied in cgroups v2
1192 (This was a historical requirement inherited from cgroups v1
1193 that was later deemed unnecessary,
1194 since the other rules suffice for containment in cgroups v2.)
1195 .P
1196 .IR Note :
1197 one consequence of these delegation containment rules is that the
1198 unprivileged delegatee can't place the first process into
1199 the delegated subtree;
1200 instead, the delegater must place the first process
1201 (a process owned by the delegatee) into the delegated subtree.
1202 .\"
1203 .SH CGROUPS VERSION 2 THREAD MODE
1204 Among the restrictions imposed by cgroups v2 that were not present
1205 in cgroups v1 are the following:
1206 .IP \[bu] 3
1207 .IR "No thread-granularity control" :
1208 all of the threads of a process must be in the same cgroup.
1209 .IP \[bu]
1210 .IR "No internal processes" :
1211 a cgroup can't both have member processes and
1212 exercise controllers on child cgroups.
1213 .P
1214 Both of these restrictions were added because
1215 the lack of these restrictions had caused problems
1216 in cgroups v1.
1217 In particular, the cgroups v1 ability to allow thread-level granularity
1218 for cgroup membership made no sense for some controllers.
1219 (A notable example was the
1220 .I memory
1221 controller: since threads share an address space,
1222 it made no sense to split threads across different
1223 .I memory
1224 cgroups.)
1225 .P
1226 Notwithstanding the initial design decision in cgroups v2,
1227 there were use cases for certain controllers, notably the
1228 .I cpu
1229 controller,
1230 for which thread-level granularity of control was meaningful and useful.
1231 To accommodate such use cases, Linux 4.14 added
1232 .I "thread mode"
1233 for cgroups v2.
1234 .P
1235 Thread mode allows the following:
1236 .IP \[bu] 3
1237 The creation of
1238 .I threaded subtrees
1239 in which the threads of a process may
1240 be spread across cgroups inside the tree.
1241 (A threaded subtree may contain multiple multithreaded processes.)
1242 .IP \[bu]
1243 The concept of
1244 .IR "threaded controllers" ,
1245 which can distribute resources across the cgroups in a threaded subtree.
1246 .IP \[bu]
1247 A relaxation of the "no internal processes rule",
1248 so that, within a threaded subtree,
1249 a cgroup can both contain member threads and
1250 exercise resource control over child cgroups.
1251 .P
1252 With the addition of thread mode,
1253 each nonroot cgroup now contains a new file,
1254 .IR cgroup.type ,
1255 that exposes, and in some circumstances can be used to change,
1256 the "type" of a cgroup.
1257 This file contains one of the following type values:
1258 .TP
1259 .I domain
1260 This is a normal v2 cgroup that provides process-granularity control.
1261 If a process is a member of this cgroup,
1262 then all threads of the process are (by definition) in the same cgroup.
1263 This is the default cgroup type,
1264 and provides the same behavior that was provided for
1265 cgroups in the initial cgroups v2 implementation.
1266 .TP
1267 .I threaded
1268 This cgroup is a member of a threaded subtree.
1269 Threads can be added to this cgroup,
1270 and controllers can be enabled for the cgroup.
1271 .TP
1272 .I domain threaded
1273 This is a domain cgroup that serves as the root of a threaded subtree.
1274 This cgroup type is also known as "threaded root".
1275 .TP
1276 .I domain invalid
1277 This is a cgroup inside a threaded subtree
1278 that is in an "invalid" state.
1279 Processes can't be added to the cgroup,
1280 and controllers can't be enabled for the cgroup.
1281 The only thing that can be done with this cgroup (other than deleting it)
1282 is to convert it to a
1283 .I threaded
1284 cgroup by writing the string
1285 .I \[dq]threaded\[dq]
1286 to the
1287 .I cgroup.type
1288 file.
1289 .IP
1290 The rationale for the existence of this "interim" type
1291 during the creation of a threaded subtree
1292 (rather than the kernel simply immediately converting all cgroups
1293 under the threaded root to the type
1294 .IR threaded )
1295 is to allow for
1296 possible future extensions to the thread mode model
1297 .\"
1298 .SS Threaded versus domain controllers
1299 With the addition of threads mode,
1300 cgroups v2 now distinguishes two types of resource controllers:
1301 .IP \[bu] 3
1302 .I Threaded
1303 .\" In the kernel source, look for ".threaded[ \t]*= true" in
1304 .\" initializations of struct cgroup_subsys
1305 controllers: these controllers support thread-granularity for
1306 resource control and can be enabled inside threaded subtrees,
1307 with the result that the corresponding controller-interface files
1308 appear inside the cgroups in the threaded subtree.
1309 As at Linux 4.19, the following controllers are threaded:
1310 .IR cpu ,
1311 .IR perf_event ,
1312 and
1313 .IR pids .
1314 .IP \[bu]
1315 .I Domain
1316 controllers: these controllers support only process granularity
1317 for resource control.
1318 From the perspective of a domain controller,
1319 all threads of a process are always in the same cgroup.
1320 Domain controllers can't be enabled inside a threaded subtree.
1321 .\"
1322 .SS Creating a threaded subtree
1323 There are two pathways that lead to the creation of a threaded subtree.
1324 The first pathway proceeds as follows:
1325 .IP (1) 5
1326 We write the string
1327 .I \[dq]threaded\[dq]
1328 to the
1329 .I cgroup.type
1330 file of a cgroup
1331 .I y/z
1332 that currently has the type
1333 .IR domain .
1334 This has the following effects:
1335 .RS
1336 .IP \[bu] 3
1337 The type of the cgroup
1338 .I y/z
1339 becomes
1340 .IR threaded .
1341 .IP \[bu]
1342 The type of the parent cgroup,
1343 .IR y ,
1344 becomes
1345 .IR "domain threaded" .
1346 The parent cgroup is the root of a threaded subtree
1347 (also known as the "threaded root").
1348 .IP \[bu]
1349 All other cgroups under
1350 .I y
1351 that were not already of type
1352 .I threaded
1353 (because they were inside already existing threaded subtrees
1354 under the new threaded root)
1355 are converted to type
1356 .IR "domain invalid" .
1357 Any subsequently created cgroups under
1358 .I y
1359 will also have the type
1360 .IR "domain invalid" .
1361 .RE
1362 .IP (2)
1363 We write the string
1364 .I \[dq]threaded\[dq]
1365 to each of the
1366 .I domain invalid
1367 cgroups under
1368 .IR y ,
1369 in order to convert them to the type
1370 .IR threaded .
1371 As a consequence of this step, all threads under the threaded root
1372 now have the type
1373 .I threaded
1374 and the threaded subtree is now fully usable.
1375 The requirement to write
1376 .I \[dq]threaded\[dq]
1377 to each of these cgroups is somewhat cumbersome,
1378 but allows for possible future extensions to the thread-mode model.
1379 .P
1380 The second way of creating a threaded subtree is as follows:
1381 .IP (1) 5
1382 In an existing cgroup,
1383 .IR z ,
1384 that currently has the type
1385 .IR domain ,
1386 we (1.1) enable one or more threaded controllers and
1387 (1.2) make a process a member of
1388 .IR z .
1389 (These two steps can be done in either order.)
1390 This has the following consequences:
1391 .RS
1392 .IP \[bu] 3
1393 The type of
1394 .I z
1395 becomes
1396 .IR "domain threaded" .
1397 .IP \[bu]
1398 All of the descendant cgroups of
1399 .I z
1400 that were not already of type
1401 .I threaded
1402 are converted to type
1403 .IR "domain invalid" .
1404 .RE
1405 .IP (2)
1406 As before, we make the threaded subtree usable by writing the string
1407 .I \[dq]threaded\[dq]
1408 to each of the
1409 .I domain invalid
1410 cgroups under
1411 .IR z ,
1412 in order to convert them to the type
1413 .IR threaded .
1414 .P
1415 One of the consequences of the above pathways to creating a threaded subtree
1416 is that the threaded root cgroup can be a parent only to
1417 .I threaded
1418 (and
1419 .IR "domain invalid" )
1420 cgroups.
1421 The threaded root cgroup can't be a parent of a
1422 .I domain
1423 cgroup, and a
1424 .I threaded
1425 cgroup
1426 can't have a sibling that is a
1427 .I domain
1428 cgroup.
1429 .\"
1430 .SS Using a threaded subtree
1431 Within a threaded subtree, threaded controllers can be enabled
1432 in each subgroup whose type has been changed to
1433 .IR threaded ;
1434 upon doing so, the corresponding controller interface files
1435 appear in the children of that cgroup.
1436 .P
1437 A process can be moved into a threaded subtree by writing its PID to the
1438 .I cgroup.procs
1439 file in one of the cgroups inside the tree.
1440 This has the effect of making all of the threads
1441 in the process members of the corresponding cgroup
1442 and makes the process a member of the threaded subtree.
1443 The threads of the process can then be spread across
1444 the threaded subtree by writing their thread IDs (see
1445 .BR gettid (2))
1446 to the
1447 .I cgroup.threads
1448 files in different cgroups inside the subtree.
1449 The threads of a process must all reside in the same threaded subtree.
1450 .P
1451 As with writing to
1452 .IR cgroup.procs ,
1453 some containment rules apply when writing to the
1454 .I cgroup.threads
1455 file:
1456 .IP \[bu] 3
1457 The writer must have write permission on the
1458 cgroup.threads
1459 file in the destination cgroup.
1460 .IP \[bu]
1461 The writer must have write permission on the
1462 .I cgroup.procs
1463 file in the common ancestor of the source and destination cgroups.
1464 (In some cases,
1465 the common ancestor may be the source or destination cgroup itself.)
1466 .IP \[bu]
1467 The source and destination cgroups must be in the same threaded subtree.
1468 (Outside a threaded subtree, an attempt to move a thread by writing
1469 its thread ID to the
1470 .I cgroup.threads
1471 file in a different
1472 .I domain
1473 cgroup fails with the error
1474 .BR EOPNOTSUPP .)
1475 .P
1476 The
1477 .I cgroup.threads
1478 file is present in each cgroup (including
1479 .I domain
1480 cgroups) and can be read in order to discover the set of threads
1481 that is present in the cgroup.
1482 The set of thread IDs obtained when reading this file
1483 is not guaranteed to be ordered or free of duplicates.
1484 .P
1485 The
1486 .I cgroup.procs
1487 file in the threaded root shows the PIDs of all processes
1488 that are members of the threaded subtree.
1489 The
1490 .I cgroup.procs
1491 files in the other cgroups in the subtree are not readable.
1492 .P
1493 Domain controllers can't be enabled in a threaded subtree;
1494 no controller-interface files appear inside the cgroups underneath the
1495 threaded root.
1496 From the point of view of a domain controller,
1497 threaded subtrees are invisible:
1498 a multithreaded process inside a threaded subtree appears to a domain
1499 controller as a process that resides in the threaded root cgroup.
1500 .P
1501 Within a threaded subtree, the "no internal processes" rule does not apply:
1502 a cgroup can both contain member processes (or thread)
1503 and exercise controllers on child cgroups.
1504 .\"
1505 .SS Rules for writing to cgroup.type and creating threaded subtrees
1506 A number of rules apply when writing to the
1507 .I cgroup.type
1508 file:
1509 .IP \[bu] 3
1510 Only the string
1511 .I \[dq]threaded\[dq]
1512 may be written.
1513 In other words, the only explicit transition that is possible is to convert a
1514 .I domain
1515 cgroup to type
1516 .IR threaded .
1517 .IP \[bu]
1518 The effect of writing
1519 .I \[dq]threaded\[dq]
1520 depends on the current value in
1521 .IR cgroup.type ,
1522 as follows:
1523 .RS
1524 .IP \[bu] 3
1525 .I domain
1526 or
1527 .IR "domain threaded" :
1528 start the creation of a threaded subtree
1529 (whose root is the parent of this cgroup) via
1530 the first of the pathways described above;
1531 .IP \[bu]
1532 .IR "domain\ invalid" :
1533 convert this cgroup (which is inside a threaded subtree) to a usable (i.e.,
1534 .IR threaded )
1535 state;
1536 .IP \[bu]
1537 .IR threaded :
1538 no effect (a "no-op").
1539 .RE
1540 .IP \[bu]
1541 We can't write to a
1542 .I cgroup.type
1543 file if the parent's type is
1544 .IR "domain invalid" .
1545 In other words, the cgroups of a threaded subtree must be converted to the
1546 .I threaded
1547 state in a top-down manner.
1548 .P
1549 There are also some constraints that must be satisfied
1550 in order to create a threaded subtree rooted at the cgroup
1551 .IR x :
1552 .IP \[bu] 3
1553 There can be no member processes in the descendant cgroups of
1554 .IR x .
1555 (The cgroup
1556 .I x
1557 can itself have member processes.)
1558 .IP \[bu]
1559 No domain controllers may be enabled in
1560 .IR x 's
1561 .I cgroup.subtree_control
1562 file.
1563 .P
1564 If any of the above constraints is violated, then an attempt to write
1565 .I \[dq]threaded\[dq]
1566 to a
1567 .I cgroup.type
1568 file fails with the error
1569 .BR ENOTSUP .
1570 .\"
1571 .SS The \[dq]domain threaded\[dq] cgroup type
1572 According to the pathways described above,
1573 the type of a cgroup can change to
1574 .I domain threaded
1575 in either of the following cases:
1576 .IP \[bu] 3
1577 The string
1578 .I \[dq]threaded\[dq]
1579 is written to a child cgroup.
1580 .IP \[bu]
1581 A threaded controller is enabled inside the cgroup and
1582 a process is made a member of the cgroup.
1583 .P
1584 A
1585 .I domain threaded
1586 cgroup,
1587 .IR x ,
1588 can revert to the type
1589 .I domain
1590 if the above conditions no longer hold true\[em]that is, if all
1591 .I threaded
1592 child cgroups of
1593 .I x
1594 are removed and either
1595 .I x
1596 no longer has threaded controllers enabled or
1597 no longer has member processes.
1598 .P
1599 When a
1600 .I domain threaded
1601 cgroup
1602 .I x
1603 reverts to the type
1604 .IR domain :
1605 .IP \[bu] 3
1606 All
1607 .I domain invalid
1608 descendants of
1609 .I x
1610 that are not in lower-level threaded subtrees revert to the type
1611 .IR domain .
1612 .IP \[bu]
1613 The root cgroups in any lower-level threaded subtrees revert to the type
1614 .IR "domain threaded" .
1615 .\"
1616 .SS Exceptions for the root cgroup
1617 The root cgroup of the v2 hierarchy is treated exceptionally:
1618 it can be the parent of both
1619 .I domain
1620 and
1621 .I threaded
1622 cgroups.
1623 If the string
1624 .I \[dq]threaded\[dq]
1625 is written to the
1626 .I cgroup.type
1627 file of one of the children of the root cgroup, then
1628 .IP \[bu] 3
1629 The type of that cgroup becomes
1630 .IR threaded .
1631 .IP \[bu]
1632 The type of any descendants of that cgroup that
1633 are not part of lower-level threaded subtrees changes to
1634 .IR "domain invalid" .
1635 .P
1636 Note that in this case, there is no cgroup whose type becomes
1637 .IR "domain threaded" .
1638 (Notionally, the root cgroup can be considered as the threaded root
1639 for the cgroup whose type was changed to
1640 .IR threaded .)
1641 .P
1642 The aim of this exceptional treatment for the root cgroup is to
1643 allow a threaded cgroup that employs the
1644 .I cpu
1645 controller to be placed as high as possible in the hierarchy,
1646 so as to minimize the (small) cost of traversing the cgroup hierarchy.
1647 .\"
1648 .SS The cgroups v2 \[dq]cpu\[dq] controller and realtime threads
1649 As at Linux 4.19, the cgroups v2
1650 .I cpu
1651 controller does not support control of realtime threads
1652 (specifically threads scheduled under any of the policies
1653 .BR SCHED_FIFO ,
1654 .BR SCHED_RR ,
1655 described
1656 .BR SCHED_DEADLINE ;
1657 see
1658 .BR sched (7)).
1659 Therefore, the
1660 .I cpu
1661 controller can be enabled in the root cgroup only
1662 if all realtime threads are in the root cgroup.
1663 (If there are realtime threads in nonroot cgroups, then a
1664 .BR write (2)
1665 of the string
1666 .I \[dq]+cpu\[dq]
1667 to the
1668 .I cgroup.subtree_control
1669 file fails with the error
1670 .BR EINVAL .)
1671 .P
1672 On some systems,
1673 .BR systemd (1)
1674 places certain realtime threads in nonroot cgroups in the v2 hierarchy.
1675 On such systems,
1676 these threads must first be moved to the root cgroup before the
1677 .I cpu
1678 controller can be enabled.
1679 .\"
1680 .SH ERRORS
1681 The following errors can occur for
1682 .BR mount (2):
1683 .TP
1684 .B EBUSY
1685 An attempt to mount a cgroup version 1 filesystem specified neither the
1686 .I name=
1687 option (to mount a named hierarchy) nor a controller name (or
1688 .IR all ).
1689 .SH NOTES
1690 A child process created via
1691 .BR fork (2)
1692 inherits its parent's cgroup memberships.
1693 A process's cgroup memberships are preserved across
1694 .BR execve (2).
1695 .P
1696 The
1697 .BR clone3 (2)
1698 .B CLONE_INTO_CGROUP
1699 flag can be used to create a child process that begins its life in
1700 a different version 2 cgroup from the parent process.
1701 .\"
1702 .SS /proc files
1703 .TP
1704 .IR /proc/cgroups " (since Linux 2.6.24)"
1705 This file contains information about the controllers
1706 that are compiled into the kernel.
1707 An example of the contents of this file (reformatted for readability)
1708 is the following:
1709 .IP
1710 .in +4n
1711 .EX
1712 #subsys_name hierarchy num_cgroups enabled
1713 cpuset 4 1 1
1714 cpu 8 1 1
1715 cpuacct 8 1 1
1716 blkio 6 1 1
1717 memory 3 1 1
1718 devices 10 84 1
1719 freezer 7 1 1
1720 net_cls 9 1 1
1721 perf_event 5 1 1
1722 net_prio 9 1 1
1723 hugetlb 0 1 0
1724 pids 2 1 1
1725 .EE
1726 .in
1727 .IP
1728 The fields in this file are, from left to right:
1729 .RS
1730 .IP [1] 5
1731 The name of the controller.
1732 .IP [2]
1733 The unique ID of the cgroup hierarchy on which this controller is mounted.
1734 If multiple cgroups v1 controllers are bound to the same hierarchy,
1735 then each will show the same hierarchy ID in this field.
1736 The value in this field will be 0 if:
1737 .RS
1738 .IP \[bu] 3
1739 the controller is not mounted on a cgroups v1 hierarchy;
1740 .IP \[bu]
1741 the controller is bound to the cgroups v2 single unified hierarchy; or
1742 .IP \[bu]
1743 the controller is disabled (see below).
1744 .RE
1745 .IP [3]
1746 The number of control groups in this hierarchy using this controller.
1747 .IP [4]
1748 This field contains the value 1 if this controller is enabled,
1749 or 0 if it has been disabled (via the
1750 .I cgroup_disable
1751 kernel command-line boot parameter).
1752 .RE
1753 .TP
1754 .IR /proc/ pid /cgroup " (since Linux 2.6.24)"
1755 This file describes control groups to which the process
1756 with the corresponding PID belongs.
1757 The displayed information differs for
1758 cgroups version 1 and version 2 hierarchies.
1759 .IP
1760 For each cgroup hierarchy of which the process is a member,
1761 there is one entry containing three colon-separated fields:
1762 .IP
1763 .in +4n
1764 .EX
1765 hierarchy\-ID:controller\-list:cgroup\-path
1766 .EE
1767 .in
1768 .IP
1769 For example:
1770 .IP
1771 .in +4n
1772 .EX
1773 5:cpuacct,cpu,cpuset:/daemons
1774 .EE
1775 .in
1776 .IP
1777 The colon-separated fields are, from left to right:
1778 .RS
1779 .IP [1] 5
1780 For cgroups version 1 hierarchies,
1781 this field contains a unique hierarchy ID number
1782 that can be matched to a hierarchy ID in
1783 .IR /proc/cgroups .
1784 For the cgroups version 2 hierarchy, this field contains the value 0.
1785 .IP [2]
1786 For cgroups version 1 hierarchies,
1787 this field contains a comma-separated list of the controllers
1788 bound to the hierarchy.
1789 For the cgroups version 2 hierarchy, this field is empty.
1790 .IP [3]
1791 This field contains the pathname of the control group in the hierarchy
1792 to which the process belongs.
1793 This pathname is relative to the mount point of the hierarchy.
1794 .RE
1795 .\"
1796 .SS /sys/kernel/cgroup files
1797 .TP
1798 .IR /sys/kernel/cgroup/delegate " (since Linux 4.15)"
1799 .\" commit 01ee6cfb1483fe57c9cbd8e73817dfbf9bacffd3
1800 This file exports a list of the cgroups v2 files
1801 (one per line) that are delegatable
1802 (i.e., whose ownership should be changed to the user ID of the delegatee).
1803 In the future, the set of delegatable files may change or grow,
1804 and this file provides a way for the kernel to inform
1805 user-space applications of which files must be delegated.
1806 As at Linux 4.15, one sees the following when inspecting this file:
1807 .IP
1808 .in +4n
1809 .EX
1810 .RB $ " cat /sys/kernel/cgroup/delegate"
1811 cgroup.procs
1812 cgroup.subtree_control
1813 cgroup.threads
1814 .EE
1815 .in
1816 .TP
1817 .IR /sys/kernel/cgroup/features " (since Linux 4.15)"
1818 .\" commit 5f2e673405b742be64e7c3604ed4ed3ac14f35ce
1819 Over time, the set of cgroups v2 features that are provided by the
1820 kernel may change or grow,
1821 or some features may not be enabled by default.
1822 This file provides a way for user-space applications to discover what
1823 features the running kernel supports and has enabled.
1824 Features are listed one per line:
1825 .IP
1826 .in +4n
1827 .EX
1828 .RB $ " cat /sys/kernel/cgroup/features"
1829 nsdelegate
1830 memory_localevents
1831 .EE
1832 .in
1833 .IP
1834 The entries that can appear in this file are:
1835 .RS
1836 .TP
1837 .IR memory_localevents " (since Linux 5.2)"
1838 The kernel supports the
1839 .I memory_localevents
1840 mount option.
1841 .TP
1842 .IR nsdelegate " (since Linux 4.15)"
1843 The kernel supports the
1844 .I nsdelegate
1845 mount option.
1846 .TP
1847 .IR memory_recursiveprot " (since Linux 5.7)"
1848 .\" commit 8a931f801340c2be10552c7b5622d5f4852f3a36
1849 The kernel supports the
1850 .I memory_recursiveprot
1851 mount option.
1852 .RE
1853 .SH SEE ALSO
1854 .BR prlimit (1),
1855 .BR systemd (1),
1856 .BR systemd\-cgls (1),
1857 .BR systemd\-cgtop (1),
1858 .BR clone (2),
1859 .BR ioprio_set (2),
1860 .BR perf_event_open (2),
1861 .BR setrlimit (2),
1862 .BR cgroup_namespaces (7),
1863 .BR cpuset (7),
1864 .BR namespaces (7),
1865 .BR sched (7),
1866 .BR user_namespaces (7)
1867 .P
1868 The kernel source file
1869 .IR Documentation/admin\-guide/cgroup\-v2.rst .