| 1 |
============================ |
| 2 |
Transparent Hugepage Support |
| 3 |
============================ |
| 4 |
|
| 5 |
Objective |
| 6 |
========= |
| 7 |
|
| 8 |
Performance critical computing applications dealing with large memory |
| 9 |
working sets are already running on top of libhugetlbfs and in turn |
| 10 |
hugetlbfs. Transparent HugePage Support (THP) is an alternative mean of |
| 11 |
using huge pages for the backing of virtual memory with huge pages |
| 12 |
that supports the automatic promotion and demotion of page sizes and |
| 13 |
without the shortcomings of hugetlbfs. |
| 14 |
|
| 15 |
Currently THP only works for anonymous memory mappings and tmpfs/shmem. |
| 16 |
But in the future it can expand to other filesystems. |
| 17 |
|
| 18 |
.. note:: |
| 19 |
in the examples below we presume that the basic page size is 4K and |
| 20 |
the huge page size is 2M, although the actual numbers may vary |
| 21 |
depending on the CPU architecture. |
| 22 |
|
| 23 |
The reason applications are running faster is because of two |
| 24 |
factors. The first factor is almost completely irrelevant and it's not |
| 25 |
of significant interest because it'll also have the downside of |
| 26 |
requiring larger clear-page copy-page in page faults which is a |
| 27 |
potentially negative effect. The first factor consists in taking a |
| 28 |
single page fault for each 2M virtual region touched by userland (so |
| 29 |
reducing the enter/exit kernel frequency by a 512 times factor). This |
| 30 |
only matters the first time the memory is accessed for the lifetime of |
| 31 |
a memory mapping. The second long lasting and much more important |
| 32 |
factor will affect all subsequent accesses to the memory for the whole |
| 33 |
runtime of the application. The second factor consist of two |
| 34 |
components: |
| 35 |
|
| 36 |
1) the TLB miss will run faster (especially with virtualization using |
| 37 |
nested pagetables but almost always also on bare metal without |
| 38 |
virtualization) |
| 39 |
|
| 40 |
2) a single TLB entry will be mapping a much larger amount of virtual |
| 41 |
memory in turn reducing the number of TLB misses. With |
| 42 |
virtualization and nested pagetables the TLB can be mapped of |
| 43 |
larger size only if both KVM and the Linux guest are using |
| 44 |
hugepages but a significant speedup already happens if only one of |
| 45 |
the two is using hugepages just because of the fact the TLB miss is |
| 46 |
going to run faster. |
| 47 |
|
| 48 |
Modern kernels support "multi-size THP" (mTHP), which introduces the |
| 49 |
ability to allocate memory in blocks that are bigger than a base page |
| 50 |
but smaller than traditional PMD-size (as described above), in |
| 51 |
increments of a power-of-2 number of pages. mTHP can back anonymous |
| 52 |
memory (for example 16K, 32K, 64K, etc). These THPs continue to be |
| 53 |
PTE-mapped, but in many cases can still provide similar benefits to |
| 54 |
those outlined above: Page faults are significantly reduced (by a |
| 55 |
factor of e.g. 4, 8, 16, etc), but latency spikes are much less |
| 56 |
prominent because the size of each page isn't as huge as the PMD-sized |
| 57 |
variant and there is less memory to clear in each page fault. Some |
| 58 |
architectures also employ TLB compression mechanisms to squeeze more |
| 59 |
entries in when a set of PTEs are virtually and physically contiguous |
| 60 |
and approporiately aligned. In this case, TLB misses will occur less |
| 61 |
often. |
| 62 |
|
| 63 |
THP can be enabled system wide or restricted to certain tasks or even |
| 64 |
memory ranges inside task's address space. Unless THP is completely |
| 65 |
disabled, there is ``khugepaged`` daemon that scans memory and |
| 66 |
collapses sequences of basic pages into PMD-sized huge pages. |
| 67 |
|
| 68 |
The THP behaviour is controlled via :ref:`sysfs <thp_sysfs>` |
| 69 |
interface and using madvise(2) and prctl(2) system calls. |
| 70 |
|
| 71 |
Transparent Hugepage Support maximizes the usefulness of free memory |
| 72 |
if compared to the reservation approach of hugetlbfs by allowing all |
| 73 |
unused memory to be used as cache or other movable (or even unmovable |
| 74 |
entities). It doesn't require reservation to prevent hugepage |
| 75 |
allocation failures to be noticeable from userland. It allows paging |
| 76 |
and all other advanced VM features to be available on the |
| 77 |
hugepages. It requires no modifications for applications to take |
| 78 |
advantage of it. |
| 79 |
|
| 80 |
Applications however can be further optimized to take advantage of |
| 81 |
this feature, like for example they've been optimized before to avoid |
| 82 |
a flood of mmap system calls for every malloc(4k). Optimizing userland |
| 83 |
is by far not mandatory and khugepaged already can take care of long |
| 84 |
lived page allocations even for hugepage unaware applications that |
| 85 |
deals with large amounts of memory. |
| 86 |
|
| 87 |
In certain cases when hugepages are enabled system wide, application |
| 88 |
may end up allocating more memory resources. An application may mmap a |
| 89 |
large region but only touch 1 byte of it, in that case a 2M page might |
| 90 |
be allocated instead of a 4k page for no good. This is why it's |
| 91 |
possible to disable hugepages system-wide and to only have them inside |
| 92 |
MADV_HUGEPAGE madvise regions. |
| 93 |
|
| 94 |
Embedded systems should enable hugepages only inside madvise regions |
| 95 |
to eliminate any risk of wasting any precious byte of memory and to |
| 96 |
only run faster. |
| 97 |
|
| 98 |
Applications that gets a lot of benefit from hugepages and that don't |
| 99 |
risk to lose memory by using hugepages, should use |
| 100 |
madvise(MADV_HUGEPAGE) on their critical mmapped regions. |
| 101 |
|
| 102 |
.. _thp_sysfs: |
| 103 |
|
| 104 |
sysfs |
| 105 |
===== |
| 106 |
|
| 107 |
Global THP controls |
| 108 |
------------------- |
| 109 |
|
| 110 |
Transparent Hugepage Support for anonymous memory can be disabled |
| 111 |
(mostly for debugging purposes) or only enabled inside MADV_HUGEPAGE |
| 112 |
regions (to avoid the risk of consuming more memory resources) or enabled |
| 113 |
system wide. This can be achieved per-supported-THP-size with one of:: |
| 114 |
|
| 115 |
echo always >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled |
| 116 |
echo madvise >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled |
| 117 |
echo never >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled |
| 118 |
|
| 119 |
where <size> is the hugepage size being addressed, the available sizes |
| 120 |
for which vary by system. |
| 121 |
|
| 122 |
.. note:: Setting "never" in all sysfs THP controls does **not** disable |
| 123 |
Transparent Huge Pages globally. This is because ``madvise(..., |
| 124 |
MADV_COLLAPSE)`` ignores these settings and collapses ranges to |
| 125 |
PMD-sized huge pages unconditionally. |
| 126 |
|
| 127 |
For example:: |
| 128 |
|
| 129 |
echo always >/sys/kernel/mm/transparent_hugepage/hugepages-2048kB/enabled |
| 130 |
|
| 131 |
Alternatively it is possible to specify that a given hugepage size |
| 132 |
will inherit the top-level "enabled" value:: |
| 133 |
|
| 134 |
echo inherit >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled |
| 135 |
|
| 136 |
For example:: |
| 137 |
|
| 138 |
echo inherit >/sys/kernel/mm/transparent_hugepage/hugepages-2048kB/enabled |
| 139 |
|
| 140 |
The top-level setting (for use with "inherit") can be set by issuing |
| 141 |
one of the following commands:: |
| 142 |
|
| 143 |
echo always >/sys/kernel/mm/transparent_hugepage/enabled |
| 144 |
echo madvise >/sys/kernel/mm/transparent_hugepage/enabled |
| 145 |
echo never >/sys/kernel/mm/transparent_hugepage/enabled |
| 146 |
|
| 147 |
By default, PMD-sized hugepages have enabled="inherit" and all other |
| 148 |
hugepage sizes have enabled="never". If enabling multiple hugepage |
| 149 |
sizes, the kernel will select the most appropriate enabled size for a |
| 150 |
given allocation. |
| 151 |
|
| 152 |
It's also possible to limit defrag efforts in the VM to generate |
| 153 |
anonymous hugepages in case they're not immediately free to madvise |
| 154 |
regions or to never try to defrag memory and simply fallback to regular |
| 155 |
pages unless hugepages are immediately available. Clearly if we spend CPU |
| 156 |
time to defrag memory, we would expect to gain even more by the fact we |
| 157 |
use hugepages later instead of regular pages. This isn't always |
| 158 |
guaranteed, but it may be more likely in case the allocation is for a |
| 159 |
MADV_HUGEPAGE region. |
| 160 |
|
| 161 |
:: |
| 162 |
|
| 163 |
echo always >/sys/kernel/mm/transparent_hugepage/defrag |
| 164 |
echo defer >/sys/kernel/mm/transparent_hugepage/defrag |
| 165 |
echo defer+madvise >/sys/kernel/mm/transparent_hugepage/defrag |
| 166 |
echo madvise >/sys/kernel/mm/transparent_hugepage/defrag |
| 167 |
echo never >/sys/kernel/mm/transparent_hugepage/defrag |
| 168 |
|
| 169 |
always |
| 170 |
means that an application requesting THP will stall on |
| 171 |
allocation failure and directly reclaim pages and compact |
| 172 |
memory in an effort to allocate a THP immediately. This may be |
| 173 |
desirable for virtual machines that benefit heavily from THP |
| 174 |
use and are willing to delay the VM start to utilise them. |
| 175 |
|
| 176 |
defer |
| 177 |
means that an application will wake kswapd in the background |
| 178 |
to reclaim pages and wake kcompactd to compact memory so that |
| 179 |
THP is available in the near future. It's the responsibility |
| 180 |
of khugepaged to then install the THP pages later. |
| 181 |
|
| 182 |
defer+madvise |
| 183 |
will enter direct reclaim and compaction like ``always``, but |
| 184 |
only for regions that have used madvise(MADV_HUGEPAGE); all |
| 185 |
other regions will wake kswapd in the background to reclaim |
| 186 |
pages and wake kcompactd to compact memory so that THP is |
| 187 |
available in the near future. |
| 188 |
|
| 189 |
madvise |
| 190 |
will enter direct reclaim like ``always`` but only for regions |
| 191 |
that are have used madvise(MADV_HUGEPAGE). This is the default |
| 192 |
behaviour. |
| 193 |
|
| 194 |
never |
| 195 |
should be self-explanatory. Note that ``madvise(..., |
| 196 |
MADV_COLLAPSE)`` can still cause transparent huge pages to be |
| 197 |
obtained even if this mode is specified everywhere. |
| 198 |
|
| 199 |
By default kernel tries to use huge, PMD-mappable zero page on read |
| 200 |
page fault to anonymous mapping. It's possible to disable huge zero |
| 201 |
page by writing 0 or enable it back by writing 1:: |
| 202 |
|
| 203 |
echo 0 >/sys/kernel/mm/transparent_hugepage/use_zero_page |
| 204 |
echo 1 >/sys/kernel/mm/transparent_hugepage/use_zero_page |
| 205 |
|
| 206 |
Some userspace (such as a test program, or an optimized memory |
| 207 |
allocation library) may want to know the size (in bytes) of a |
| 208 |
PMD-mappable transparent hugepage:: |
| 209 |
|
| 210 |
cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size |
| 211 |
|
| 212 |
All THPs at fault and collapse time will be added to _deferred_list, |
| 213 |
and will therefore be split under memory presure if they are considered |
| 214 |
"underused". A THP is underused if the number of zero-filled pages in |
| 215 |
the THP is above max_ptes_none (see below). It is possible to disable |
| 216 |
this behaviour by writing 0 to shrink_underused, and enable it by writing |
| 217 |
1 to it:: |
| 218 |
|
| 219 |
echo 0 > /sys/kernel/mm/transparent_hugepage/shrink_underused |
| 220 |
echo 1 > /sys/kernel/mm/transparent_hugepage/shrink_underused |
| 221 |
|
| 222 |
khugepaged will be automatically started when PMD-sized THP is enabled |
| 223 |
(either of the per-size anon control or the top-level control are set |
| 224 |
to "always" or "madvise"), and it'll be automatically shutdown when |
| 225 |
PMD-sized THP is disabled (when both the per-size anon control and the |
| 226 |
top-level control are "never") |
| 227 |
|
| 228 |
process THP controls |
| 229 |
-------------------- |
| 230 |
|
| 231 |
A process can control its own THP behaviour using the ``PR_SET_THP_DISABLE`` |
| 232 |
and ``PR_GET_THP_DISABLE`` pair of prctl(2) calls. The THP behaviour set using |
| 233 |
``PR_SET_THP_DISABLE`` is inherited across fork(2) and execve(2). These calls |
| 234 |
support the following arguments:: |
| 235 |
|
| 236 |
prctl(PR_SET_THP_DISABLE, 1, 0, 0, 0): |
| 237 |
This will disable THPs completely for the process, irrespective |
| 238 |
of global THP controls or madvise(..., MADV_COLLAPSE) being used. |
| 239 |
|
| 240 |
prctl(PR_SET_THP_DISABLE, 1, PR_THP_DISABLE_EXCEPT_ADVISED, 0, 0): |
| 241 |
This will disable THPs for the process except when the usage of THPs is |
| 242 |
advised. Consequently, THPs will only be used when: |
| 243 |
- Global THP controls are set to "always" or "madvise" and |
| 244 |
madvise(..., MADV_HUGEPAGE) or madvise(..., MADV_COLLAPSE) is used. |
| 245 |
- Global THP controls are set to "never" and madvise(..., MADV_COLLAPSE) |
| 246 |
is used. This is the same behavior as if THPs would not be disabled on |
| 247 |
a process level. |
| 248 |
Note that MADV_COLLAPSE is currently always rejected if |
| 249 |
madvise(..., MADV_NOHUGEPAGE) is set on an area. |
| 250 |
|
| 251 |
prctl(PR_SET_THP_DISABLE, 0, 0, 0, 0): |
| 252 |
This will re-enable THPs for the process, as if they were never disabled. |
| 253 |
Whether THPs will actually be used depends on global THP controls and |
| 254 |
madvise() calls. |
| 255 |
|
| 256 |
prctl(PR_GET_THP_DISABLE, 0, 0, 0, 0): |
| 257 |
This returns a value whose bits indicate how THP-disable is configured: |
| 258 |
Bits |
| 259 |
1 0 Value Description |
| 260 |
|0|0| 0 No THP-disable behaviour specified. |
| 261 |
|0|1| 1 THP is entirely disabled for this process. |
| 262 |
|1|1| 3 THP-except-advised mode is set for this process. |
| 263 |
|
| 264 |
Khugepaged controls |
| 265 |
------------------- |
| 266 |
|
| 267 |
.. note:: |
| 268 |
khugepaged currently only searches for opportunities to collapse to |
| 269 |
PMD-sized THP and no attempt is made to collapse to other THP |
| 270 |
sizes. |
| 271 |
|
| 272 |
khugepaged runs usually at low frequency so while one may not want to |
| 273 |
invoke defrag algorithms synchronously during the page faults, it |
| 274 |
should be worth invoking defrag at least in khugepaged. However it's |
| 275 |
also possible to disable defrag in khugepaged by writing 0 or enable |
| 276 |
defrag in khugepaged by writing 1:: |
| 277 |
|
| 278 |
echo 0 >/sys/kernel/mm/transparent_hugepage/khugepaged/defrag |
| 279 |
echo 1 >/sys/kernel/mm/transparent_hugepage/khugepaged/defrag |
| 280 |
|
| 281 |
You can also control how many pages khugepaged should scan at each |
| 282 |
pass:: |
| 283 |
|
| 284 |
/sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan |
| 285 |
|
| 286 |
and how many milliseconds to wait in khugepaged between each pass (you |
| 287 |
can set this to 0 to run khugepaged at 100% utilization of one core):: |
| 288 |
|
| 289 |
/sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs |
| 290 |
|
| 291 |
and how many milliseconds to wait in khugepaged if there's an hugepage |
| 292 |
allocation failure to throttle the next allocation attempt:: |
| 293 |
|
| 294 |
/sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs |
| 295 |
|
| 296 |
The khugepaged progress can be seen in the number of pages collapsed (note |
| 297 |
that this counter may not be an exact count of the number of pages |
| 298 |
collapsed, since "collapsed" could mean multiple things: (1) A PTE mapping |
| 299 |
being replaced by a PMD mapping, or (2) All 4K physical pages replaced by |
| 300 |
one 2M hugepage. Each may happen independently, or together, depending on |
| 301 |
the type of memory and the failures that occur. As such, this value should |
| 302 |
be interpreted roughly as a sign of progress, and counters in /proc/vmstat |
| 303 |
consulted for more accurate accounting):: |
| 304 |
|
| 305 |
/sys/kernel/mm/transparent_hugepage/khugepaged/pages_collapsed |
| 306 |
|
| 307 |
for each pass:: |
| 308 |
|
| 309 |
/sys/kernel/mm/transparent_hugepage/khugepaged/full_scans |
| 310 |
|
| 311 |
``max_ptes_none`` specifies how many extra small pages (that are |
| 312 |
not already mapped) can be allocated when collapsing a group |
| 313 |
of small pages into one large page:: |
| 314 |
|
| 315 |
/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none |
| 316 |
|
| 317 |
A higher value leads to use additional memory for programs. |
| 318 |
A lower value leads to gain less thp performance. Value of |
| 319 |
max_ptes_none can waste cpu time very little, you can |
| 320 |
ignore it. |
| 321 |
|
| 322 |
``max_ptes_swap`` specifies how many pages can be brought in from |
| 323 |
swap when collapsing a group of pages into a transparent huge page:: |
| 324 |
|
| 325 |
/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_swap |
| 326 |
|
| 327 |
A higher value can cause excessive swap IO and waste |
| 328 |
memory. A lower value can prevent THPs from being |
| 329 |
collapsed, resulting fewer pages being collapsed into |
| 330 |
THPs, and lower memory access performance. |
| 331 |
|
| 332 |
``max_ptes_shared`` specifies how many pages can be shared across multiple |
| 333 |
processes. khugepaged might treat pages of THPs as shared if any page of |
| 334 |
that THP is shared. Exceeding the number would block the collapse:: |
| 335 |
|
| 336 |
/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_shared |
| 337 |
|
| 338 |
A higher value may increase memory footprint for some workloads. |
| 339 |
|
| 340 |
Boot parameters |
| 341 |
=============== |
| 342 |
|
| 343 |
You can change the sysfs boot time default for the top-level "enabled" |
| 344 |
control by passing the parameter ``transparent_hugepage=always`` or |
| 345 |
``transparent_hugepage=madvise`` or ``transparent_hugepage=never`` to the |
| 346 |
kernel command line. |
| 347 |
|
| 348 |
Alternatively, each supported anonymous THP size can be controlled by |
| 349 |
passing ``thp_anon=<size>[KMG],<size>[KMG]:<state>;<size>[KMG]-<size>[KMG]:<state>``, |
| 350 |
where ``<size>`` is the THP size (must be a power of 2 of PAGE_SIZE and |
| 351 |
supported anonymous THP) and ``<state>`` is one of ``always``, ``madvise``, |
| 352 |
``never`` or ``inherit``. |
| 353 |
|
| 354 |
For example, the following will set 16K, 32K, 64K THP to ``always``, |
| 355 |
set 128K, 512K to ``inherit``, set 256K to ``madvise`` and 1M, 2M |
| 356 |
to ``never``:: |
| 357 |
|
| 358 |
thp_anon=16K-64K:always;128K,512K:inherit;256K:madvise;1M-2M:never |
| 359 |
|
| 360 |
``thp_anon=`` may be specified multiple times to configure all THP sizes as |
| 361 |
required. If ``thp_anon=`` is specified at least once, any anon THP sizes |
| 362 |
not explicitly configured on the command line are implicitly set to |
| 363 |
``never``. |
| 364 |
|
| 365 |
``transparent_hugepage`` setting only affects the global toggle. If |
| 366 |
``thp_anon`` is not specified, PMD_ORDER THP will default to ``inherit``. |
| 367 |
However, if a valid ``thp_anon`` setting is provided by the user, the |
| 368 |
PMD_ORDER THP policy will be overridden. If the policy for PMD_ORDER |
| 369 |
is not defined within a valid ``thp_anon``, its policy will default to |
| 370 |
``never``. |
| 371 |
|
| 372 |
Similarly to ``transparent_hugepage``, you can control the hugepage |
| 373 |
allocation policy for the internal shmem mount by using the kernel parameter |
| 374 |
``transparent_hugepage_shmem=<policy>``, where ``<policy>`` is one of the |
| 375 |
seven valid policies for shmem (``always``, ``within_size``, ``advise``, |
| 376 |
``never``, ``deny``, and ``force``). |
| 377 |
|
| 378 |
Similarly to ``transparent_hugepage_shmem``, you can control the default |
| 379 |
hugepage allocation policy for the tmpfs mount by using the kernel parameter |
| 380 |
``transparent_hugepage_tmpfs=<policy>``, where ``<policy>`` is one of the |
| 381 |
four valid policies for tmpfs (``always``, ``within_size``, ``advise``, |
| 382 |
``never``). The tmpfs mount default policy is ``never``. |
| 383 |
|
| 384 |
Additionally, Kconfig options are available to set the default hugepage |
| 385 |
policies for shmem (``CONFIG_TRANSPARENT_HUGEPAGE_SHMEM_HUGE_*``) and tmpfs |
| 386 |
(``CONFIG_TRANSPARENT_HUGEPAGE_TMPFS_HUGE_*``) at build time. Refer to the |
| 387 |
Kconfig help for more details. |
| 388 |
|
| 389 |
In the same manner as ``thp_anon`` controls each supported anonymous THP |
| 390 |
size, ``thp_shmem`` controls each supported shmem THP size. ``thp_shmem`` |
| 391 |
has the same format as ``thp_anon``, but also supports the policy |
| 392 |
``within_size``. |
| 393 |
|
| 394 |
``thp_shmem=`` may be specified multiple times to configure all THP sizes |
| 395 |
as required. If ``thp_shmem=`` is specified at least once, any shmem THP |
| 396 |
sizes not explicitly configured on the command line are implicitly set to |
| 397 |
``never``. |
| 398 |
|
| 399 |
``transparent_hugepage_shmem`` setting only affects the global toggle. If |
| 400 |
``thp_shmem`` is not specified, PMD_ORDER hugepage will default to |
| 401 |
``inherit``. However, if a valid ``thp_shmem`` setting is provided by the |
| 402 |
user, the PMD_ORDER hugepage policy will be overridden. If the policy for |
| 403 |
PMD_ORDER is not defined within a valid ``thp_shmem``, its policy will |
| 404 |
default to ``never``. |
| 405 |
|
| 406 |
Hugepages in tmpfs/shmem |
| 407 |
======================== |
| 408 |
|
| 409 |
Traditionally, tmpfs only supported a single huge page size ("PMD"). Today, |
| 410 |
it also supports smaller sizes just like anonymous memory, often referred |
| 411 |
to as "multi-size THP" (mTHP). Huge pages of any size are commonly |
| 412 |
represented in the kernel as "large folios". |
| 413 |
|
| 414 |
While there is fine control over the huge page sizes to use for the internal |
| 415 |
shmem mount (see below), ordinary tmpfs mounts will make use of all available |
| 416 |
huge page sizes without any control over the exact sizes, behaving more like |
| 417 |
other file systems. |
| 418 |
|
| 419 |
tmpfs mounts |
| 420 |
------------ |
| 421 |
|
| 422 |
The THP allocation policy for tmpfs mounts can be adjusted using the mount |
| 423 |
option: ``huge=``. It can have following values: |
| 424 |
|
| 425 |
always |
| 426 |
Attempt to allocate huge pages every time we need a new page; |
| 427 |
Always try PMD-sized huge pages first, and fall back to smaller-sized |
| 428 |
huge pages if the PMD-sized huge page allocation fails; |
| 429 |
|
| 430 |
never |
| 431 |
Do not allocate huge pages. Note that ``madvise(..., MADV_COLLAPSE)`` |
| 432 |
can still cause transparent huge pages to be obtained even if this mode |
| 433 |
is specified everywhere; |
| 434 |
|
| 435 |
within_size |
| 436 |
Only allocate huge page if it will be fully within i_size; |
| 437 |
Always try PMD-sized huge pages first, and fall back to smaller-sized |
| 438 |
huge pages if the PMD-sized huge page allocation fails; |
| 439 |
Also respect madvise() hints; |
| 440 |
|
| 441 |
advise |
| 442 |
Only allocate huge pages if requested with madvise(); |
| 443 |
|
| 444 |
Remember, that the kernel may use huge pages of all available sizes, and |
| 445 |
that no fine control as for the internal tmpfs mount is available. |
| 446 |
|
| 447 |
The default policy in the past was ``never``, but it can now be adjusted |
| 448 |
using the kernel parameter ``transparent_hugepage_tmpfs=<policy>``. |
| 449 |
|
| 450 |
``mount -o remount,huge= /mountpoint`` works fine after mount: remounting |
| 451 |
``huge=never`` will not attempt to break up huge pages at all, just stop more |
| 452 |
from being allocated. |
| 453 |
|
| 454 |
In addition to policies listed above, the sysfs knob |
| 455 |
/sys/kernel/mm/transparent_hugepage/shmem_enabled will affect the |
| 456 |
allocation policy of tmpfs mounts, when set to the following values: |
| 457 |
|
| 458 |
deny |
| 459 |
For use in emergencies, to force the huge option off from |
| 460 |
all mounts; |
| 461 |
force |
| 462 |
Force the huge option on for all - very useful for testing; |
| 463 |
|
| 464 |
shmem / internal tmpfs |
| 465 |
---------------------- |
| 466 |
The mount internal tmpfs mount is used for SysV SHM, memfds, shared anonymous |
| 467 |
mmaps (of /dev/zero or MAP_ANONYMOUS), GPU drivers' DRM objects, Ashmem. |
| 468 |
|
| 469 |
To control the THP allocation policy for this internal tmpfs mount, the |
| 470 |
sysfs knob /sys/kernel/mm/transparent_hugepage/shmem_enabled and the knobs |
| 471 |
per THP size in |
| 472 |
'/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/shmem_enabled' |
| 473 |
can be used. |
| 474 |
|
| 475 |
The global knob has the same semantics as the ``huge=`` mount options |
| 476 |
for tmpfs mounts, except that the different huge page sizes can be controlled |
| 477 |
individually, and will only use the setting of the global knob when the |
| 478 |
per-size knob is set to 'inherit'. |
| 479 |
|
| 480 |
The options 'force' and 'deny' are dropped for the individual sizes, which |
| 481 |
are rather testing artifacts from the old ages. |
| 482 |
|
| 483 |
always |
| 484 |
Attempt to allocate <size> huge pages every time we need a new page; |
| 485 |
|
| 486 |
inherit |
| 487 |
Inherit the top-level "shmem_enabled" value. By default, PMD-sized hugepages |
| 488 |
have enabled="inherit" and all other hugepage sizes have enabled="never"; |
| 489 |
|
| 490 |
never |
| 491 |
Do not allocate <size> huge pages. Note that ``madvise(..., |
| 492 |
MADV_COLLAPSE)`` can still cause transparent huge pages to be obtained |
| 493 |
even if this mode is specified everywhere; |
| 494 |
|
| 495 |
within_size |
| 496 |
Only allocate <size> huge page if it will be fully within i_size. |
| 497 |
Also respect madvise() hints; |
| 498 |
|
| 499 |
advise |
| 500 |
Only allocate <size> huge pages if requested with madvise(); |
| 501 |
|
| 502 |
Need of application restart |
| 503 |
=========================== |
| 504 |
|
| 505 |
The transparent_hugepage/enabled and |
| 506 |
transparent_hugepage/hugepages-<size>kB/enabled values and tmpfs mount |
| 507 |
option only affect future behavior. So to make them effective you need |
| 508 |
to restart any application that could have been using hugepages. This |
| 509 |
also applies to the regions registered in khugepaged. |
| 510 |
|
| 511 |
Monitoring usage |
| 512 |
================ |
| 513 |
|
| 514 |
The number of PMD-sized anonymous transparent huge pages currently used by the |
| 515 |
system is available by reading the AnonHugePages field in ``/proc/meminfo``. |
| 516 |
To identify what applications are using PMD-sized anonymous transparent huge |
| 517 |
pages, it is necessary to read ``/proc/PID/smaps`` and count the AnonHugePages |
| 518 |
fields for each mapping. (Note that AnonHugePages only applies to traditional |
| 519 |
PMD-sized THP for historical reasons and should have been called |
| 520 |
AnonHugePmdMapped). |
| 521 |
|
| 522 |
The number of file transparent huge pages mapped to userspace is available |
| 523 |
by reading ShmemPmdMapped and ShmemHugePages fields in ``/proc/meminfo``. |
| 524 |
To identify what applications are mapping file transparent huge pages, it |
| 525 |
is necessary to read ``/proc/PID/smaps`` and count the FilePmdMapped fields |
| 526 |
for each mapping. |
| 527 |
|
| 528 |
Note that reading the smaps file is expensive and reading it |
| 529 |
frequently will incur overhead. |
| 530 |
|
| 531 |
There are a number of counters in ``/proc/vmstat`` that may be used to |
| 532 |
monitor how successfully the system is providing huge pages for use. |
| 533 |
|
| 534 |
thp_fault_alloc |
| 535 |
is incremented every time a huge page is successfully |
| 536 |
allocated and charged to handle a page fault. |
| 537 |
|
| 538 |
thp_collapse_alloc |
| 539 |
is incremented by khugepaged when it has found |
| 540 |
a range of pages to collapse into one huge page and has |
| 541 |
successfully allocated a new huge page to store the data. |
| 542 |
|
| 543 |
thp_fault_fallback |
| 544 |
is incremented if a page fault fails to allocate or charge |
| 545 |
a huge page and instead falls back to using small pages. |
| 546 |
|
| 547 |
thp_fault_fallback_charge |
| 548 |
is incremented if a page fault fails to charge a huge page and |
| 549 |
instead falls back to using small pages even though the |
| 550 |
allocation was successful. |
| 551 |
|
| 552 |
thp_collapse_alloc_failed |
| 553 |
is incremented if khugepaged found a range |
| 554 |
of pages that should be collapsed into one huge page but failed |
| 555 |
the allocation. |
| 556 |
|
| 557 |
thp_file_alloc |
| 558 |
is incremented every time a shmem huge page is successfully |
| 559 |
allocated (Note that despite being named after "file", the counter |
| 560 |
measures only shmem). |
| 561 |
|
| 562 |
thp_file_fallback |
| 563 |
is incremented if a shmem huge page is attempted to be allocated |
| 564 |
but fails and instead falls back to using small pages. (Note that |
| 565 |
despite being named after "file", the counter measures only shmem). |
| 566 |
|
| 567 |
thp_file_fallback_charge |
| 568 |
is incremented if a shmem huge page cannot be charged and instead |
| 569 |
falls back to using small pages even though the allocation was |
| 570 |
successful. (Note that despite being named after "file", the |
| 571 |
counter measures only shmem). |
| 572 |
|
| 573 |
thp_file_mapped |
| 574 |
is incremented every time a file or shmem huge page is mapped into |
| 575 |
user address space. |
| 576 |
|
| 577 |
thp_split_page |
| 578 |
is incremented every time a huge page is split into base |
| 579 |
pages. This can happen for a variety of reasons but a common |
| 580 |
reason is that a huge page is old and is being reclaimed. |
| 581 |
This action implies splitting all PMD the page mapped with. |
| 582 |
|
| 583 |
thp_split_page_failed |
| 584 |
is incremented if kernel fails to split huge |
| 585 |
page. This can happen if the page was pinned by somebody. |
| 586 |
|
| 587 |
thp_deferred_split_page |
| 588 |
is incremented when a huge page is put onto split |
| 589 |
queue. This happens when a huge page is partially unmapped and |
| 590 |
splitting it would free up some memory. Pages on split queue are |
| 591 |
going to be split under memory pressure. |
| 592 |
|
| 593 |
thp_underused_split_page |
| 594 |
is incremented when a huge page on the split queue was split |
| 595 |
because it was underused. A THP is underused if the number of |
| 596 |
zero pages in the THP is above a certain threshold |
| 597 |
(/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none). |
| 598 |
|
| 599 |
thp_split_pmd |
| 600 |
is incremented every time a PMD split into table of PTEs. |
| 601 |
This can happen, for instance, when application calls mprotect() or |
| 602 |
munmap() on part of huge page. It doesn't split huge page, only |
| 603 |
page table entry. |
| 604 |
|
| 605 |
thp_zero_page_alloc |
| 606 |
is incremented every time a huge zero page used for thp is |
| 607 |
successfully allocated. Note, it doesn't count every map of |
| 608 |
the huge zero page, only its allocation. |
| 609 |
|
| 610 |
thp_zero_page_alloc_failed |
| 611 |
is incremented if kernel fails to allocate |
| 612 |
huge zero page and falls back to using small pages. |
| 613 |
|
| 614 |
thp_swpout |
| 615 |
is incremented every time a huge page is swapout in one |
| 616 |
piece without splitting. |
| 617 |
|
| 618 |
thp_swpout_fallback |
| 619 |
is incremented if a huge page has to be split before swapout. |
| 620 |
Usually because failed to allocate some continuous swap space |
| 621 |
for the huge page. |
| 622 |
|
| 623 |
In /sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/stats, There are |
| 624 |
also individual counters for each huge page size, which can be utilized to |
| 625 |
monitor the system's effectiveness in providing huge pages for usage. Each |
| 626 |
counter has its own corresponding file. |
| 627 |
|
| 628 |
anon_fault_alloc |
| 629 |
is incremented every time a huge page is successfully |
| 630 |
allocated and charged to handle a page fault. |
| 631 |
|
| 632 |
anon_fault_fallback |
| 633 |
is incremented if a page fault fails to allocate or charge |
| 634 |
a huge page and instead falls back to using huge pages with |
| 635 |
lower orders or small pages. |
| 636 |
|
| 637 |
anon_fault_fallback_charge |
| 638 |
is incremented if a page fault fails to charge a huge page and |
| 639 |
instead falls back to using huge pages with lower orders or |
| 640 |
small pages even though the allocation was successful. |
| 641 |
|
| 642 |
zswpout |
| 643 |
is incremented every time a huge page is swapped out to zswap in one |
| 644 |
piece without splitting. |
| 645 |
|
| 646 |
swpin |
| 647 |
is incremented every time a huge page is swapped in from a non-zswap |
| 648 |
swap device in one piece. |
| 649 |
|
| 650 |
swpin_fallback |
| 651 |
is incremented if swapin fails to allocate or charge a huge page |
| 652 |
and instead falls back to using huge pages with lower orders or |
| 653 |
small pages. |
| 654 |
|
| 655 |
swpin_fallback_charge |
| 656 |
is incremented if swapin fails to charge a huge page and instead |
| 657 |
falls back to using huge pages with lower orders or small pages |
| 658 |
even though the allocation was successful. |
| 659 |
|
| 660 |
swpout |
| 661 |
is incremented every time a huge page is swapped out to a non-zswap |
| 662 |
swap device in one piece without splitting. |
| 663 |
|
| 664 |
swpout_fallback |
| 665 |
is incremented if a huge page has to be split before swapout. |
| 666 |
Usually because failed to allocate some continuous swap space |
| 667 |
for the huge page. |
| 668 |
|
| 669 |
shmem_alloc |
| 670 |
is incremented every time a shmem huge page is successfully |
| 671 |
allocated. |
| 672 |
|
| 673 |
shmem_fallback |
| 674 |
is incremented if a shmem huge page is attempted to be allocated |
| 675 |
but fails and instead falls back to using small pages. |
| 676 |
|
| 677 |
shmem_fallback_charge |
| 678 |
is incremented if a shmem huge page cannot be charged and instead |
| 679 |
falls back to using small pages even though the allocation was |
| 680 |
successful. |
| 681 |
|
| 682 |
split |
| 683 |
is incremented every time a huge page is successfully split into |
| 684 |
smaller orders. This can happen for a variety of reasons but a |
| 685 |
common reason is that a huge page is old and is being reclaimed. |
| 686 |
|
| 687 |
split_failed |
| 688 |
is incremented if kernel fails to split huge |
| 689 |
page. This can happen if the page was pinned by somebody. |
| 690 |
|
| 691 |
split_deferred |
| 692 |
is incremented when a huge page is put onto split queue. |
| 693 |
This happens when a huge page is partially unmapped and splitting |
| 694 |
it would free up some memory. Pages on split queue are going to |
| 695 |
be split under memory pressure, if splitting is possible. |
| 696 |
|
| 697 |
nr_anon |
| 698 |
the number of anonymous THP we have in the whole system. These THPs |
| 699 |
might be currently entirely mapped or have partially unmapped/unused |
| 700 |
subpages. |
| 701 |
|
| 702 |
nr_anon_partially_mapped |
| 703 |
the number of anonymous THP which are likely partially mapped, possibly |
| 704 |
wasting memory, and have been queued for deferred memory reclamation. |
| 705 |
Note that in corner some cases (e.g., failed migration), we might detect |
| 706 |
an anonymous THP as "partially mapped" and count it here, even though it |
| 707 |
is not actually partially mapped anymore. |
| 708 |
|
| 709 |
As the system ages, allocating huge pages may be expensive as the |
| 710 |
system uses memory compaction to copy data around memory to free a |
| 711 |
huge page for use. There are some counters in ``/proc/vmstat`` to help |
| 712 |
monitor this overhead. |
| 713 |
|
| 714 |
compact_stall |
| 715 |
is incremented every time a process stalls to run |
| 716 |
memory compaction so that a huge page is free for use. |
| 717 |
|
| 718 |
compact_success |
| 719 |
is incremented if the system compacted memory and |
| 720 |
freed a huge page for use. |
| 721 |
|
| 722 |
compact_fail |
| 723 |
is incremented if the system tries to compact memory |
| 724 |
but failed. |
| 725 |
|
| 726 |
It is possible to establish how long the stalls were using the function |
| 727 |
tracer to record how long was spent in __alloc_pages() and |
| 728 |
using the mm_page_alloc tracepoint to identify which allocations were |
| 729 |
for huge pages. |
| 730 |
|
| 731 |
Optimizing the applications |
| 732 |
=========================== |
| 733 |
|
| 734 |
To be guaranteed that the kernel will map a THP immediately in any |
| 735 |
memory region, the mmap region has to be hugepage naturally |
| 736 |
aligned. posix_memalign() can provide that guarantee. |
| 737 |
|
| 738 |
Hugetlbfs |
| 739 |
========= |
| 740 |
|
| 741 |
You can use hugetlbfs on a kernel that has transparent hugepage |
| 742 |
support enabled just fine as always. No difference can be noted in |
| 743 |
hugetlbfs other than there will be less overall fragmentation. All |
| 744 |
usual features belonging to hugetlbfs are preserved and |
| 745 |
unaffected. libhugetlbfs will also work fine as usual. |