| 1 |
======================= |
| 2 |
NUMA Memory Performance |
| 3 |
======================= |
| 4 |
|
| 5 |
NUMA Locality |
| 6 |
============= |
| 7 |
|
| 8 |
Some platforms may have multiple types of memory attached to a compute |
| 9 |
node. These disparate memory ranges may share some characteristics, such |
| 10 |
as CPU cache coherence, but may have different performance. For example, |
| 11 |
different media types and buses affect bandwidth and latency. |
| 12 |
|
| 13 |
A system supports such heterogeneous memory by grouping each memory type |
| 14 |
under different domains, or "nodes", based on locality and performance |
| 15 |
characteristics. Some memory may share the same node as a CPU, and others |
| 16 |
are provided as memory only nodes. While memory only nodes do not provide |
| 17 |
CPUs, they may still be local to one or more compute nodes relative to |
| 18 |
other nodes. The following diagram shows one such example of two compute |
| 19 |
nodes with local memory and a memory only node for each of compute node:: |
| 20 |
|
| 21 |
+------------------+ +------------------+ |
| 22 |
| Compute Node 0 +-----+ Compute Node 1 | |
| 23 |
| Local Node0 Mem | | Local Node1 Mem | |
| 24 |
+--------+---------+ +--------+---------+ |
| 25 |
| | |
| 26 |
+--------+---------+ +--------+---------+ |
| 27 |
| Slower Node2 Mem | | Slower Node3 Mem | |
| 28 |
+------------------+ +--------+---------+ |
| 29 |
|
| 30 |
A "memory initiator" is a node containing one or more devices such as |
| 31 |
CPUs or separate memory I/O devices that can initiate memory requests. |
| 32 |
A "memory target" is a node containing one or more physical address |
| 33 |
ranges accessible from one or more memory initiators. |
| 34 |
|
| 35 |
When multiple memory initiators exist, they may not all have the same |
| 36 |
performance when accessing a given memory target. Each initiator-target |
| 37 |
pair may be organized into different ranked access classes to represent |
| 38 |
this relationship. The highest performing initiator to a given target |
| 39 |
is considered to be one of that target's local initiators, and given |
| 40 |
the highest access class, 0. Any given target may have one or more |
| 41 |
local initiators, and any given initiator may have multiple local |
| 42 |
memory targets. |
| 43 |
|
| 44 |
To aid applications matching memory targets with their initiators, the |
| 45 |
kernel provides symlinks to each other. The following example lists the |
| 46 |
relationship for the access class "0" memory initiators and targets:: |
| 47 |
|
| 48 |
# symlinks -v /sys/devices/system/node/nodeX/access0/targets/ |
| 49 |
relative: /sys/devices/system/node/nodeX/access0/targets/nodeY -> ../../nodeY |
| 50 |
|
| 51 |
# symlinks -v /sys/devices/system/node/nodeY/access0/initiators/ |
| 52 |
relative: /sys/devices/system/node/nodeY/access0/initiators/nodeX -> ../../nodeX |
| 53 |
|
| 54 |
A memory initiator may have multiple memory targets in the same access |
| 55 |
class. The target memory's initiators in a given class indicate the |
| 56 |
nodes' access characteristics share the same performance relative to other |
| 57 |
linked initiator nodes. Each target within an initiator's access class, |
| 58 |
though, do not necessarily perform the same as each other. |
| 59 |
|
| 60 |
The access class "1" is used to allow differentiation between initiators |
| 61 |
that are CPUs and hence suitable for generic task scheduling, and |
| 62 |
IO initiators such as GPUs and NICs. Unlike access class 0, only |
| 63 |
nodes containing CPUs are considered. |
| 64 |
|
| 65 |
NUMA Performance |
| 66 |
================ |
| 67 |
|
| 68 |
Applications may wish to consider which node they want their memory to |
| 69 |
be allocated from based on the node's performance characteristics. If |
| 70 |
the system provides these attributes, the kernel exports them under the |
| 71 |
node sysfs hierarchy by appending the attributes directory under the |
| 72 |
memory node's access class 0 initiators as follows:: |
| 73 |
|
| 74 |
/sys/devices/system/node/nodeY/access0/initiators/ |
| 75 |
|
| 76 |
These attributes apply only when accessed from nodes that have the |
| 77 |
are linked under the this access's initiators. |
| 78 |
|
| 79 |
The performance characteristics the kernel provides for the local initiators |
| 80 |
are exported are as follows:: |
| 81 |
|
| 82 |
# tree -P "read*|write*" /sys/devices/system/node/nodeY/access0/initiators/ |
| 83 |
/sys/devices/system/node/nodeY/access0/initiators/ |
| 84 |
|-- read_bandwidth |
| 85 |
|-- read_latency |
| 86 |
|-- write_bandwidth |
| 87 |
`-- write_latency |
| 88 |
|
| 89 |
The bandwidth attributes are provided in MiB/second. |
| 90 |
|
| 91 |
The latency attributes are provided in nanoseconds. |
| 92 |
|
| 93 |
The values reported here correspond to the rated latency and bandwidth |
| 94 |
for the platform. |
| 95 |
|
| 96 |
Access class 1 takes the same form but only includes values for CPU to |
| 97 |
memory activity. |
| 98 |
|
| 99 |
NUMA Cache |
| 100 |
========== |
| 101 |
|
| 102 |
System memory may be constructed in a hierarchy of elements with various |
| 103 |
performance characteristics in order to provide large address space of |
| 104 |
slower performing memory cached by a smaller higher performing memory. The |
| 105 |
system physical addresses memory initiators are aware of are provided |
| 106 |
by the last memory level in the hierarchy. The system meanwhile uses |
| 107 |
higher performing memory to transparently cache access to progressively |
| 108 |
slower levels. |
| 109 |
|
| 110 |
The term "far memory" is used to denote the last level memory in the |
| 111 |
hierarchy. Each increasing cache level provides higher performing |
| 112 |
initiator access, and the term "near memory" represents the fastest |
| 113 |
cache provided by the system. |
| 114 |
|
| 115 |
This numbering is different than CPU caches where the cache level (ex: |
| 116 |
L1, L2, L3) uses the CPU-side view where each increased level is lower |
| 117 |
performing. In contrast, the memory cache level is centric to the last |
| 118 |
level memory, so the higher numbered cache level corresponds to memory |
| 119 |
nearer to the CPU, and further from far memory. |
| 120 |
|
| 121 |
The memory-side caches are not directly addressable by software. When |
| 122 |
software accesses a system address, the system will return it from the |
| 123 |
near memory cache if it is present. If it is not present, the system |
| 124 |
accesses the next level of memory until there is either a hit in that |
| 125 |
cache level, or it reaches far memory. |
| 126 |
|
| 127 |
An application does not need to know about caching attributes in order |
| 128 |
to use the system. Software may optionally query the memory cache |
| 129 |
attributes in order to maximize the performance out of such a setup. |
| 130 |
If the system provides a way for the kernel to discover this information, |
| 131 |
for example with ACPI HMAT (Heterogeneous Memory Attribute Table), |
| 132 |
the kernel will append these attributes to the NUMA node memory target. |
| 133 |
|
| 134 |
When the kernel first registers a memory cache with a node, the kernel |
| 135 |
will create the following directory:: |
| 136 |
|
| 137 |
/sys/devices/system/node/nodeX/memory_side_cache/ |
| 138 |
|
| 139 |
If that directory is not present, the system either does not provide |
| 140 |
a memory-side cache, or that information is not accessible to the kernel. |
| 141 |
|
| 142 |
The attributes for each level of cache is provided under its cache |
| 143 |
level index:: |
| 144 |
|
| 145 |
/sys/devices/system/node/nodeX/memory_side_cache/indexA/ |
| 146 |
/sys/devices/system/node/nodeX/memory_side_cache/indexB/ |
| 147 |
/sys/devices/system/node/nodeX/memory_side_cache/indexC/ |
| 148 |
|
| 149 |
Each cache level's directory provides its attributes. For example, the |
| 150 |
following shows a single cache level and the attributes available for |
| 151 |
software to query:: |
| 152 |
|
| 153 |
# tree /sys/devices/system/node/node0/memory_side_cache/ |
| 154 |
/sys/devices/system/node/node0/memory_side_cache/ |
| 155 |
|-- index1 |
| 156 |
| |-- indexing |
| 157 |
| |-- line_size |
| 158 |
| |-- size |
| 159 |
| `-- write_policy |
| 160 |
|
| 161 |
The "indexing" will be 0 if it is a direct-mapped cache, and non-zero |
| 162 |
for any other indexed based, multi-way associativity. |
| 163 |
|
| 164 |
The "line_size" is the number of bytes accessed from the next cache |
| 165 |
level on a miss. |
| 166 |
|
| 167 |
The "size" is the number of bytes provided by this cache level. |
| 168 |
|
| 169 |
The "write_policy" will be 0 for write-back, and non-zero for |
| 170 |
write-through caching. |
| 171 |
|
| 172 |
See Also |
| 173 |
======== |
| 174 |
|
| 175 |
[1] https://www.uefi.org/sites/default/files/resources/ACPI_6_2.pdf |
| 176 |
- Section 5.2.27 |