| 1 |
.. SPDX-License-Identifier: GPL-2.0 |
| 2 |
|
| 3 |
====================== |
| 4 |
BIOS/EFI Configuration |
| 5 |
====================== |
| 6 |
|
| 7 |
BIOS and EFI are largely responsible for configuring static information about |
| 8 |
devices (or potential future devices) such that Linux can build the appropriate |
| 9 |
logical representations of these devices. |
| 10 |
|
| 11 |
At a high level, this is what occurs during this phase of configuration. |
| 12 |
|
| 13 |
* The bootloader starts the BIOS/EFI. |
| 14 |
|
| 15 |
* BIOS/EFI do early device probe to determine static configuration |
| 16 |
|
| 17 |
* BIOS/EFI creates ACPI Tables that describe static config for the OS |
| 18 |
|
| 19 |
* BIOS/EFI create the system memory map (EFI Memory Map, E820, etc) |
| 20 |
|
| 21 |
* BIOS/EFI calls :code:`start_kernel` and begins the Linux Early Boot process. |
| 22 |
|
| 23 |
Much of what this section is concerned with is ACPI Table production and |
| 24 |
static memory map configuration. More detail on these tables can be found |
| 25 |
at :doc:`ACPI Tables <acpi>`. |
| 26 |
|
| 27 |
.. note:: |
| 28 |
Platform Vendors should read carefully, as this sections has recommendations |
| 29 |
on physical memory region size and alignment, memory holes, HDM interleave, |
| 30 |
and what linux expects of HDM decoders trying to work with these features. |
| 31 |
|
| 32 |
|
| 33 |
Linux Expectations of BIOS/EFI Software |
| 34 |
======================================= |
| 35 |
Linux expects BIOS/EFI software to construct sufficient ACPI tables (such as |
| 36 |
CEDT, SRAT, HMAT, etc) and platform-specific configurations (such as HPA spaces |
| 37 |
and host-bridge interleave configurations) to allow the Linux driver to |
| 38 |
subsequently configure the devices in the CXL fabric at runtime. |
| 39 |
|
| 40 |
Programming of HDM decoders and switch ports is not required, and may be |
| 41 |
deferred to the CXL driver based on admin policy (e.g. udev rules). |
| 42 |
|
| 43 |
Some platforms may require pre-programming HDM decoders and locking them |
| 44 |
due to quirks (see: Zen5 address translation), but this is not the normal, |
| 45 |
"expected" configuration path. This should be avoided if possible. |
| 46 |
|
| 47 |
Some platforms may wish to pre-configure these resources to bring memory |
| 48 |
up without requiring CXL driver support. These platform vendors should |
| 49 |
test their configurations with the existing CXL driver and provide driver |
| 50 |
support for their auto-configurations if features like RAS are required. |
| 51 |
|
| 52 |
Platforms requiring boot-time programming and/or locking of CXL fabric |
| 53 |
components may prevent features, such as device hot-plug, from working. |
| 54 |
|
| 55 |
UEFI Settings |
| 56 |
============= |
| 57 |
If your platform supports it, the :code:`uefisettings` command can be used to |
| 58 |
read/write EFI settings. Changes will be reflected on the next reboot. Kexec |
| 59 |
is not a sufficient reboot. |
| 60 |
|
| 61 |
One notable configuration here is the EFI_MEMORY_SP (Specific Purpose) bit. |
| 62 |
When this is enabled, this bit tells linux to defer management of a memory |
| 63 |
region to a driver (in this case, the CXL driver). Otherwise, the memory is |
| 64 |
treated as "normal memory", and is exposed to the page allocator during |
| 65 |
:code:`__init`. |
| 66 |
|
| 67 |
uefisettings examples |
| 68 |
--------------------- |
| 69 |
|
| 70 |
:code:`uefisettings identify` :: |
| 71 |
|
| 72 |
uefisettings identify |
| 73 |
|
| 74 |
bios_vendor: xxx |
| 75 |
bios_version: xxx |
| 76 |
bios_release: xxx |
| 77 |
bios_date: xxx |
| 78 |
product_name: xxx |
| 79 |
product_family: xxx |
| 80 |
product_version: xxx |
| 81 |
|
| 82 |
On some AMD platforms, the :code:`EFI_MEMORY_SP` bit is set via the :code:`CXL |
| 83 |
Memory Attribute` field. This may be called something else on your platform. |
| 84 |
|
| 85 |
:code:`uefisettings get "CXL Memory Attribute"` :: |
| 86 |
|
| 87 |
selector: xxx |
| 88 |
... |
| 89 |
question: Question { |
| 90 |
name: "CXL Memory Attribute", |
| 91 |
answer: "Enabled", |
| 92 |
... |
| 93 |
} |
| 94 |
|
| 95 |
Physical Memory Map |
| 96 |
=================== |
| 97 |
|
| 98 |
Physical Address Region Alignment |
| 99 |
--------------------------------- |
| 100 |
|
| 101 |
As of Linux v6.14, the hotplug memory system requires memory regions to be |
| 102 |
uniform in size and alignment. While the CXL specification allows for memory |
| 103 |
regions as small as 256MB, the supported memory block size and alignment for |
| 104 |
hotplugged memory is architecture-defined. |
| 105 |
|
| 106 |
A Linux memory blocks may be as small as 128MB and increase in powers of two. |
| 107 |
|
| 108 |
* On ARM, the default block size and alignment is either 128MB or 256MB. |
| 109 |
|
| 110 |
* On x86, the default block size is 256MB, and increases to 2GB as the |
| 111 |
capacity of the system increases up to 64GB. |
| 112 |
|
| 113 |
For best support across versions, platform vendors should place CXL memory at |
| 114 |
a 2GB aligned base address, and regions should be 2GB aligned. This also helps |
| 115 |
prevent the creating thousands of memory devices (one per block). |
| 116 |
|
| 117 |
Memory Holes |
| 118 |
------------ |
| 119 |
|
| 120 |
Holes in the memory map are tricky. Consider a 4GB device located at base |
| 121 |
address 0x100000000, but with the following memory map :: |
| 122 |
|
| 123 |
--------------------- |
| 124 |
| 0x100000000 | |
| 125 |
| CXL | |
| 126 |
| 0x1BFFFFFFF | |
| 127 |
--------------------- |
| 128 |
| 0x1C0000000 | |
| 129 |
| MEMORY HOLE | |
| 130 |
| 0x1FFFFFFFF | |
| 131 |
--------------------- |
| 132 |
| 0x200000000 | |
| 133 |
| CXL CONT. | |
| 134 |
| 0x23FFFFFFF | |
| 135 |
--------------------- |
| 136 |
|
| 137 |
There are two issues to consider: |
| 138 |
|
| 139 |
* decoder programming, and |
| 140 |
* memory block alignment. |
| 141 |
|
| 142 |
If your architecture requires 2GB uniform size and aligned memory blocks, the |
| 143 |
only capacity Linux is capable of mapping (as of v6.14) would be the capacity |
| 144 |
from `0x100000000-0x180000000`. The remaining capacity will be stranded, as |
| 145 |
they are not of 2GB aligned length. |
| 146 |
|
| 147 |
Assuming your architecture and memory configuration allows 1GB memory blocks, |
| 148 |
this memory map is supported and this should be presented as multiple CFMWS |
| 149 |
in the CEDT that describe each side of the memory hole separately - along with |
| 150 |
matching decoders. |
| 151 |
|
| 152 |
Multiple decoders can (and should) be used to manage such a memory hole (see |
| 153 |
below), but each chunk of a memory hole should be aligned to a reasonable block |
| 154 |
size (larger alignment is always better). If you intend to have memory holes |
| 155 |
in the memory map, expect to use one decoder per contiguous chunk of host |
| 156 |
physical memory. |
| 157 |
|
| 158 |
As of v6.14, Linux does provide support for memory hotplug of multiple |
| 159 |
physical memory regions separated by a memory hole described by a single |
| 160 |
HDM decoder. |
| 161 |
|
| 162 |
|
| 163 |
Decoder Programming |
| 164 |
=================== |
| 165 |
If BIOS/EFI intends to program the decoders to be statically configured, |
| 166 |
there are a few things to consider to avoid major pitfalls that will |
| 167 |
prevent Linux compatibility. Some of these recommendations are not |
| 168 |
required "per the specification", but Linux makes no guarantees of support |
| 169 |
otherwise. |
| 170 |
|
| 171 |
|
| 172 |
Translation Point |
| 173 |
----------------- |
| 174 |
Per the specification, the only decoders which **TRANSLATE** Host Physical |
| 175 |
Address (HPA) to Device Physical Address (DPA) are the **Endpoint Decoders**. |
| 176 |
All other decoders in the fabric are intended to route accesses without |
| 177 |
translating the addresses. |
| 178 |
|
| 179 |
This is heavily implied by the specification, see: :: |
| 180 |
|
| 181 |
CXL Specification 3.1 |
| 182 |
8.2.4.20: CXL HDM Decoder Capability Structure |
| 183 |
- Implementation Note: CXL Host Bridge and Upstream Switch Port Decoder Flow |
| 184 |
- Implementation Note: Device Decoder Logic |
| 185 |
|
| 186 |
Given this, Linux makes a strong assumption that decoders between CPU and |
| 187 |
endpoint will all be programmed with addresses ranges that are subsets of |
| 188 |
their parent decoder. |
| 189 |
|
| 190 |
Due to some ambiguity in how Architecture, ACPI, PCI, and CXL specifications |
| 191 |
"hand off" responsibility between domains, some early adopting platforms |
| 192 |
attempted to do translation at the originating memory controller or host |
| 193 |
bridge. This configuration requires a platform specific extension to the |
| 194 |
driver and is not officially endorsed - despite being supported. |
| 195 |
|
| 196 |
It is *highly recommended* **NOT** to do this; otherwise, you are on your own |
| 197 |
to implement driver support for your platform. |
| 198 |
|
| 199 |
Interleave and Configuration Flexibility |
| 200 |
---------------------------------------- |
| 201 |
If providing cross-host-bridge interleave, a CFMWS entry in the :doc:`CEDT |
| 202 |
<acpi/cedt>` must be presented with target host-bridges for the interleaved |
| 203 |
device sets (there may be multiple behind each host bridge). |
| 204 |
|
| 205 |
If providing intra-host-bridge interleaving, only 1 CFMWS entry in the CEDT is |
| 206 |
required for that host bridge - if it covers the entire capacity of the devices |
| 207 |
behind the host bridge. |
| 208 |
|
| 209 |
If intending to provide users flexibility in programming decoders beyond the |
| 210 |
root, you may want to provide multiple CFMWS entries in the CEDT intended for |
| 211 |
different purposes. For example, you may want to consider adding: |
| 212 |
|
| 213 |
1) A CFMWS entry to cover all interleavable host bridges. |
| 214 |
2) A CFMWS entry to cover all devices on a single host bridge. |
| 215 |
3) A CFMWS entry to cover each device. |
| 216 |
|
| 217 |
A platform may choose to add all of these, or change the mode based on a BIOS |
| 218 |
setting. For each CFMWS entry, Linux expects descriptions of the described |
| 219 |
memory regions in the :doc:`SRAT <acpi/srat>` to determine the number of |
| 220 |
NUMA nodes it should reserve during early boot / init. |
| 221 |
|
| 222 |
As of v6.14, Linux will create a NUMA node for each CEDT CFMWS entry, even if |
| 223 |
a matching SRAT entry does not exist; however, this is not guaranteed in the |
| 224 |
future and such a configuration should be avoided. |
| 225 |
|
| 226 |
Memory Holes |
| 227 |
------------ |
| 228 |
If your platform includes memory holes interspersed between your CXL memory, it |
| 229 |
is recommended to utilize multiple decoders to cover these regions of memory, |
| 230 |
rather than try to program the decoders to accept the entire range and expect |
| 231 |
Linux to manage the overlap. |
| 232 |
|
| 233 |
For example, consider the Memory Hole described above :: |
| 234 |
|
| 235 |
--------------------- |
| 236 |
| 0x100000000 | |
| 237 |
| CXL | |
| 238 |
| 0x1BFFFFFFF | |
| 239 |
--------------------- |
| 240 |
| 0x1C0000000 | |
| 241 |
| MEMORY HOLE | |
| 242 |
| 0x1FFFFFFFF | |
| 243 |
--------------------- |
| 244 |
| 0x200000000 | |
| 245 |
| CXL CONT. | |
| 246 |
| 0x23FFFFFFF | |
| 247 |
--------------------- |
| 248 |
|
| 249 |
Assuming this is provided by a single device attached directly to a host bridge, |
| 250 |
Linux would expect the following decoder programming :: |
| 251 |
|
| 252 |
----------------------- ----------------------- |
| 253 |
| root-decoder-0 | | root-decoder-1 | |
| 254 |
| base: 0x100000000 | | base: 0x200000000 | |
| 255 |
| size: 0xC0000000 | | size: 0x40000000 | |
| 256 |
----------------------- ----------------------- |
| 257 |
| | |
| 258 |
----------------------- ----------------------- |
| 259 |
| HB-decoder-0 | | HB-decoder-1 | |
| 260 |
| base: 0x100000000 | | base: 0x200000000 | |
| 261 |
| size: 0xC0000000 | | size: 0x40000000 | |
| 262 |
----------------------- ----------------------- |
| 263 |
| | |
| 264 |
----------------------- ----------------------- |
| 265 |
| ep-decoder-0 | | ep-decoder-1 | |
| 266 |
| base: 0x100000000 | | base: 0x200000000 | |
| 267 |
| size: 0xC0000000 | | size: 0x40000000 | |
| 268 |
----------------------- ----------------------- |
| 269 |
|
| 270 |
With a CEDT configuration with two CFMWS describing the above root decoders. |
| 271 |
|
| 272 |
Linux makes no guarantee of support for strange memory hole situations. |
| 273 |
|
| 274 |
Multi-Media Devices |
| 275 |
------------------- |
| 276 |
The CFMWS field of the CEDT has special restriction bits which describe whether |
| 277 |
the described memory region allows volatile or persistent memory (or both). If |
| 278 |
the platform intends to support either: |
| 279 |
|
| 280 |
1) A device with multiple medias, or |
| 281 |
2) Using a persistent memory device as normal memory |
| 282 |
|
| 283 |
A platform may wish to create multiple CEDT CFMWS entries to describe the same |
| 284 |
memory, with the intent of allowing the end user flexibility in how that memory |
| 285 |
is configured. Linux does not presently have strong requirements in this area. |