Debian Codegraph @

Documentation/driver-api/cxl/platform/bios-and-efi.rst

raw
1 .. SPDX-License-Identifier: GPL-2.0
2
3 ======================
4 BIOS/EFI Configuration
5 ======================
6
7 BIOS and EFI are largely responsible for configuring static information about
8 devices (or potential future devices) such that Linux can build the appropriate
9 logical representations of these devices.
10
11 At a high level, this is what occurs during this phase of configuration.
12
13 * The bootloader starts the BIOS/EFI.
14
15 * BIOS/EFI do early device probe to determine static configuration
16
17 * BIOS/EFI creates ACPI Tables that describe static config for the OS
18
19 * BIOS/EFI create the system memory map (EFI Memory Map, E820, etc)
20
21 * BIOS/EFI calls :code:`start_kernel` and begins the Linux Early Boot process.
22
23 Much of what this section is concerned with is ACPI Table production and
24 static memory map configuration. More detail on these tables can be found
25 at :doc:`ACPI Tables <acpi>`.
26
27 .. note::
28 Platform Vendors should read carefully, as this sections has recommendations
29 on physical memory region size and alignment, memory holes, HDM interleave,
30 and what linux expects of HDM decoders trying to work with these features.
31
32
33 Linux Expectations of BIOS/EFI Software
34 =======================================
35 Linux expects BIOS/EFI software to construct sufficient ACPI tables (such as
36 CEDT, SRAT, HMAT, etc) and platform-specific configurations (such as HPA spaces
37 and host-bridge interleave configurations) to allow the Linux driver to
38 subsequently configure the devices in the CXL fabric at runtime.
39
40 Programming of HDM decoders and switch ports is not required, and may be
41 deferred to the CXL driver based on admin policy (e.g. udev rules).
42
43 Some platforms may require pre-programming HDM decoders and locking them
44 due to quirks (see: Zen5 address translation), but this is not the normal,
45 "expected" configuration path. This should be avoided if possible.
46
47 Some platforms may wish to pre-configure these resources to bring memory
48 up without requiring CXL driver support. These platform vendors should
49 test their configurations with the existing CXL driver and provide driver
50 support for their auto-configurations if features like RAS are required.
51
52 Platforms requiring boot-time programming and/or locking of CXL fabric
53 components may prevent features, such as device hot-plug, from working.
54
55 UEFI Settings
56 =============
57 If your platform supports it, the :code:`uefisettings` command can be used to
58 read/write EFI settings. Changes will be reflected on the next reboot. Kexec
59 is not a sufficient reboot.
60
61 One notable configuration here is the EFI_MEMORY_SP (Specific Purpose) bit.
62 When this is enabled, this bit tells linux to defer management of a memory
63 region to a driver (in this case, the CXL driver). Otherwise, the memory is
64 treated as "normal memory", and is exposed to the page allocator during
65 :code:`__init`.
66
67 uefisettings examples
68 ---------------------
69
70 :code:`uefisettings identify` ::
71
72 uefisettings identify
73
74 bios_vendor: xxx
75 bios_version: xxx
76 bios_release: xxx
77 bios_date: xxx
78 product_name: xxx
79 product_family: xxx
80 product_version: xxx
81
82 On some AMD platforms, the :code:`EFI_MEMORY_SP` bit is set via the :code:`CXL
83 Memory Attribute` field. This may be called something else on your platform.
84
85 :code:`uefisettings get "CXL Memory Attribute"` ::
86
87 selector: xxx
88 ...
89 question: Question {
90 name: "CXL Memory Attribute",
91 answer: "Enabled",
92 ...
93 }
94
95 Physical Memory Map
96 ===================
97
98 Physical Address Region Alignment
99 ---------------------------------
100
101 As of Linux v6.14, the hotplug memory system requires memory regions to be
102 uniform in size and alignment. While the CXL specification allows for memory
103 regions as small as 256MB, the supported memory block size and alignment for
104 hotplugged memory is architecture-defined.
105
106 A Linux memory blocks may be as small as 128MB and increase in powers of two.
107
108 * On ARM, the default block size and alignment is either 128MB or 256MB.
109
110 * On x86, the default block size is 256MB, and increases to 2GB as the
111 capacity of the system increases up to 64GB.
112
113 For best support across versions, platform vendors should place CXL memory at
114 a 2GB aligned base address, and regions should be 2GB aligned. This also helps
115 prevent the creating thousands of memory devices (one per block).
116
117 Memory Holes
118 ------------
119
120 Holes in the memory map are tricky. Consider a 4GB device located at base
121 address 0x100000000, but with the following memory map ::
122
123 ---------------------
124 | 0x100000000 |
125 | CXL |
126 | 0x1BFFFFFFF |
127 ---------------------
128 | 0x1C0000000 |
129 | MEMORY HOLE |
130 | 0x1FFFFFFFF |
131 ---------------------
132 | 0x200000000 |
133 | CXL CONT. |
134 | 0x23FFFFFFF |
135 ---------------------
136
137 There are two issues to consider:
138
139 * decoder programming, and
140 * memory block alignment.
141
142 If your architecture requires 2GB uniform size and aligned memory blocks, the
143 only capacity Linux is capable of mapping (as of v6.14) would be the capacity
144 from `0x100000000-0x180000000`. The remaining capacity will be stranded, as
145 they are not of 2GB aligned length.
146
147 Assuming your architecture and memory configuration allows 1GB memory blocks,
148 this memory map is supported and this should be presented as multiple CFMWS
149 in the CEDT that describe each side of the memory hole separately - along with
150 matching decoders.
151
152 Multiple decoders can (and should) be used to manage such a memory hole (see
153 below), but each chunk of a memory hole should be aligned to a reasonable block
154 size (larger alignment is always better). If you intend to have memory holes
155 in the memory map, expect to use one decoder per contiguous chunk of host
156 physical memory.
157
158 As of v6.14, Linux does provide support for memory hotplug of multiple
159 physical memory regions separated by a memory hole described by a single
160 HDM decoder.
161
162
163 Decoder Programming
164 ===================
165 If BIOS/EFI intends to program the decoders to be statically configured,
166 there are a few things to consider to avoid major pitfalls that will
167 prevent Linux compatibility. Some of these recommendations are not
168 required "per the specification", but Linux makes no guarantees of support
169 otherwise.
170
171
172 Translation Point
173 -----------------
174 Per the specification, the only decoders which **TRANSLATE** Host Physical
175 Address (HPA) to Device Physical Address (DPA) are the **Endpoint Decoders**.
176 All other decoders in the fabric are intended to route accesses without
177 translating the addresses.
178
179 This is heavily implied by the specification, see: ::
180
181 CXL Specification 3.1
182 8.2.4.20: CXL HDM Decoder Capability Structure
183 - Implementation Note: CXL Host Bridge and Upstream Switch Port Decoder Flow
184 - Implementation Note: Device Decoder Logic
185
186 Given this, Linux makes a strong assumption that decoders between CPU and
187 endpoint will all be programmed with addresses ranges that are subsets of
188 their parent decoder.
189
190 Due to some ambiguity in how Architecture, ACPI, PCI, and CXL specifications
191 "hand off" responsibility between domains, some early adopting platforms
192 attempted to do translation at the originating memory controller or host
193 bridge. This configuration requires a platform specific extension to the
194 driver and is not officially endorsed - despite being supported.
195
196 It is *highly recommended* **NOT** to do this; otherwise, you are on your own
197 to implement driver support for your platform.
198
199 Interleave and Configuration Flexibility
200 ----------------------------------------
201 If providing cross-host-bridge interleave, a CFMWS entry in the :doc:`CEDT
202 <acpi/cedt>` must be presented with target host-bridges for the interleaved
203 device sets (there may be multiple behind each host bridge).
204
205 If providing intra-host-bridge interleaving, only 1 CFMWS entry in the CEDT is
206 required for that host bridge - if it covers the entire capacity of the devices
207 behind the host bridge.
208
209 If intending to provide users flexibility in programming decoders beyond the
210 root, you may want to provide multiple CFMWS entries in the CEDT intended for
211 different purposes. For example, you may want to consider adding:
212
213 1) A CFMWS entry to cover all interleavable host bridges.
214 2) A CFMWS entry to cover all devices on a single host bridge.
215 3) A CFMWS entry to cover each device.
216
217 A platform may choose to add all of these, or change the mode based on a BIOS
218 setting. For each CFMWS entry, Linux expects descriptions of the described
219 memory regions in the :doc:`SRAT <acpi/srat>` to determine the number of
220 NUMA nodes it should reserve during early boot / init.
221
222 As of v6.14, Linux will create a NUMA node for each CEDT CFMWS entry, even if
223 a matching SRAT entry does not exist; however, this is not guaranteed in the
224 future and such a configuration should be avoided.
225
226 Memory Holes
227 ------------
228 If your platform includes memory holes interspersed between your CXL memory, it
229 is recommended to utilize multiple decoders to cover these regions of memory,
230 rather than try to program the decoders to accept the entire range and expect
231 Linux to manage the overlap.
232
233 For example, consider the Memory Hole described above ::
234
235 ---------------------
236 | 0x100000000 |
237 | CXL |
238 | 0x1BFFFFFFF |
239 ---------------------
240 | 0x1C0000000 |
241 | MEMORY HOLE |
242 | 0x1FFFFFFFF |
243 ---------------------
244 | 0x200000000 |
245 | CXL CONT. |
246 | 0x23FFFFFFF |
247 ---------------------
248
249 Assuming this is provided by a single device attached directly to a host bridge,
250 Linux would expect the following decoder programming ::
251
252 ----------------------- -----------------------
253 | root-decoder-0 | | root-decoder-1 |
254 | base: 0x100000000 | | base: 0x200000000 |
255 | size: 0xC0000000 | | size: 0x40000000 |
256 ----------------------- -----------------------
257 | |
258 ----------------------- -----------------------
259 | HB-decoder-0 | | HB-decoder-1 |
260 | base: 0x100000000 | | base: 0x200000000 |
261 | size: 0xC0000000 | | size: 0x40000000 |
262 ----------------------- -----------------------
263 | |
264 ----------------------- -----------------------
265 | ep-decoder-0 | | ep-decoder-1 |
266 | base: 0x100000000 | | base: 0x200000000 |
267 | size: 0xC0000000 | | size: 0x40000000 |
268 ----------------------- -----------------------
269
270 With a CEDT configuration with two CFMWS describing the above root decoders.
271
272 Linux makes no guarantee of support for strange memory hole situations.
273
274 Multi-Media Devices
275 -------------------
276 The CFMWS field of the CEDT has special restriction bits which describe whether
277 the described memory region allows volatile or persistent memory (or both). If
278 the platform intends to support either:
279
280 1) A device with multiple medias, or
281 2) Using a persistent memory device as normal memory
282
283 A platform may wish to create multiple CEDT CFMWS entries to describe the same
284 memory, with the intent of allowing the end user flexibility in how that memory
285 is configured. Linux does not presently have strong requirements in this area.