Debian Codegraph @

Documentation/driver-api/cxl/conventions/cxl-atl.rst

raw
1 .. SPDX-License-Identifier: GPL-2.0
2
3 ACPI PRM CXL Address Translation
4 ================================
5
6 Document
7 --------
8
9 CXL Revision 3.2, Version 1.0
10
11 License
12 -------
13
14 SPDX-License Identifier: CC-BY-4.0
15
16 Creator/Contributors
17 --------------------
18
19 - Robert Richter, AMD et al.
20
21 Summary of the Change
22 ---------------------
23
24 The CXL Fixed Memory Window Structures (CFMWS) describe zero or more Host
25 Physical Address (HPA) windows associated with one or more CXL Host Bridges.
26 Each HPA range of a CXL Host Bridge is represented by a CFMWS entry. An HPA
27 range may include addresses currently assigned to CXL.mem devices, or an OS may
28 assign ranges from an address window to a device.
29
30 Host-managed Device Memory is Device-attached memory that is mapped to system
31 coherent address space and accessible to the Host using standard write-back
32 semantics. The managed address range is configured in the CXL HDM Decoder
33 registers of the device. An HDM Decoder in a device is responsible for
34 converting HPA into DPA by stripping off specific address bits.
35
36 CXL devices and CXL bridges use the same HPA space. It is common across all
37 components that belong to the same host domain. The view of the address region
38 must be consistent on the CXL.mem path between the Host and the Device.
39
40 This is described in the *CXL 3.2 specification* (Table 1-1, 3.3.1,
41 8.2.4.20, 9.13.1, 9.18.1.3). [#cxl-spec-3.2]_
42
43 Depending on the interconnect architecture of the platform, components attached
44 to a host may not share the same host physical address space. Those platforms
45 need address translation to convert an HPA between the host and the attached
46 component, such as a CXL device. The translation mechanism is host-specific and
47 implementation dependent.
48
49 For example, x86 AMD platforms use a Data Fabric that manages access to physical
50 memory. Devices have their own memory space and can be configured to use
51 'Normalized addresses' different from System Physical Addresses (SPA). Address
52 translation is then needed. For details, see
53 :doc:`x86 AMD Address Translation </admin-guide/RAS/address-translation>`.
54
55 Those AMD platforms provide PRM [#prm-spec]_ handlers in firmware to perform
56 various types of address translation, including for CXL endpoints. AMD Zen5
57 systems implement the ACPI PRM CXL Address Translation firmware call. The ACPI
58 PRM handler has a specific GUID to uniquely identify platforms with support for
59 Normalized addressing. This is documented in the *ACPI v6.5 Porting Guide*
60 (Address Translation - CXL DPA to System Physical Address). [#amd-ppr-58088]_
61
62 When in Normalized address mode, HDM decoder address ranges must be configured
63 and handled differently. Hardware addresses used in the HDM decoder
64 configurations of an endpoint are not SPA and need to be translated from the
65 address range of the endpoint to that of the CXL host bridge. This is especially
66 important for finding an endpoint's associated CXL Host Bridge and HPA window
67 described in the CFMWS. Additionally, the interleave decoding is done by the
68 Data Fabric and the endpoint does not perform decoding when converting HPA to
69 DPA. Instead, interleaving is switched off for the endpoint (1-way). Finally,
70 address translation might also be needed to inspect the endpoint's hardware
71 addresses, such as during profiling, tracing, or error handling.
72
73 For example, with Normalized addressing the HDM decoders could look as follows::
74
75 -------------------------------
76 | Root Decoder (CFMWS) |
77 | SPA Range: 0x850000000 |
78 | Size: 0x8000000000 (512 GB) |
79 | Interleave Ways: 1 |
80 -------------------------------
81 |
82 v
83 -------------------------------
84 | Host Bridge Decoder (HDM) |
85 | SPA Range: 0x850000000 |
86 | Size: 0x8000000000 (512 GB) |
87 | Interleave Ways: 4 |
88 | Targets: endpoint5,8,11,13 |
89 | Granularity: 256 |
90 -------------------------------
91 |
92 -----------------------------+------------------------------
93 | | | |
94 v v v v
95 ------------------- ------------------- ------------------- -------------------
96 | endpoint5 | | endpoint8 | | endpoint11 | | endpoint13 |
97 | decoder5.0 | | decoder8.0 | | decoder11.0 | | decoder13.0 |
98 | PCIe: | | PCIe: | | PCIe: | | PCIe: |
99 | 0000:e2:00.0 | | 0000:e3:00.0 | | 0000:e4:00.0 | | 0000:e1:00.0 |
100 | DPA: | | DPA: | | DPA: | | DPA: |
101 | Start: 0x0 | | Start: 0x0 | | Start: 0x0 | | Start: 0x0 |
102 | Size: | | Size: | | Size: | | Size: |
103 | 0x2000000000 | | 0x2000000000 | | 0x2000000000 | | 0x2000000000 |
104 | (128 GB) | | (128 GB) | | (128 GB) | | (128 GB) |
105 | Interleaving: | | Interleaving: | | Interleaving: | | Interleaving: |
106 | Ways: 1 | | Ways: 1 | | Ways: 1 | | Ways: 1 |
107 | Gran: 256 | | Gran: 256 | | Gran: 256 | | Gran: 256 |
108 ------------------- ------------------- ------------------- -------------------
109 | | | |
110 v v v v
111 DPA DPA DPA DPA
112
113 This shows the representation in sysfs:
114
115 .. code-block:: none
116
117 /sys/bus/cxl/devices/endpoint5/decoder5.0/interleave_granularity:256
118 /sys/bus/cxl/devices/endpoint5/decoder5.0/interleave_ways:1
119 /sys/bus/cxl/devices/endpoint5/decoder5.0/size:0x2000000000
120 /sys/bus/cxl/devices/endpoint5/decoder5.0/start:0x0
121 /sys/bus/cxl/devices/endpoint8/decoder8.0/interleave_granularity:256
122 /sys/bus/cxl/devices/endpoint8/decoder8.0/interleave_ways:1
123 /sys/bus/cxl/devices/endpoint8/decoder8.0/size:0x2000000000
124 /sys/bus/cxl/devices/endpoint8/decoder8.0/start:0x0
125 /sys/bus/cxl/devices/endpoint11/decoder11.0/interleave_granularity:256
126 /sys/bus/cxl/devices/endpoint11/decoder11.0/interleave_ways:1
127 /sys/bus/cxl/devices/endpoint11/decoder11.0/size:0x2000000000
128 /sys/bus/cxl/devices/endpoint11/decoder11.0/start:0x0
129 /sys/bus/cxl/devices/endpoint13/decoder13.0/interleave_granularity:256
130 /sys/bus/cxl/devices/endpoint13/decoder13.0/interleave_ways:1
131 /sys/bus/cxl/devices/endpoint13/decoder13.0/size:0x2000000000
132 /sys/bus/cxl/devices/endpoint13/decoder13.0/start:0x0
133
134 Note that the endpoint interleaving configurations use direct mapping (1-way).
135
136 With PRM calls, the kernel can determine the following mappings:
137
138 .. code-block:: none
139
140 cxl decoder5.0: address mapping found for 0000:e2:00.0 (hpa -> spa):
141 0x0+0x2000000000 -> 0x850000000+0x8000000000 ways:4 granularity:256
142 cxl decoder8.0: address mapping found for 0000:e3:00.0 (hpa -> spa):
143 0x0+0x2000000000 -> 0x850000000+0x8000000000 ways:4 granularity:256
144 cxl decoder11.0: address mapping found for 0000:e4:00.0 (hpa -> spa):
145 0x0+0x2000000000 -> 0x850000000+0x8000000000 ways:4 granularity:256
146 cxl decoder13.0: address mapping found for 0000:e1:00.0 (hpa -> spa):
147 0x0+0x2000000000 -> 0x850000000+0x8000000000 ways:4 granularity:256
148
149 The corresponding CXL host bridge (HDM) decoders and root decoder (CFMWS) match
150 the calculated endpoint mappings shown:
151
152 .. code-block:: none
153
154 /sys/bus/cxl/devices/port1/decoder1.0/interleave_granularity:256
155 /sys/bus/cxl/devices/port1/decoder1.0/interleave_ways:4
156 /sys/bus/cxl/devices/port1/decoder1.0/size:0x8000000000
157 /sys/bus/cxl/devices/port1/decoder1.0/start:0x850000000
158 /sys/bus/cxl/devices/port1/decoder1.0/target_list:0,1,2,3
159 /sys/bus/cxl/devices/port1/decoder1.0/target_type:expander
160 /sys/bus/cxl/devices/root0/decoder0.0/interleave_granularity:256
161 /sys/bus/cxl/devices/root0/decoder0.0/interleave_ways:1
162 /sys/bus/cxl/devices/root0/decoder0.0/size:0x8000000000
163 /sys/bus/cxl/devices/root0/decoder0.0/start:0x850000000
164 /sys/bus/cxl/devices/root0/decoder0.0/target_list:7
165
166 The following changes to the specification are needed:
167
168 * Allow a CXL device to be in an HPA space other than the host's address space.
169
170 * Allow the platform to use implementation-specific address translation when
171 crossing memory domains on the CXL.mem path between the host and the device.
172
173 * Define a PRM handler method for converting device addresses to SPAs.
174
175 * Specify that the platform shall provide the PRM handler method to the
176 Operating System to detect Normalized addressing and for determining Endpoint
177 SPA ranges and interleaving configurations.
178
179 * Add reference to:
180
181 | Platform Runtime Mechanism Specification, Version 1.1 – November 2020
182 | https://uefi.org/sites/default/files/resources/PRM_Platform_Runtime_Mechanism_1_1_release_candidate.pdf
183
184 Benefits of the Change
185 ----------------------
186
187 Without the change, the Operating System may be unable to determine the memory
188 region and Root Decoder for an Endpoint and its corresponding HDM decoder.
189 Region creation would fail. Platforms with a different interconnect architecture
190 would fail to set up and use CXL.
191
192 References
193 ----------
194
195 .. [#cxl-spec-3.2] Compute Express Link Specification, Revision 3.2, Version 1.0,
196 https://www.computeexpresslink.org/
197
198 .. [#amd-ppr-58088] AMD Family 1Ah Models 00h–0Fh and Models 10h–1Fh,
199 ACPI v6.5 Porting Guide, Publication # 58088,
200 https://www.amd.com/en/search/documentation/hub.html
201
202 .. [#prm-spec] Platform Runtime Mechanism, Version: 1.1,
203 https://uefi.org/sites/default/files/resources/PRM_Platform_Runtime_Mechanism_1_1_release_candidate.pdf
204
205 Detailed Description of the Change
206 ----------------------------------
207
208 The following describes the necessary changes to the *CXL 3.2 specification*
209 [#cxl-spec-3.2]_:
210
211 Add the following reference to the table:
212
213 Table 1-2. Reference Documents
214
215 +----------------------------+-------------------+---------------------------+
216 | Document | Chapter Reference | Document No./Location |
217 +============================+===================+===========================+
218 | Platform Runtime Mechanism | Chapter 8, 9 | https://www.uefi.org/acpi |
219 | Version: 1.1 | | |
220 +----------------------------+-------------------+---------------------------+
221
222 Add the following paragraphs to the end of the section:
223
224 **8.2.4.20 CXL HDM Decoder Capability Structure**
225
226 "A device may use an HPA space that is not common to other components of the
227 host domain. The platform is responsible for address translation when crossing
228 HPA spaces. The Operating System must determine the interleaving configuration
229 and perform address translation to the HPA ranges of the HDM decoders as needed.
230 The translation mechanism is host-specific and implementation dependent.
231
232 The platform indicates support of independent HPA spaces and the need for
233 address translation by providing a Platform Runtime Mechanism (PRM) handler. The
234 OS shall use that handler to perform the necessary translations from the DPA
235 space to the HPA space. The handler is defined in Section 9.18.4 *PRM Handler
236 for CXL DPA to System Physical Address Translation*."
237
238 Add the following section and sub-section including tables:
239
240 **9.18.4 PRM Handler for CXL DPA to System Physical Address Translation**
241
242 "A platform may be configured to use 'Normalized addresses'. Host physical
243 address (HPA) spaces are component-specific and differ from system physical
244 addresses (SPAs). The endpoint has its own physical address space. All requests
245 presented to the device already use Device Physical Addresses (DPAs). The CXL
246 endpoint decoders have interleaving disabled (1-way interleaving) and the device
247 does not perform HPA decoding to determine a DPA.
248
249 The platform provides a PRM handler for CXL DPA to System Physical Address
250 Translation. The PRM handler translates a Device Physical Address (DPA) to a
251 System Physical Address (SPA) for a specified CXL endpoint. In the address space
252 of the host, SPA and HPA are equivalent, and the OS shall use this handler to
253 determine the HPA that corresponds to a device address, for example when
254 configuring HDM decoders on platforms with Normalized addressing. The GUID and
255 the parameter buffer format of the handler are specified in section 9.18.4.1. If
256 the OS identifies the PRM handler, the platform supports Normalized addressing
257 and the OS must perform DPA address translation as needed."
258
259 **9.18.4.1 PRM Handler Invocation**
260
261 "The OS calls the PRM handler for CXL DPA to System Physical Address Translation
262 using the direct invocation mechanism. Details of calling a PRM handler are
263 described in the Platform Runtime Mechanism (PRM) specification.
264
265 The PRM handler is identified by the following GUID:
266
267 EE41B397-25D4-452C-AD54-48C6E3480B94
268
269 The caller allocates and prepares a Parameter Buffer, then passes the PRM
270 handler GUID and a pointer to the Parameter Buffer to invoke the handler. The
271 Parameter Buffer is described in Table 9-32."
272
273 **Table 9-32. PRM Parameter Buffer used for CXL DPA to System Physical Address Translation**
274
275 +-------------+-----------+------------------------------------------------------------------------+
276 | Byte Offset | Length in | Description |
277 | | Bytes | |
278 +=============+===========+========================================================================+
279 | 00h | 8 | **CXL Device Physical Address (DPA)**: CXL DPA (e.g., from |
280 | | | CXL Component Event Log) |
281 +-------------+-----------+------------------------------------------------------------------------+
282 | 08h | 4 | **CXL Endpoint SBDF**: |
283 | | | |
284 | | | - Byte 3 - PCIe Segment |
285 | | | - Byte 2 - Bus Number |
286 | | | - Byte 1: |
287 | | | - Device Number Bits[7:3] |
288 | | | - Function Number Bits[2:0] |
289 | | | - Byte 0 - RESERVED (MBZ) |
290 | | | |
291 +-------------+-----------+------------------------------------------------------------------------+
292 | 0Ch | 8 | **Output Buffer**: Virtual Address Pointer to the buffer, |
293 | | | as defined in Table 9-33. |
294 +-------------+-----------+------------------------------------------------------------------------+
295
296 **Table 9-33. PRM Output Buffer used for CXL DPA to System Physical Address Translation**
297
298 +-------------+-----------+------------------------------------------------------------------------+
299 | Byte Offset | Length in | Description |
300 | | Bytes | |
301 +=============+===========+========================================================================+
302 | 00h | 8 | **System Physical Address (SPA)**: The SPA converted |
303 | | | from the CXL DPA. |
304 +-------------+-----------+------------------------------------------------------------------------+