| 1 |
.. SPDX-License-Identifier: GPL-2.0 |
| 2 |
|
| 3 |
ACPI PRM CXL Address Translation |
| 4 |
================================ |
| 5 |
|
| 6 |
Document |
| 7 |
-------- |
| 8 |
|
| 9 |
CXL Revision 3.2, Version 1.0 |
| 10 |
|
| 11 |
License |
| 12 |
------- |
| 13 |
|
| 14 |
SPDX-License Identifier: CC-BY-4.0 |
| 15 |
|
| 16 |
Creator/Contributors |
| 17 |
-------------------- |
| 18 |
|
| 19 |
- Robert Richter, AMD et al. |
| 20 |
|
| 21 |
Summary of the Change |
| 22 |
--------------------- |
| 23 |
|
| 24 |
The CXL Fixed Memory Window Structures (CFMWS) describe zero or more Host |
| 25 |
Physical Address (HPA) windows associated with one or more CXL Host Bridges. |
| 26 |
Each HPA range of a CXL Host Bridge is represented by a CFMWS entry. An HPA |
| 27 |
range may include addresses currently assigned to CXL.mem devices, or an OS may |
| 28 |
assign ranges from an address window to a device. |
| 29 |
|
| 30 |
Host-managed Device Memory is Device-attached memory that is mapped to system |
| 31 |
coherent address space and accessible to the Host using standard write-back |
| 32 |
semantics. The managed address range is configured in the CXL HDM Decoder |
| 33 |
registers of the device. An HDM Decoder in a device is responsible for |
| 34 |
converting HPA into DPA by stripping off specific address bits. |
| 35 |
|
| 36 |
CXL devices and CXL bridges use the same HPA space. It is common across all |
| 37 |
components that belong to the same host domain. The view of the address region |
| 38 |
must be consistent on the CXL.mem path between the Host and the Device. |
| 39 |
|
| 40 |
This is described in the *CXL 3.2 specification* (Table 1-1, 3.3.1, |
| 41 |
8.2.4.20, 9.13.1, 9.18.1.3). [#cxl-spec-3.2]_ |
| 42 |
|
| 43 |
Depending on the interconnect architecture of the platform, components attached |
| 44 |
to a host may not share the same host physical address space. Those platforms |
| 45 |
need address translation to convert an HPA between the host and the attached |
| 46 |
component, such as a CXL device. The translation mechanism is host-specific and |
| 47 |
implementation dependent. |
| 48 |
|
| 49 |
For example, x86 AMD platforms use a Data Fabric that manages access to physical |
| 50 |
memory. Devices have their own memory space and can be configured to use |
| 51 |
'Normalized addresses' different from System Physical Addresses (SPA). Address |
| 52 |
translation is then needed. For details, see |
| 53 |
:doc:`x86 AMD Address Translation </admin-guide/RAS/address-translation>`. |
| 54 |
|
| 55 |
Those AMD platforms provide PRM [#prm-spec]_ handlers in firmware to perform |
| 56 |
various types of address translation, including for CXL endpoints. AMD Zen5 |
| 57 |
systems implement the ACPI PRM CXL Address Translation firmware call. The ACPI |
| 58 |
PRM handler has a specific GUID to uniquely identify platforms with support for |
| 59 |
Normalized addressing. This is documented in the *ACPI v6.5 Porting Guide* |
| 60 |
(Address Translation - CXL DPA to System Physical Address). [#amd-ppr-58088]_ |
| 61 |
|
| 62 |
When in Normalized address mode, HDM decoder address ranges must be configured |
| 63 |
and handled differently. Hardware addresses used in the HDM decoder |
| 64 |
configurations of an endpoint are not SPA and need to be translated from the |
| 65 |
address range of the endpoint to that of the CXL host bridge. This is especially |
| 66 |
important for finding an endpoint's associated CXL Host Bridge and HPA window |
| 67 |
described in the CFMWS. Additionally, the interleave decoding is done by the |
| 68 |
Data Fabric and the endpoint does not perform decoding when converting HPA to |
| 69 |
DPA. Instead, interleaving is switched off for the endpoint (1-way). Finally, |
| 70 |
address translation might also be needed to inspect the endpoint's hardware |
| 71 |
addresses, such as during profiling, tracing, or error handling. |
| 72 |
|
| 73 |
For example, with Normalized addressing the HDM decoders could look as follows:: |
| 74 |
|
| 75 |
------------------------------- |
| 76 |
| Root Decoder (CFMWS) | |
| 77 |
| SPA Range: 0x850000000 | |
| 78 |
| Size: 0x8000000000 (512 GB) | |
| 79 |
| Interleave Ways: 1 | |
| 80 |
------------------------------- |
| 81 |
| |
| 82 |
v |
| 83 |
------------------------------- |
| 84 |
| Host Bridge Decoder (HDM) | |
| 85 |
| SPA Range: 0x850000000 | |
| 86 |
| Size: 0x8000000000 (512 GB) | |
| 87 |
| Interleave Ways: 4 | |
| 88 |
| Targets: endpoint5,8,11,13 | |
| 89 |
| Granularity: 256 | |
| 90 |
------------------------------- |
| 91 |
| |
| 92 |
-----------------------------+------------------------------ |
| 93 |
| | | | |
| 94 |
v v v v |
| 95 |
------------------- ------------------- ------------------- ------------------- |
| 96 |
| endpoint5 | | endpoint8 | | endpoint11 | | endpoint13 | |
| 97 |
| decoder5.0 | | decoder8.0 | | decoder11.0 | | decoder13.0 | |
| 98 |
| PCIe: | | PCIe: | | PCIe: | | PCIe: | |
| 99 |
| 0000:e2:00.0 | | 0000:e3:00.0 | | 0000:e4:00.0 | | 0000:e1:00.0 | |
| 100 |
| DPA: | | DPA: | | DPA: | | DPA: | |
| 101 |
| Start: 0x0 | | Start: 0x0 | | Start: 0x0 | | Start: 0x0 | |
| 102 |
| Size: | | Size: | | Size: | | Size: | |
| 103 |
| 0x2000000000 | | 0x2000000000 | | 0x2000000000 | | 0x2000000000 | |
| 104 |
| (128 GB) | | (128 GB) | | (128 GB) | | (128 GB) | |
| 105 |
| Interleaving: | | Interleaving: | | Interleaving: | | Interleaving: | |
| 106 |
| Ways: 1 | | Ways: 1 | | Ways: 1 | | Ways: 1 | |
| 107 |
| Gran: 256 | | Gran: 256 | | Gran: 256 | | Gran: 256 | |
| 108 |
------------------- ------------------- ------------------- ------------------- |
| 109 |
| | | | |
| 110 |
v v v v |
| 111 |
DPA DPA DPA DPA |
| 112 |
|
| 113 |
This shows the representation in sysfs: |
| 114 |
|
| 115 |
.. code-block:: none |
| 116 |
|
| 117 |
/sys/bus/cxl/devices/endpoint5/decoder5.0/interleave_granularity:256 |
| 118 |
/sys/bus/cxl/devices/endpoint5/decoder5.0/interleave_ways:1 |
| 119 |
/sys/bus/cxl/devices/endpoint5/decoder5.0/size:0x2000000000 |
| 120 |
/sys/bus/cxl/devices/endpoint5/decoder5.0/start:0x0 |
| 121 |
/sys/bus/cxl/devices/endpoint8/decoder8.0/interleave_granularity:256 |
| 122 |
/sys/bus/cxl/devices/endpoint8/decoder8.0/interleave_ways:1 |
| 123 |
/sys/bus/cxl/devices/endpoint8/decoder8.0/size:0x2000000000 |
| 124 |
/sys/bus/cxl/devices/endpoint8/decoder8.0/start:0x0 |
| 125 |
/sys/bus/cxl/devices/endpoint11/decoder11.0/interleave_granularity:256 |
| 126 |
/sys/bus/cxl/devices/endpoint11/decoder11.0/interleave_ways:1 |
| 127 |
/sys/bus/cxl/devices/endpoint11/decoder11.0/size:0x2000000000 |
| 128 |
/sys/bus/cxl/devices/endpoint11/decoder11.0/start:0x0 |
| 129 |
/sys/bus/cxl/devices/endpoint13/decoder13.0/interleave_granularity:256 |
| 130 |
/sys/bus/cxl/devices/endpoint13/decoder13.0/interleave_ways:1 |
| 131 |
/sys/bus/cxl/devices/endpoint13/decoder13.0/size:0x2000000000 |
| 132 |
/sys/bus/cxl/devices/endpoint13/decoder13.0/start:0x0 |
| 133 |
|
| 134 |
Note that the endpoint interleaving configurations use direct mapping (1-way). |
| 135 |
|
| 136 |
With PRM calls, the kernel can determine the following mappings: |
| 137 |
|
| 138 |
.. code-block:: none |
| 139 |
|
| 140 |
cxl decoder5.0: address mapping found for 0000:e2:00.0 (hpa -> spa): |
| 141 |
0x0+0x2000000000 -> 0x850000000+0x8000000000 ways:4 granularity:256 |
| 142 |
cxl decoder8.0: address mapping found for 0000:e3:00.0 (hpa -> spa): |
| 143 |
0x0+0x2000000000 -> 0x850000000+0x8000000000 ways:4 granularity:256 |
| 144 |
cxl decoder11.0: address mapping found for 0000:e4:00.0 (hpa -> spa): |
| 145 |
0x0+0x2000000000 -> 0x850000000+0x8000000000 ways:4 granularity:256 |
| 146 |
cxl decoder13.0: address mapping found for 0000:e1:00.0 (hpa -> spa): |
| 147 |
0x0+0x2000000000 -> 0x850000000+0x8000000000 ways:4 granularity:256 |
| 148 |
|
| 149 |
The corresponding CXL host bridge (HDM) decoders and root decoder (CFMWS) match |
| 150 |
the calculated endpoint mappings shown: |
| 151 |
|
| 152 |
.. code-block:: none |
| 153 |
|
| 154 |
/sys/bus/cxl/devices/port1/decoder1.0/interleave_granularity:256 |
| 155 |
/sys/bus/cxl/devices/port1/decoder1.0/interleave_ways:4 |
| 156 |
/sys/bus/cxl/devices/port1/decoder1.0/size:0x8000000000 |
| 157 |
/sys/bus/cxl/devices/port1/decoder1.0/start:0x850000000 |
| 158 |
/sys/bus/cxl/devices/port1/decoder1.0/target_list:0,1,2,3 |
| 159 |
/sys/bus/cxl/devices/port1/decoder1.0/target_type:expander |
| 160 |
/sys/bus/cxl/devices/root0/decoder0.0/interleave_granularity:256 |
| 161 |
/sys/bus/cxl/devices/root0/decoder0.0/interleave_ways:1 |
| 162 |
/sys/bus/cxl/devices/root0/decoder0.0/size:0x8000000000 |
| 163 |
/sys/bus/cxl/devices/root0/decoder0.0/start:0x850000000 |
| 164 |
/sys/bus/cxl/devices/root0/decoder0.0/target_list:7 |
| 165 |
|
| 166 |
The following changes to the specification are needed: |
| 167 |
|
| 168 |
* Allow a CXL device to be in an HPA space other than the host's address space. |
| 169 |
|
| 170 |
* Allow the platform to use implementation-specific address translation when |
| 171 |
crossing memory domains on the CXL.mem path between the host and the device. |
| 172 |
|
| 173 |
* Define a PRM handler method for converting device addresses to SPAs. |
| 174 |
|
| 175 |
* Specify that the platform shall provide the PRM handler method to the |
| 176 |
Operating System to detect Normalized addressing and for determining Endpoint |
| 177 |
SPA ranges and interleaving configurations. |
| 178 |
|
| 179 |
* Add reference to: |
| 180 |
|
| 181 |
| Platform Runtime Mechanism Specification, Version 1.1 – November 2020 |
| 182 |
| https://uefi.org/sites/default/files/resources/PRM_Platform_Runtime_Mechanism_1_1_release_candidate.pdf |
| 183 |
|
| 184 |
Benefits of the Change |
| 185 |
---------------------- |
| 186 |
|
| 187 |
Without the change, the Operating System may be unable to determine the memory |
| 188 |
region and Root Decoder for an Endpoint and its corresponding HDM decoder. |
| 189 |
Region creation would fail. Platforms with a different interconnect architecture |
| 190 |
would fail to set up and use CXL. |
| 191 |
|
| 192 |
References |
| 193 |
---------- |
| 194 |
|
| 195 |
.. [#cxl-spec-3.2] Compute Express Link Specification, Revision 3.2, Version 1.0, |
| 196 |
https://www.computeexpresslink.org/ |
| 197 |
|
| 198 |
.. [#amd-ppr-58088] AMD Family 1Ah Models 00h–0Fh and Models 10h–1Fh, |
| 199 |
ACPI v6.5 Porting Guide, Publication # 58088, |
| 200 |
https://www.amd.com/en/search/documentation/hub.html |
| 201 |
|
| 202 |
.. [#prm-spec] Platform Runtime Mechanism, Version: 1.1, |
| 203 |
https://uefi.org/sites/default/files/resources/PRM_Platform_Runtime_Mechanism_1_1_release_candidate.pdf |
| 204 |
|
| 205 |
Detailed Description of the Change |
| 206 |
---------------------------------- |
| 207 |
|
| 208 |
The following describes the necessary changes to the *CXL 3.2 specification* |
| 209 |
[#cxl-spec-3.2]_: |
| 210 |
|
| 211 |
Add the following reference to the table: |
| 212 |
|
| 213 |
Table 1-2. Reference Documents |
| 214 |
|
| 215 |
+----------------------------+-------------------+---------------------------+ |
| 216 |
| Document | Chapter Reference | Document No./Location | |
| 217 |
+============================+===================+===========================+ |
| 218 |
| Platform Runtime Mechanism | Chapter 8, 9 | https://www.uefi.org/acpi | |
| 219 |
| Version: 1.1 | | | |
| 220 |
+----------------------------+-------------------+---------------------------+ |
| 221 |
|
| 222 |
Add the following paragraphs to the end of the section: |
| 223 |
|
| 224 |
**8.2.4.20 CXL HDM Decoder Capability Structure** |
| 225 |
|
| 226 |
"A device may use an HPA space that is not common to other components of the |
| 227 |
host domain. The platform is responsible for address translation when crossing |
| 228 |
HPA spaces. The Operating System must determine the interleaving configuration |
| 229 |
and perform address translation to the HPA ranges of the HDM decoders as needed. |
| 230 |
The translation mechanism is host-specific and implementation dependent. |
| 231 |
|
| 232 |
The platform indicates support of independent HPA spaces and the need for |
| 233 |
address translation by providing a Platform Runtime Mechanism (PRM) handler. The |
| 234 |
OS shall use that handler to perform the necessary translations from the DPA |
| 235 |
space to the HPA space. The handler is defined in Section 9.18.4 *PRM Handler |
| 236 |
for CXL DPA to System Physical Address Translation*." |
| 237 |
|
| 238 |
Add the following section and sub-section including tables: |
| 239 |
|
| 240 |
**9.18.4 PRM Handler for CXL DPA to System Physical Address Translation** |
| 241 |
|
| 242 |
"A platform may be configured to use 'Normalized addresses'. Host physical |
| 243 |
address (HPA) spaces are component-specific and differ from system physical |
| 244 |
addresses (SPAs). The endpoint has its own physical address space. All requests |
| 245 |
presented to the device already use Device Physical Addresses (DPAs). The CXL |
| 246 |
endpoint decoders have interleaving disabled (1-way interleaving) and the device |
| 247 |
does not perform HPA decoding to determine a DPA. |
| 248 |
|
| 249 |
The platform provides a PRM handler for CXL DPA to System Physical Address |
| 250 |
Translation. The PRM handler translates a Device Physical Address (DPA) to a |
| 251 |
System Physical Address (SPA) for a specified CXL endpoint. In the address space |
| 252 |
of the host, SPA and HPA are equivalent, and the OS shall use this handler to |
| 253 |
determine the HPA that corresponds to a device address, for example when |
| 254 |
configuring HDM decoders on platforms with Normalized addressing. The GUID and |
| 255 |
the parameter buffer format of the handler are specified in section 9.18.4.1. If |
| 256 |
the OS identifies the PRM handler, the platform supports Normalized addressing |
| 257 |
and the OS must perform DPA address translation as needed." |
| 258 |
|
| 259 |
**9.18.4.1 PRM Handler Invocation** |
| 260 |
|
| 261 |
"The OS calls the PRM handler for CXL DPA to System Physical Address Translation |
| 262 |
using the direct invocation mechanism. Details of calling a PRM handler are |
| 263 |
described in the Platform Runtime Mechanism (PRM) specification. |
| 264 |
|
| 265 |
The PRM handler is identified by the following GUID: |
| 266 |
|
| 267 |
EE41B397-25D4-452C-AD54-48C6E3480B94 |
| 268 |
|
| 269 |
The caller allocates and prepares a Parameter Buffer, then passes the PRM |
| 270 |
handler GUID and a pointer to the Parameter Buffer to invoke the handler. The |
| 271 |
Parameter Buffer is described in Table 9-32." |
| 272 |
|
| 273 |
**Table 9-32. PRM Parameter Buffer used for CXL DPA to System Physical Address Translation** |
| 274 |
|
| 275 |
+-------------+-----------+------------------------------------------------------------------------+ |
| 276 |
| Byte Offset | Length in | Description | |
| 277 |
| | Bytes | | |
| 278 |
+=============+===========+========================================================================+ |
| 279 |
| 00h | 8 | **CXL Device Physical Address (DPA)**: CXL DPA (e.g., from | |
| 280 |
| | | CXL Component Event Log) | |
| 281 |
+-------------+-----------+------------------------------------------------------------------------+ |
| 282 |
| 08h | 4 | **CXL Endpoint SBDF**: | |
| 283 |
| | | | |
| 284 |
| | | - Byte 3 - PCIe Segment | |
| 285 |
| | | - Byte 2 - Bus Number | |
| 286 |
| | | - Byte 1: | |
| 287 |
| | | - Device Number Bits[7:3] | |
| 288 |
| | | - Function Number Bits[2:0] | |
| 289 |
| | | - Byte 0 - RESERVED (MBZ) | |
| 290 |
| | | | |
| 291 |
+-------------+-----------+------------------------------------------------------------------------+ |
| 292 |
| 0Ch | 8 | **Output Buffer**: Virtual Address Pointer to the buffer, | |
| 293 |
| | | as defined in Table 9-33. | |
| 294 |
+-------------+-----------+------------------------------------------------------------------------+ |
| 295 |
|
| 296 |
**Table 9-33. PRM Output Buffer used for CXL DPA to System Physical Address Translation** |
| 297 |
|
| 298 |
+-------------+-----------+------------------------------------------------------------------------+ |
| 299 |
| Byte Offset | Length in | Description | |
| 300 |
| | Bytes | | |
| 301 |
+=============+===========+========================================================================+ |
| 302 |
| 00h | 8 | **System Physical Address (SPA)**: The SPA converted | |
| 303 |
| | | from the CXL DPA. | |
| 304 |
+-------------+-----------+------------------------------------------------------------------------+ |