| 1 |
.. SPDX-License-Identifier: GPL-2.0 |
| 2 |
|
| 3 |
===================== |
| 4 |
Devices and Protocols |
| 5 |
===================== |
| 6 |
|
| 7 |
The type of CXL device (Memory, Accelerator, etc) dictates many configuration steps. This section |
| 8 |
covers some basic background on device types and on-device resources used by the platform and OS |
| 9 |
which impact configuration. |
| 10 |
|
| 11 |
Protocols |
| 12 |
========= |
| 13 |
|
| 14 |
There are three core protocols to CXL. For the purpose of this documentation, |
| 15 |
we will only discuss very high level definitions as the specific hardware |
| 16 |
details are largely abstracted away from Linux. See the CXL specification |
| 17 |
for more details. |
| 18 |
|
| 19 |
CXL.io |
| 20 |
------ |
| 21 |
The basic interaction protocol, similar to PCIe configuration mechanisms. |
| 22 |
Typically used for initialization, configuration, and I/O access for anything |
| 23 |
other than memory (CXL.mem) or cache (CXL.cache) operations. |
| 24 |
|
| 25 |
The Linux CXL driver exposes access to .io functionality via the various sysfs |
| 26 |
interfaces and /dev/cxl/ devices (which exposes direct access to device |
| 27 |
mailboxes). |
| 28 |
|
| 29 |
CXL.cache |
| 30 |
--------- |
| 31 |
The mechanism by which a device may coherently access and cache host memory. |
| 32 |
|
| 33 |
Largely transparent to Linux once configured. |
| 34 |
|
| 35 |
CXL.mem |
| 36 |
--------- |
| 37 |
The mechanism by which the CPU may coherently access and cache device memory. |
| 38 |
|
| 39 |
Largely transparent to Linux once configured. |
| 40 |
|
| 41 |
|
| 42 |
Device Types |
| 43 |
============ |
| 44 |
|
| 45 |
Type-1 |
| 46 |
------ |
| 47 |
|
| 48 |
A Type-1 CXL device: |
| 49 |
|
| 50 |
* Supports cxl.io and cxl.cache protocols |
| 51 |
* Implements a fully coherent cache |
| 52 |
* Allows Device-to-Host coherence and Host-to-Device snoops. |
| 53 |
* Does NOT have host-managed device memory (HDM) |
| 54 |
|
| 55 |
Typical examples of type-1 devices is a Smart NIC - which may want to |
| 56 |
directly operate on host-memory (DMA) to store incoming packets. These |
| 57 |
devices largely rely on CPU-attached memory. |
| 58 |
|
| 59 |
Type-2 |
| 60 |
------ |
| 61 |
|
| 62 |
A Type-2 CXL Device: |
| 63 |
|
| 64 |
* Supports cxl.io, cxl.cache, and cxl.mem protocols |
| 65 |
* Optionally implements coherent cache and Host-Managed Device Memory |
| 66 |
* Is typically an accelerator device with high bandwidth memory. |
| 67 |
|
| 68 |
The primary difference between a type-1 and type-2 device is the presence |
| 69 |
of host-managed device memory, which allows the device to operate on a |
| 70 |
local memory bank - while the CPU still has coherent DMA to the same memory. |
| 71 |
|
| 72 |
This allows things like GPUs to expose their memory via DAX devices or file |
| 73 |
descriptors, allows drivers and programs direct access to device memory |
| 74 |
rather than use block-transfer semantics. |
| 75 |
|
| 76 |
Type-3 |
| 77 |
------ |
| 78 |
|
| 79 |
A Type-3 CXL Device |
| 80 |
|
| 81 |
* Supports cxl.io and cxl.mem |
| 82 |
* Implements Host-Managed Device Memory |
| 83 |
* May provide either Volatile or Persistent memory capacity (or both). |
| 84 |
|
| 85 |
A basic example of a type-3 device is a simple memory expander, whose |
| 86 |
local memory capacity is exposed to the CPU for access directly via |
| 87 |
basic coherent DMA. |
| 88 |
|
| 89 |
Switch |
| 90 |
------ |
| 91 |
|
| 92 |
A CXL switch is a device capable of routing any CXL (and by extension, PCIe) |
| 93 |
protocol between an upstream, downstream, or peer devices. Many devices, such |
| 94 |
as Multi-Logical Devices, imply the presence of switching in some manner. |
| 95 |
|
| 96 |
Logical Devices and Heads |
| 97 |
------------------------- |
| 98 |
|
| 99 |
A CXL device may present one or more "Logical Devices" to one or more hosts |
| 100 |
(via physical "Heads"). |
| 101 |
|
| 102 |
A Single-Logical Device (SLD) is a device which presents a single device to |
| 103 |
one or more heads. |
| 104 |
|
| 105 |
A Multi-Logical Device (MLD) is a device which may present multiple devices |
| 106 |
to one or more upstream devices. |
| 107 |
|
| 108 |
A Single-Headed Device exposes only a single physical connection. |
| 109 |
|
| 110 |
A Multi-Headed Device exposes multiple physical connections. |
| 111 |
|
| 112 |
MHSLD |
| 113 |
~~~~~ |
| 114 |
A Multi-Headed Single-Logical Device (MHSLD) exposes a single logical |
| 115 |
device to multiple heads which may be connected to one or more discrete |
| 116 |
hosts. An example of this would be a simple memory-pool which may be |
| 117 |
statically configured (prior to boot) to expose portions of its memory |
| 118 |
to Linux via :doc:`CEDT <../platform/acpi/cedt>`. |
| 119 |
|
| 120 |
MHMLD |
| 121 |
~~~~~ |
| 122 |
A Multi-Headed Multi-Logical Device (MHMLD) exposes multiple logical |
| 123 |
devices to multiple heads which may be connected to one or more discrete |
| 124 |
hosts. An example of this would be a Dynamic Capacity Device or which |
| 125 |
may be configured at runtime to expose portions of its memory to Linux. |
| 126 |
|
| 127 |
Example Devices |
| 128 |
=============== |
| 129 |
|
| 130 |
Memory Expander |
| 131 |
--------------- |
| 132 |
The simplest form of Type-3 device is a memory expander. A memory expander |
| 133 |
exposes Host-Managed Device Memory (HDM) to Linux. This memory may be |
| 134 |
Volatile or Non-Volatile (Persistent). |
| 135 |
|
| 136 |
Memory Expanders will typically be considered a form of Single-Headed, |
| 137 |
Single-Logical Device - as its form factor will typically be an add-in-card |
| 138 |
(AIC) or some other similar form-factor. |
| 139 |
|
| 140 |
The Linux CXL driver provides support for static or dynamic configuration of |
| 141 |
basic memory expanders. The platform may program decoders prior to OS init |
| 142 |
(e.g. auto-decoders), or the user may program the fabric if the platform |
| 143 |
defers these operations to the OS. |
| 144 |
|
| 145 |
Multiple Memory Expanders may be added to an external chassis and exposed to |
| 146 |
a host via a head attached to a CXL switch. This is a "memory pool", and |
| 147 |
would be considered an MHSLD or MHMLD depending on the management capabilities |
| 148 |
provided by the switch platform. |
| 149 |
|
| 150 |
As of v6.14, Linux does not provide a formalized interface to manage non-DCD |
| 151 |
MHSLD or MHMLD devices. |
| 152 |
|
| 153 |
Dynamic Capacity Device (DCD) |
| 154 |
----------------------------- |
| 155 |
|
| 156 |
A Dynamic Capacity Device is a Type-3 device which provides dynamic management |
| 157 |
of memory capacity. The basic premise of a DCD to provide an allocator-like |
| 158 |
interface for physical memory capacity to a "Fabric Manager" (an external, |
| 159 |
privileged host with privileges to change configurations for other hosts). |
| 160 |
|
| 161 |
A DCD manages "Memory Extents", which may be volatile or persistent. Extents |
| 162 |
may also be exclusive to a single host or shared across multiple hosts. |
| 163 |
|
| 164 |
As of v6.14, Linux does not provide a formalized interface to manage DCD |
| 165 |
devices, however there is active work on LKML targeting future release. |