Debian Codegraph @

Documentation/driver-api/cxl/devices/device-types.rst

raw
1 .. SPDX-License-Identifier: GPL-2.0
2
3 =====================
4 Devices and Protocols
5 =====================
6
7 The type of CXL device (Memory, Accelerator, etc) dictates many configuration steps. This section
8 covers some basic background on device types and on-device resources used by the platform and OS
9 which impact configuration.
10
11 Protocols
12 =========
13
14 There are three core protocols to CXL. For the purpose of this documentation,
15 we will only discuss very high level definitions as the specific hardware
16 details are largely abstracted away from Linux. See the CXL specification
17 for more details.
18
19 CXL.io
20 ------
21 The basic interaction protocol, similar to PCIe configuration mechanisms.
22 Typically used for initialization, configuration, and I/O access for anything
23 other than memory (CXL.mem) or cache (CXL.cache) operations.
24
25 The Linux CXL driver exposes access to .io functionality via the various sysfs
26 interfaces and /dev/cxl/ devices (which exposes direct access to device
27 mailboxes).
28
29 CXL.cache
30 ---------
31 The mechanism by which a device may coherently access and cache host memory.
32
33 Largely transparent to Linux once configured.
34
35 CXL.mem
36 ---------
37 The mechanism by which the CPU may coherently access and cache device memory.
38
39 Largely transparent to Linux once configured.
40
41
42 Device Types
43 ============
44
45 Type-1
46 ------
47
48 A Type-1 CXL device:
49
50 * Supports cxl.io and cxl.cache protocols
51 * Implements a fully coherent cache
52 * Allows Device-to-Host coherence and Host-to-Device snoops.
53 * Does NOT have host-managed device memory (HDM)
54
55 Typical examples of type-1 devices is a Smart NIC - which may want to
56 directly operate on host-memory (DMA) to store incoming packets. These
57 devices largely rely on CPU-attached memory.
58
59 Type-2
60 ------
61
62 A Type-2 CXL Device:
63
64 * Supports cxl.io, cxl.cache, and cxl.mem protocols
65 * Optionally implements coherent cache and Host-Managed Device Memory
66 * Is typically an accelerator device with high bandwidth memory.
67
68 The primary difference between a type-1 and type-2 device is the presence
69 of host-managed device memory, which allows the device to operate on a
70 local memory bank - while the CPU still has coherent DMA to the same memory.
71
72 This allows things like GPUs to expose their memory via DAX devices or file
73 descriptors, allows drivers and programs direct access to device memory
74 rather than use block-transfer semantics.
75
76 Type-3
77 ------
78
79 A Type-3 CXL Device
80
81 * Supports cxl.io and cxl.mem
82 * Implements Host-Managed Device Memory
83 * May provide either Volatile or Persistent memory capacity (or both).
84
85 A basic example of a type-3 device is a simple memory expander, whose
86 local memory capacity is exposed to the CPU for access directly via
87 basic coherent DMA.
88
89 Switch
90 ------
91
92 A CXL switch is a device capable of routing any CXL (and by extension, PCIe)
93 protocol between an upstream, downstream, or peer devices. Many devices, such
94 as Multi-Logical Devices, imply the presence of switching in some manner.
95
96 Logical Devices and Heads
97 -------------------------
98
99 A CXL device may present one or more "Logical Devices" to one or more hosts
100 (via physical "Heads").
101
102 A Single-Logical Device (SLD) is a device which presents a single device to
103 one or more heads.
104
105 A Multi-Logical Device (MLD) is a device which may present multiple devices
106 to one or more upstream devices.
107
108 A Single-Headed Device exposes only a single physical connection.
109
110 A Multi-Headed Device exposes multiple physical connections.
111
112 MHSLD
113 ~~~~~
114 A Multi-Headed Single-Logical Device (MHSLD) exposes a single logical
115 device to multiple heads which may be connected to one or more discrete
116 hosts. An example of this would be a simple memory-pool which may be
117 statically configured (prior to boot) to expose portions of its memory
118 to Linux via :doc:`CEDT <../platform/acpi/cedt>`.
119
120 MHMLD
121 ~~~~~
122 A Multi-Headed Multi-Logical Device (MHMLD) exposes multiple logical
123 devices to multiple heads which may be connected to one or more discrete
124 hosts. An example of this would be a Dynamic Capacity Device or which
125 may be configured at runtime to expose portions of its memory to Linux.
126
127 Example Devices
128 ===============
129
130 Memory Expander
131 ---------------
132 The simplest form of Type-3 device is a memory expander. A memory expander
133 exposes Host-Managed Device Memory (HDM) to Linux. This memory may be
134 Volatile or Non-Volatile (Persistent).
135
136 Memory Expanders will typically be considered a form of Single-Headed,
137 Single-Logical Device - as its form factor will typically be an add-in-card
138 (AIC) or some other similar form-factor.
139
140 The Linux CXL driver provides support for static or dynamic configuration of
141 basic memory expanders. The platform may program decoders prior to OS init
142 (e.g. auto-decoders), or the user may program the fabric if the platform
143 defers these operations to the OS.
144
145 Multiple Memory Expanders may be added to an external chassis and exposed to
146 a host via a head attached to a CXL switch. This is a "memory pool", and
147 would be considered an MHSLD or MHMLD depending on the management capabilities
148 provided by the switch platform.
149
150 As of v6.14, Linux does not provide a formalized interface to manage non-DCD
151 MHSLD or MHMLD devices.
152
153 Dynamic Capacity Device (DCD)
154 -----------------------------
155
156 A Dynamic Capacity Device is a Type-3 device which provides dynamic management
157 of memory capacity. The basic premise of a DCD to provide an allocator-like
158 interface for physical memory capacity to a "Fabric Manager" (an external,
159 privileged host with privileges to change configurations for other hosts).
160
161 A DCD manages "Memory Extents", which may be volatile or persistent. Extents
162 may also be exclusive to a single host or shared across multiple hosts.
163
164 As of v6.14, Linux does not provide a formalized interface to manage DCD
165 devices, however there is active work on LKML targeting future release.