Debian Codegraph @

Documentation/driver-api/cxl/platform/device-hotplug.rst

raw
1 .. SPDX-License-Identifier: GPL-2.0
2
3 ==================
4 CXL Device Hotplug
5 ==================
6
7 Device hotplug refers to *physical* hotplug of a device (addition or removal
8 of a physical device from the machine).
9
10 BIOS/EFI software is expected to configure sufficient resources **at boot
11 time** to allow hotplugged devices to be configured by software (such as
12 proximity domains, HPA regions, and host-bridge configurations).
13
14 BIOS/EFI is not expected (**nor suggested**) to configure hotplugged
15 devices at hotplug time (i.e. HDM decoders should be left unprogrammed).
16
17 This document covers some examples of those resources, but should not
18 be considered exhaustive.
19
20 Hot-Remove
21 ==========
22 Hot removal of a device typically requires careful removal of software
23 constructs (memory regions, associated drivers) which manage these devices.
24
25 Hard-removing a CXL.mem device without carefully tearing down driver stacks
26 is likely to cause the system to machine-check (or at least SIGBUS if memory
27 access is limited to user space).
28
29 Memory Device Hot-Add
30 =====================
31 A device present at boot may be associated with a CXL Fixed Memory Window
32 reported in :doc:`CEDT<acpi/cedt>`. That CFMWS may match the size of the
33 device, but the construction of the CEDT CFMWS is platform-defined.
34
35 Hot-adding a memory device requires this pre-defined, **static** CFMWS to
36 have sufficient HPA space to describe that device.
37
38 There are a few common scenarios to consider.
39
40 Single-Endpoint Memory Device Present at Boot
41 ---------------------------------------------
42 A device present at boot likely had its capacity reported in the
43 :doc:`CEDT<acpi/cedt>`. If a device is removed and a new device hotplugged,
44 the capacity of the new device will be limited to the original CFMWS capacity.
45
46 Adding capacity larger than the original device will cause memory region
47 creation to fail if the region size is greater than the CFMWS size.
48
49 The CFMWS is **static** and cannot be adjusted. Platforms which may expect
50 different sized devices to be hotplugged must allocate sufficient CFMWS space
51 **at boot time** to cover all future expected devices.
52
53 Multi-Endpoint Memory Device Present at Boot
54 --------------------------------------------
55 Non-switch-based Multi-Endpoint devices are outside the scope of what the
56 CXL specification describes, but they are technically possible. We describe
57 them here for instructive reasons only - this does not imply Linux support.
58
59 A hot-plug capable CXL memory device, such as one which presents multiple
60 expanders as a single large-capacity device, should report the **maximum
61 possible capacity** for the device at boot. ::
62
63 HB0
64 RP0
65 |
66 [Multi-Endpoint Memory Device]
67 _____|_____
68 | |
69 [Endpoint0] [Empty]
70
71
72 Limiting the size to the capacity preset at boot will limit hot-add support
73 to replacing capacity that was present at boot.
74
75 No CXL Device Present at Boot
76 -----------------------------
77 When no CXL memory device is present on boot, some platforms omit the CFMWS
78 in the :doc:`CEDT<acpi/cedt>`. When this occurs, hot-add is not possible.
79
80 This describes the base case for any given device not being present at boot.
81 If a future possible device is not described in the CEDT at boot, hot-add
82 of that device is either limited or not possible.
83
84 For a platform to support hot-add of a full memory device, it must allocate
85 a CEDT CFMWS region with sufficient memory capacity to cover all future
86 potentially added capacity (along with any relevant CEDT CHBS entry).
87
88 To support memory hotplug directly on the host bridge/root port, or on a switch
89 downstream of the host bridge, a platform must construct a CEDT CFMWS at boot
90 with sufficient resources to support the max possible (or expected) hotplug
91 memory capacity. ::
92
93 HB0 HB1
94 RP0 RP1 RP2
95 | | |
96 Empty Empty USP
97 ________|________
98 | | | |
99 DSP DSP DSP DSP
100 | | | |
101 All Empty
102
103 For example, a BIOS/EFI may expose an option to configure a CEDT CFMWS with
104 a pre-configured amount of memory capacity (per host bridge, or host bridge
105 interleave set), even if no device is attached to Root Ports or Downstream
106 Ports at boot (as depicted in the figure above).
107
108
109 Interleave Sets
110 ===============
111
112 Host Bridge Interleave
113 ----------------------
114 Host-bridge interleaved memory regions are defined **statically** in the
115 :doc:`CEDT<acpi/cedt>`. To apply cross-host-bridge interleave, a CFMWS entry
116 describing that interleave must have been provided **at boot**. Hotplugged
117 devices cannot add host-bridge interleave capabilities at hotplug time.
118
119 See the :doc:`Flexible CEDT Configuration<example-configurations/flexible>`
120 example to see how a platform can provide this kind of flexibility regarding
121 hotplugged memory devices. BIOS/EFI software should consider options to
122 present flexible CEDT configurations with hotplug support.
123
124 HDM Interleave
125 --------------
126 Decoder-applied interleave can flexibly handle hotplugged devices, as decoders
127 can be re-programmed after hotplug.
128
129 To add or remove a device to/from an existing HDM-applied interleaved region,
130 that region must be torn down an re-created.