| 1 |
.. SPDX-License-Identifier: GPL-2.0 |
| 2 |
|
| 3 |
================== |
| 4 |
CXL Device Hotplug |
| 5 |
================== |
| 6 |
|
| 7 |
Device hotplug refers to *physical* hotplug of a device (addition or removal |
| 8 |
of a physical device from the machine). |
| 9 |
|
| 10 |
BIOS/EFI software is expected to configure sufficient resources **at boot |
| 11 |
time** to allow hotplugged devices to be configured by software (such as |
| 12 |
proximity domains, HPA regions, and host-bridge configurations). |
| 13 |
|
| 14 |
BIOS/EFI is not expected (**nor suggested**) to configure hotplugged |
| 15 |
devices at hotplug time (i.e. HDM decoders should be left unprogrammed). |
| 16 |
|
| 17 |
This document covers some examples of those resources, but should not |
| 18 |
be considered exhaustive. |
| 19 |
|
| 20 |
Hot-Remove |
| 21 |
========== |
| 22 |
Hot removal of a device typically requires careful removal of software |
| 23 |
constructs (memory regions, associated drivers) which manage these devices. |
| 24 |
|
| 25 |
Hard-removing a CXL.mem device without carefully tearing down driver stacks |
| 26 |
is likely to cause the system to machine-check (or at least SIGBUS if memory |
| 27 |
access is limited to user space). |
| 28 |
|
| 29 |
Memory Device Hot-Add |
| 30 |
===================== |
| 31 |
A device present at boot may be associated with a CXL Fixed Memory Window |
| 32 |
reported in :doc:`CEDT<acpi/cedt>`. That CFMWS may match the size of the |
| 33 |
device, but the construction of the CEDT CFMWS is platform-defined. |
| 34 |
|
| 35 |
Hot-adding a memory device requires this pre-defined, **static** CFMWS to |
| 36 |
have sufficient HPA space to describe that device. |
| 37 |
|
| 38 |
There are a few common scenarios to consider. |
| 39 |
|
| 40 |
Single-Endpoint Memory Device Present at Boot |
| 41 |
--------------------------------------------- |
| 42 |
A device present at boot likely had its capacity reported in the |
| 43 |
:doc:`CEDT<acpi/cedt>`. If a device is removed and a new device hotplugged, |
| 44 |
the capacity of the new device will be limited to the original CFMWS capacity. |
| 45 |
|
| 46 |
Adding capacity larger than the original device will cause memory region |
| 47 |
creation to fail if the region size is greater than the CFMWS size. |
| 48 |
|
| 49 |
The CFMWS is **static** and cannot be adjusted. Platforms which may expect |
| 50 |
different sized devices to be hotplugged must allocate sufficient CFMWS space |
| 51 |
**at boot time** to cover all future expected devices. |
| 52 |
|
| 53 |
Multi-Endpoint Memory Device Present at Boot |
| 54 |
-------------------------------------------- |
| 55 |
Non-switch-based Multi-Endpoint devices are outside the scope of what the |
| 56 |
CXL specification describes, but they are technically possible. We describe |
| 57 |
them here for instructive reasons only - this does not imply Linux support. |
| 58 |
|
| 59 |
A hot-plug capable CXL memory device, such as one which presents multiple |
| 60 |
expanders as a single large-capacity device, should report the **maximum |
| 61 |
possible capacity** for the device at boot. :: |
| 62 |
|
| 63 |
HB0 |
| 64 |
RP0 |
| 65 |
| |
| 66 |
[Multi-Endpoint Memory Device] |
| 67 |
_____|_____ |
| 68 |
| | |
| 69 |
[Endpoint0] [Empty] |
| 70 |
|
| 71 |
|
| 72 |
Limiting the size to the capacity preset at boot will limit hot-add support |
| 73 |
to replacing capacity that was present at boot. |
| 74 |
|
| 75 |
No CXL Device Present at Boot |
| 76 |
----------------------------- |
| 77 |
When no CXL memory device is present on boot, some platforms omit the CFMWS |
| 78 |
in the :doc:`CEDT<acpi/cedt>`. When this occurs, hot-add is not possible. |
| 79 |
|
| 80 |
This describes the base case for any given device not being present at boot. |
| 81 |
If a future possible device is not described in the CEDT at boot, hot-add |
| 82 |
of that device is either limited or not possible. |
| 83 |
|
| 84 |
For a platform to support hot-add of a full memory device, it must allocate |
| 85 |
a CEDT CFMWS region with sufficient memory capacity to cover all future |
| 86 |
potentially added capacity (along with any relevant CEDT CHBS entry). |
| 87 |
|
| 88 |
To support memory hotplug directly on the host bridge/root port, or on a switch |
| 89 |
downstream of the host bridge, a platform must construct a CEDT CFMWS at boot |
| 90 |
with sufficient resources to support the max possible (or expected) hotplug |
| 91 |
memory capacity. :: |
| 92 |
|
| 93 |
HB0 HB1 |
| 94 |
RP0 RP1 RP2 |
| 95 |
| | | |
| 96 |
Empty Empty USP |
| 97 |
________|________ |
| 98 |
| | | | |
| 99 |
DSP DSP DSP DSP |
| 100 |
| | | | |
| 101 |
All Empty |
| 102 |
|
| 103 |
For example, a BIOS/EFI may expose an option to configure a CEDT CFMWS with |
| 104 |
a pre-configured amount of memory capacity (per host bridge, or host bridge |
| 105 |
interleave set), even if no device is attached to Root Ports or Downstream |
| 106 |
Ports at boot (as depicted in the figure above). |
| 107 |
|
| 108 |
|
| 109 |
Interleave Sets |
| 110 |
=============== |
| 111 |
|
| 112 |
Host Bridge Interleave |
| 113 |
---------------------- |
| 114 |
Host-bridge interleaved memory regions are defined **statically** in the |
| 115 |
:doc:`CEDT<acpi/cedt>`. To apply cross-host-bridge interleave, a CFMWS entry |
| 116 |
describing that interleave must have been provided **at boot**. Hotplugged |
| 117 |
devices cannot add host-bridge interleave capabilities at hotplug time. |
| 118 |
|
| 119 |
See the :doc:`Flexible CEDT Configuration<example-configurations/flexible>` |
| 120 |
example to see how a platform can provide this kind of flexibility regarding |
| 121 |
hotplugged memory devices. BIOS/EFI software should consider options to |
| 122 |
present flexible CEDT configurations with hotplug support. |
| 123 |
|
| 124 |
HDM Interleave |
| 125 |
-------------- |
| 126 |
Decoder-applied interleave can flexibly handle hotplugged devices, as decoders |
| 127 |
can be re-programmed after hotplug. |
| 128 |
|
| 129 |
To add or remove a device to/from an existing HDM-applied interleaved region, |
| 130 |
that region must be torn down an re-created. |