| 1 |
.. SPDX-License-Identifier: GPL-2.0 |
| 2 |
|
| 3 |
=============== |
| 4 |
fwctl subsystem |
| 5 |
=============== |
| 6 |
|
| 7 |
:Author: Jason Gunthorpe |
| 8 |
|
| 9 |
Overview |
| 10 |
======== |
| 11 |
|
| 12 |
Modern devices contain extensive amounts of FW, and in many cases, are largely |
| 13 |
software-defined pieces of hardware. The evolution of this approach is largely a |
| 14 |
reaction to Moore's Law where a chip tape out is now highly expensive, and the |
| 15 |
chip design is extremely large. Replacing fixed HW logic with a flexible and |
| 16 |
tightly coupled FW/HW combination is an effective risk mitigation against chip |
| 17 |
respin. Problems in the HW design can be counteracted in device FW. This is |
| 18 |
especially true for devices which present a stable and backwards compatible |
| 19 |
interface to the operating system driver (such as NVMe). |
| 20 |
|
| 21 |
The FW layer in devices has grown to incredible size and devices frequently |
| 22 |
integrate clusters of fast processors to run it. For example, mlx5 devices have |
| 23 |
over 30MB of FW code, and big configurations operate with over 1GB of FW managed |
| 24 |
runtime state. |
| 25 |
|
| 26 |
The availability of such a flexible layer has created quite a variety in the |
| 27 |
industry where single pieces of silicon are now configurable software-defined |
| 28 |
devices and can operate in substantially different ways depending on the need. |
| 29 |
Further, we often see cases where specific sites wish to operate devices in ways |
| 30 |
that are highly specialized and require applications that have been tailored to |
| 31 |
their unique configuration. |
| 32 |
|
| 33 |
Further, devices have become multi-functional and integrated to the point they |
| 34 |
no longer fit neatly into the kernel's division of subsystems. Modern |
| 35 |
multi-functional devices have drivers, such as bnxt/ice/mlx5/pds, that span many |
| 36 |
subsystems while sharing the underlying hardware using the auxiliary device |
| 37 |
system. |
| 38 |
|
| 39 |
All together this creates a challenge for the operating system, where devices |
| 40 |
have an expansive FW environment that needs robust device-specific debugging |
| 41 |
support, and FW-driven functionality that is not well suited to “generic” |
| 42 |
interfaces. fwctl seeks to allow access to the full device functionality from |
| 43 |
user space in the areas of debuggability, management, and first-boot/nth-boot |
| 44 |
provisioning. |
| 45 |
|
| 46 |
fwctl is aimed at the common device design pattern where the OS and FW |
| 47 |
communicate via an RPC message layer constructed with a queue or mailbox scheme. |
| 48 |
In this case the driver will typically have some layer to deliver RPC messages |
| 49 |
and collect RPC responses from device FW. The in-kernel subsystem drivers that |
| 50 |
operate the device for its primary purposes will use these RPCs to build their |
| 51 |
drivers, but devices also usually have a set of ancillary RPCs that don't really |
| 52 |
fit into any specific subsystem. For example, a HW RAID controller is primarily |
| 53 |
operated by the block layer but also comes with a set of RPCs to administer the |
| 54 |
construction of drives within the HW RAID. |
| 55 |
|
| 56 |
In the past when devices were more single function, individual subsystems would |
| 57 |
grow different approaches to solving some of these common problems. For instance, |
| 58 |
monitoring device health, manipulating its FLASH, debugging the FW, |
| 59 |
provisioning, all have various unique interfaces across the kernel. |
| 60 |
|
| 61 |
fwctl's purpose is to define a common set of limited rules, described below, |
| 62 |
that allow user space to securely construct and execute RPCs inside device FW. |
| 63 |
The rules serve as an agreement between the operating system and FW on how to |
| 64 |
correctly design the RPC interface. As a uAPI the subsystem provides a thin |
| 65 |
layer of discovery and a generic uAPI to deliver the RPCs and collect the |
| 66 |
response. It supports a system of user space libraries and tools which will |
| 67 |
use this interface to control the device using the device native protocols. |
| 68 |
|
| 69 |
Scope of Action |
| 70 |
--------------- |
| 71 |
|
| 72 |
fwctl drivers are strictly restricted to being a way to operate the device FW. |
| 73 |
It is not an avenue to access random kernel internals, or other operating system |
| 74 |
SW states. |
| 75 |
|
| 76 |
fwctl instances must operate on a well-defined device function, and the device |
| 77 |
should have a well-defined security model for what scope within the physical |
| 78 |
device the function is permitted to access. For instance, the most complex PCIe |
| 79 |
device today may broadly have several function-level scopes: |
| 80 |
|
| 81 |
1. A privileged function with full access to the on-device global state and |
| 82 |
configuration |
| 83 |
|
| 84 |
2. Multiple hypervisor functions with control over itself and child functions |
| 85 |
used with VMs |
| 86 |
|
| 87 |
3. Multiple VM functions tightly scoped within the VM |
| 88 |
|
| 89 |
The device may create a logical parent/child relationship between these scopes. |
| 90 |
For instance, a child VM's FW may be within the scope of the hypervisor FW. It is |
| 91 |
quite common in the VFIO world that the hypervisor environment has a complex |
| 92 |
provisioning/profiling/configuration responsibility for the function VFIO |
| 93 |
assigns to the VM. |
| 94 |
|
| 95 |
Further, within the function, devices often have RPC commands that fall within |
| 96 |
some general scopes of action (see enum fwctl_rpc_scope): |
| 97 |
|
| 98 |
1. Access to function & child configuration, FLASH, etc. that becomes live at a |
| 99 |
function reset. Access to function & child runtime configuration that is |
| 100 |
transparent or non-disruptive to any driver or VM. |
| 101 |
|
| 102 |
2. Read-only access to function debug information that may report on FW objects |
| 103 |
in the function & child, including FW objects owned by other kernel |
| 104 |
subsystems. |
| 105 |
|
| 106 |
3. Write access to function & child debug information strictly compatible with |
| 107 |
the principles of kernel lockdown and kernel integrity protection. Triggers |
| 108 |
a kernel taint. |
| 109 |
|
| 110 |
4. Full debug device access. Triggers a kernel taint, requires CAP_SYS_RAWIO. |
| 111 |
|
| 112 |
User space will provide a scope label on each RPC and the kernel must enforce the |
| 113 |
above CAPs and taints based on that scope. A combination of kernel and FW can |
| 114 |
enforce that RPCs are placed in the correct scope by user space. |
| 115 |
|
| 116 |
Disallowed behavior |
| 117 |
------------------- |
| 118 |
|
| 119 |
There are many things this interface must not allow user space to do (without a |
| 120 |
taint or CAP), broadly derived from the principles of kernel lockdown. Some |
| 121 |
examples: |
| 122 |
|
| 123 |
1. DMA to/from arbitrary memory, hang the system, compromise FW integrity with |
| 124 |
untrusted code, or otherwise compromise device or system security and |
| 125 |
integrity. |
| 126 |
|
| 127 |
2. Provide an abnormal “back door” to kernel drivers. No manipulation of kernel |
| 128 |
objects owned by kernel drivers. |
| 129 |
|
| 130 |
3. Directly configure or otherwise control kernel drivers. A subsystem kernel |
| 131 |
driver can react to the device configuration at function reset/driver load |
| 132 |
time, but otherwise must not be coupled to fwctl. |
| 133 |
|
| 134 |
4. Operate the HW in a way that overlaps with the core purpose of another |
| 135 |
primary kernel subsystem, such as read/write to LBAs, send/receive of |
| 136 |
network packets, or operate an accelerator's data plane. |
| 137 |
|
| 138 |
fwctl is not a replacement for device direct access subsystems like uacce or |
| 139 |
VFIO. |
| 140 |
|
| 141 |
Operations exposed through fwctl's non-tainting interfaces should be fully |
| 142 |
sharable with other users of the device. For instance, exposing a RPC through |
| 143 |
fwctl should never prevent a kernel subsystem from also concurrently using that |
| 144 |
same RPC or hardware unit down the road. In such cases fwctl will be less |
| 145 |
important than proper kernel subsystems that eventually emerge. Mistakes in this |
| 146 |
area resulting in clashes will be resolved in favour of a kernel implementation. |
| 147 |
|
| 148 |
fwctl User API |
| 149 |
============== |
| 150 |
|
| 151 |
.. kernel-doc:: include/uapi/fwctl/fwctl.h |
| 152 |
.. kernel-doc:: include/uapi/fwctl/mlx5.h |
| 153 |
.. kernel-doc:: include/uapi/fwctl/pds.h |
| 154 |
|
| 155 |
sysfs Class |
| 156 |
----------- |
| 157 |
|
| 158 |
fwctl has a sysfs class (/sys/class/fwctl/fwctlNN/) and character devices |
| 159 |
(/dev/fwctl/fwctlNN) with a simple numbered scheme. The character device |
| 160 |
operates the iotcl uAPI described above. |
| 161 |
|
| 162 |
fwctl devices can be related to driver components in other subsystems through |
| 163 |
sysfs:: |
| 164 |
|
| 165 |
$ ls /sys/class/fwctl/fwctl0/device/infiniband/ |
| 166 |
ibp0s10f0 |
| 167 |
|
| 168 |
$ ls /sys/class/infiniband/ibp0s10f0/device/fwctl/ |
| 169 |
fwctl0/ |
| 170 |
|
| 171 |
$ ls /sys/devices/pci0000:00/0000:00:0a.0/fwctl/fwctl0 |
| 172 |
dev device power subsystem uevent |
| 173 |
|
| 174 |
User space Community |
| 175 |
-------------------- |
| 176 |
|
| 177 |
Drawing inspiration from nvme-cli, participating in the kernel side must come |
| 178 |
with a user space in a common TBD git tree, at a minimum to usefully operate the |
| 179 |
kernel driver. Providing such an implementation is a pre-condition to merging a |
| 180 |
kernel driver. |
| 181 |
|
| 182 |
The goal is to build user space community around some of the shared problems |
| 183 |
we all have, and ideally develop some common user space programs with some |
| 184 |
starting themes of: |
| 185 |
|
| 186 |
- Device in-field debugging |
| 187 |
|
| 188 |
- HW provisioning |
| 189 |
|
| 190 |
- VFIO child device profiling before VM boot |
| 191 |
|
| 192 |
- Confidential Compute topics (attestation, secure provisioning) |
| 193 |
|
| 194 |
that stretch across all subsystems in the kernel. fwupd is a great example of |
| 195 |
how an excellent user space experience can emerge out of kernel-side diversity. |
| 196 |
|
| 197 |
fwctl Kernel API |
| 198 |
================ |
| 199 |
|
| 200 |
.. kernel-doc:: drivers/fwctl/main.c |
| 201 |
:export: |
| 202 |
.. kernel-doc:: include/linux/fwctl.h |
| 203 |
|
| 204 |
fwctl Driver design |
| 205 |
------------------- |
| 206 |
|
| 207 |
In many cases a fwctl driver is going to be part of a larger cross-subsystem |
| 208 |
device possibly using the auxiliary_device mechanism. In that case several |
| 209 |
subsystems are going to be sharing the same device and FW interface layer so the |
| 210 |
device design must already provide for isolation and cooperation between kernel |
| 211 |
subsystems. fwctl should fit into that same model. |
| 212 |
|
| 213 |
Part of the driver should include a description of how its scope restrictions |
| 214 |
and security model work. The driver and FW together must ensure that RPCs |
| 215 |
provided by user space are mapped to the appropriate scope. If the validation is |
| 216 |
done in the driver then the validation can read a 'command effects' report from |
| 217 |
the device, or hardwire the enforcement. If the validation is done in the FW, |
| 218 |
then the driver should pass the fwctl_rpc_scope to the FW along with the command. |
| 219 |
|
| 220 |
The driver and FW must cooperate to ensure that either fwctl cannot allocate |
| 221 |
any FW resources, or any resources it does allocate are freed on FD closure. A |
| 222 |
driver primarily constructed around FW RPCs may find that its core PCI function |
| 223 |
and RPC layer belongs under fwctl with auxiliary devices connecting to other |
| 224 |
subsystems. |
| 225 |
|
| 226 |
Each device type must be mindful of Linux's philosophy for stable ABI. The FW |
| 227 |
RPC interface does not have to meet a strictly stable ABI, but it does need to |
| 228 |
meet an expectation that user space tools that are deployed and in significant |
| 229 |
use don't needlessly break. FW upgrade and kernel upgrade should keep widely |
| 230 |
deployed tooling working. |
| 231 |
|
| 232 |
Development and debugging focused RPCs under more permissive scopes can have |
| 233 |
less stability if the tools using them are only run under exceptional |
| 234 |
circumstances and not for every day use of the device. Debugging tools may even |
| 235 |
require exact version matching as they may require something similar to DWARF |
| 236 |
debug information from the FW binary. |
| 237 |
|
| 238 |
Security Response |
| 239 |
================= |
| 240 |
|
| 241 |
The kernel remains the gatekeeper for this interface. If violations of the |
| 242 |
scopes, security or isolation principles are found, we have options to let |
| 243 |
devices fix them with a FW update, push a kernel patch to parse and block RPC |
| 244 |
commands or push a kernel patch to block entire firmware versions/devices. |
| 245 |
|
| 246 |
While the kernel can always directly parse and restrict RPCs, it is expected |
| 247 |
that the existing kernel pattern of allowing drivers to delegate validation to |
| 248 |
FW to be a useful design. |
| 249 |
|
| 250 |
Existing Similar Examples |
| 251 |
========================= |
| 252 |
|
| 253 |
The approach described in this document is not a new idea. Direct, or near |
| 254 |
direct device access has been offered by the kernel in different areas for |
| 255 |
decades. With more devices wanting to follow this design pattern it is becoming |
| 256 |
clear that it is not entirely well understood and, more importantly, the |
| 257 |
security considerations are not well defined or agreed upon. |
| 258 |
|
| 259 |
Some examples: |
| 260 |
|
| 261 |
- HW RAID controllers. This includes RPCs to do things like compose drives into |
| 262 |
a RAID volume, configure RAID parameters, monitor the HW and more. |
| 263 |
|
| 264 |
- Baseboard managers. RPCs for configuring settings in the device and more. |
| 265 |
|
| 266 |
- NVMe vendor command capsules. nvme-cli provides access to some monitoring |
| 267 |
functions that different products have defined, but more exist. |
| 268 |
|
| 269 |
- CXL also has a NVMe-like vendor command system. |
| 270 |
|
| 271 |
- DRM allows user space drivers to send commands to the device via kernel |
| 272 |
mediation. |
| 273 |
|
| 274 |
- RDMA allows user space drivers to directly push commands to the device |
| 275 |
without kernel involvement. |
| 276 |
|
| 277 |
- Various “raw” APIs, raw HID (SDL2), raw USB, NVMe Generic Interface, etc. |
| 278 |
|
| 279 |
The first 4 are examples of areas that fwctl intends to cover. The latter three |
| 280 |
are examples of disallowed behavior as they fully overlap with the primary purpose |
| 281 |
of a kernel subsystem. |
| 282 |
|
| 283 |
Some key lessons learned from these past efforts are the importance of having a |
| 284 |
common user space project to use as a pre-condition for obtaining a kernel |
| 285 |
driver. Developing good community around useful software in user space is key to |
| 286 |
getting companies to fund participation to enable their products. |