| 1 |
.. SPDX-License-Identifier: GPL-2.0 |
| 2 |
|
| 3 |
============================================================ |
| 4 |
Linux kernel driver for Elastic Network Adapter (ENA) family |
| 5 |
============================================================ |
| 6 |
|
| 7 |
Overview |
| 8 |
======== |
| 9 |
|
| 10 |
ENA is a networking interface designed to make good use of modern CPU |
| 11 |
features and system architectures. |
| 12 |
|
| 13 |
The ENA device exposes a lightweight management interface with a |
| 14 |
minimal set of memory mapped registers and extendible command set |
| 15 |
through an Admin Queue. |
| 16 |
|
| 17 |
The driver supports a range of ENA devices, is link-speed independent |
| 18 |
(i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc), and has |
| 19 |
a negotiated and extendible feature set. |
| 20 |
|
| 21 |
Some ENA devices support SR-IOV. This driver is used for both the |
| 22 |
SR-IOV Physical Function (PF) and Virtual Function (VF) devices. |
| 23 |
|
| 24 |
ENA devices enable high speed and low overhead network traffic |
| 25 |
processing by providing multiple Tx/Rx queue pairs (the maximum number |
| 26 |
is advertised by the device via the Admin Queue), a dedicated MSI-X |
| 27 |
interrupt vector per Tx/Rx queue pair, adaptive interrupt moderation, |
| 28 |
and CPU cacheline optimized data placement. |
| 29 |
|
| 30 |
The ENA driver supports industry standard TCP/IP offload features such as |
| 31 |
checksum offload. Receive-side scaling (RSS) is supported for multi-core |
| 32 |
scaling. |
| 33 |
|
| 34 |
The ENA driver and its corresponding devices implement health |
| 35 |
monitoring mechanisms such as watchdog, enabling the device and driver |
| 36 |
to recover in a manner transparent to the application, as well as |
| 37 |
debug logs. |
| 38 |
|
| 39 |
Some of the ENA devices support a working mode called Low-latency |
| 40 |
Queue (LLQ), which saves several more microseconds. |
| 41 |
|
| 42 |
ENA Source Code Directory Structure |
| 43 |
=================================== |
| 44 |
|
| 45 |
================= ====================================================== |
| 46 |
ena_com.[ch] Management communication layer. This layer is |
| 47 |
responsible for the handling all the management |
| 48 |
(admin) communication between the device and the |
| 49 |
driver. |
| 50 |
ena_eth_com.[ch] Tx/Rx data path. |
| 51 |
ena_admin_defs.h Definition of ENA management interface. |
| 52 |
ena_eth_io_defs.h Definition of ENA data path interface. |
| 53 |
ena_common_defs.h Common definitions for ena_com layer. |
| 54 |
ena_regs_defs.h Definition of ENA PCI memory-mapped (MMIO) registers. |
| 55 |
ena_netdev.[ch] Main Linux kernel driver. |
| 56 |
ena_ethtool.c ethtool callbacks. |
| 57 |
ena_xdp.[ch] XDP files |
| 58 |
ena_pci_id_tbl.h Supported device IDs. |
| 59 |
ena_phc.[ch] PTP hardware clock infrastructure (see `PHC`_ for more info) |
| 60 |
ena_devlink.[ch] devlink files. |
| 61 |
ena_debugfs.[ch] debugfs files. |
| 62 |
================= ====================================================== |
| 63 |
|
| 64 |
Management Interface: |
| 65 |
===================== |
| 66 |
|
| 67 |
ENA management interface is exposed by means of: |
| 68 |
|
| 69 |
- PCIe Configuration Space |
| 70 |
- Device Registers |
| 71 |
- Admin Queue (AQ) and Admin Completion Queue (ACQ) |
| 72 |
- Asynchronous Event Notification Queue (AENQ) |
| 73 |
|
| 74 |
ENA device MMIO Registers are accessed only during driver |
| 75 |
initialization and are not used during further normal device |
| 76 |
operation. |
| 77 |
|
| 78 |
AQ is used for submitting management commands, and the |
| 79 |
results/responses are reported asynchronously through ACQ. |
| 80 |
|
| 81 |
ENA introduces a small set of management commands with room for |
| 82 |
vendor-specific extensions. Most of the management operations are |
| 83 |
framed in a generic Get/Set feature command. |
| 84 |
|
| 85 |
The following admin queue commands are supported: |
| 86 |
|
| 87 |
- Create I/O submission queue |
| 88 |
- Create I/O completion queue |
| 89 |
- Destroy I/O submission queue |
| 90 |
- Destroy I/O completion queue |
| 91 |
- Get feature |
| 92 |
- Set feature |
| 93 |
- Configure AENQ |
| 94 |
- Get statistics |
| 95 |
|
| 96 |
Refer to ena_admin_defs.h for the list of supported Get/Set Feature |
| 97 |
properties. |
| 98 |
|
| 99 |
The Asynchronous Event Notification Queue (AENQ) is a uni-directional |
| 100 |
queue used by the ENA device to send to the driver events that cannot |
| 101 |
be reported using ACQ. AENQ events are subdivided into groups. Each |
| 102 |
group may have multiple syndromes, as shown below |
| 103 |
|
| 104 |
The events are: |
| 105 |
|
| 106 |
==================== =============== |
| 107 |
Group Syndrome |
| 108 |
==================== =============== |
| 109 |
Link state change **X** |
| 110 |
Fatal error **X** |
| 111 |
Notification Suspend traffic |
| 112 |
Notification Resume traffic |
| 113 |
Keep-Alive **X** |
| 114 |
==================== =============== |
| 115 |
|
| 116 |
ACQ and AENQ share the same MSI-X vector. |
| 117 |
|
| 118 |
Keep-Alive is a special mechanism that allows monitoring the device's health. |
| 119 |
A Keep-Alive event is delivered by the device every second. |
| 120 |
The driver maintains a watchdog (WD) handler which logs the current state and |
| 121 |
statistics. If the keep-alive events aren't delivered as expected the WD resets |
| 122 |
the device and the driver. |
| 123 |
|
| 124 |
Data Path Interface |
| 125 |
=================== |
| 126 |
|
| 127 |
I/O operations are based on Tx and Rx Submission Queues (Tx SQ and Rx |
| 128 |
SQ correspondingly). Each SQ has a completion queue (CQ) associated |
| 129 |
with it. |
| 130 |
|
| 131 |
The SQs and CQs are implemented as descriptor rings in contiguous |
| 132 |
physical memory. |
| 133 |
|
| 134 |
The ENA driver supports two Queue Operation modes for Tx SQs: |
| 135 |
|
| 136 |
- **Regular mode:** |
| 137 |
In this mode the Tx SQs reside in the host's memory. The ENA |
| 138 |
device fetches the ENA Tx descriptors and packet data from host |
| 139 |
memory. |
| 140 |
|
| 141 |
- **Low Latency Queue (LLQ) mode or "push-mode":** |
| 142 |
In this mode the driver pushes the transmit descriptors and the |
| 143 |
first 96 bytes of the packet directly to the ENA device memory |
| 144 |
space. The rest of the packet payload is fetched by the |
| 145 |
device. For this operation mode, the driver uses a dedicated PCI |
| 146 |
device memory BAR, which is mapped with write-combine capability. |
| 147 |
|
| 148 |
**Note that** not all ENA devices support LLQ, and this feature is negotiated |
| 149 |
with the device upon initialization. If the ENA device does not |
| 150 |
support LLQ mode, the driver falls back to the regular mode. |
| 151 |
|
| 152 |
The Rx SQs support only the regular mode. |
| 153 |
|
| 154 |
The driver supports multi-queue for both Tx and Rx. This has various |
| 155 |
benefits: |
| 156 |
|
| 157 |
- Reduced CPU/thread/process contention on a given Ethernet interface. |
| 158 |
- Cache miss rate on completion is reduced, particularly for data |
| 159 |
cache lines that hold the sk_buff structures. |
| 160 |
- Increased process-level parallelism when handling received packets. |
| 161 |
- Increased data cache hit rate, by steering kernel processing of |
| 162 |
packets to the CPU, where the application thread consuming the |
| 163 |
packet is running. |
| 164 |
- In hardware interrupt re-direction. |
| 165 |
|
| 166 |
Interrupt Modes |
| 167 |
=============== |
| 168 |
|
| 169 |
The driver assigns a single MSI-X vector per queue pair (for both Tx |
| 170 |
and Rx directions). The driver assigns an additional dedicated MSI-X vector |
| 171 |
for management (for ACQ and AENQ). |
| 172 |
|
| 173 |
Management interrupt registration is performed when the Linux kernel |
| 174 |
probes the adapter, and it is de-registered when the adapter is |
| 175 |
removed. I/O queue interrupt registration is performed when the Linux |
| 176 |
interface of the adapter is opened, and it is de-registered when the |
| 177 |
interface is closed. |
| 178 |
|
| 179 |
The management interrupt is named:: |
| 180 |
|
| 181 |
ena-mgmnt@pci:<PCI domain:bus:slot.function> |
| 182 |
|
| 183 |
and for each queue pair, an interrupt is named:: |
| 184 |
|
| 185 |
<interface name>-Tx-Rx-<queue index> |
| 186 |
|
| 187 |
The ENA device operates in auto-mask and auto-clear interrupt |
| 188 |
modes. That is, once MSI-X is delivered to the host, its Cause bit is |
| 189 |
automatically cleared and the interrupt is masked. The interrupt is |
| 190 |
unmasked by the driver after NAPI processing is complete. |
| 191 |
|
| 192 |
Interrupt Moderation |
| 193 |
==================== |
| 194 |
|
| 195 |
ENA driver and device can operate in conventional or adaptive interrupt |
| 196 |
moderation mode. |
| 197 |
|
| 198 |
**In conventional mode** the driver instructs device to postpone interrupt |
| 199 |
posting according to static interrupt delay value. The interrupt delay |
| 200 |
value can be configured through `ethtool(8)`. The following `ethtool` |
| 201 |
parameters are supported by the driver: ``tx-usecs``, ``rx-usecs`` |
| 202 |
|
| 203 |
**In adaptive interrupt** moderation mode the interrupt delay value is |
| 204 |
updated by the driver dynamically and adjusted every NAPI cycle |
| 205 |
according to the traffic nature. |
| 206 |
|
| 207 |
Adaptive coalescing can be switched on/off through `ethtool(8)`'s |
| 208 |
:code:`adaptive_rx on|off` parameter. |
| 209 |
|
| 210 |
More information about Adaptive Interrupt Moderation (DIM) can be found in |
| 211 |
Documentation/networking/net_dim.rst |
| 212 |
|
| 213 |
.. _`RX copybreak`: |
| 214 |
|
| 215 |
RX copybreak |
| 216 |
============ |
| 217 |
|
| 218 |
The rx_copybreak is initialized by default to ENA_DEFAULT_RX_COPYBREAK |
| 219 |
and can be configured by the ETHTOOL_STUNABLE command of the |
| 220 |
SIOCETHTOOL ioctl. |
| 221 |
|
| 222 |
This option controls the maximum packet length for which the RX |
| 223 |
descriptor it was received on would be recycled. When a packet smaller |
| 224 |
than RX copybreak bytes is received, it is copied into a new memory |
| 225 |
buffer and the RX descriptor is returned to HW. |
| 226 |
|
| 227 |
.. _`PHC`: |
| 228 |
|
| 229 |
PTP Hardware Clock (PHC) |
| 230 |
======================== |
| 231 |
.. _`ptp-userspace-api`: https://docs.kernel.org/driver-api/ptp.html#ptp-hardware-clock-user-space-api |
| 232 |
.. _`testptp`: https://elixir.bootlin.com/linux/latest/source/tools/testing/selftests/ptp/testptp.c |
| 233 |
|
| 234 |
ENA Linux driver supports PTP hardware clock providing timestamp reference to achieve nanosecond resolution. |
| 235 |
|
| 236 |
**PHC support** |
| 237 |
|
| 238 |
PHC depends on the PTP module, which needs to be either loaded as a module or compiled into the kernel. |
| 239 |
|
| 240 |
Verify if the PTP module is present: |
| 241 |
|
| 242 |
.. code-block:: shell |
| 243 |
|
| 244 |
grep -w '^CONFIG_PTP_1588_CLOCK=[ym]' /boot/config-`uname -r` |
| 245 |
|
| 246 |
- If no output is provided, the ENA driver cannot be loaded with PHC support. |
| 247 |
|
| 248 |
**PHC activation** |
| 249 |
|
| 250 |
The feature is turned off by default, in order to turn the feature on, the ENA driver |
| 251 |
can be loaded in the following way: |
| 252 |
|
| 253 |
- devlink: |
| 254 |
|
| 255 |
.. code-block:: shell |
| 256 |
|
| 257 |
sudo devlink dev param set pci/<domain:bus:slot.function> name enable_phc value true cmode driverinit |
| 258 |
sudo devlink dev reload pci/<domain:bus:slot.function> |
| 259 |
# for example: |
| 260 |
sudo devlink dev param set pci/0000:00:06.0 name enable_phc value true cmode driverinit |
| 261 |
sudo devlink dev reload pci/0000:00:06.0 |
| 262 |
|
| 263 |
All available PTP clock sources can be tracked here: |
| 264 |
|
| 265 |
.. code-block:: shell |
| 266 |
|
| 267 |
ls /sys/class/ptp |
| 268 |
|
| 269 |
PHC support and capabilities can be verified using ethtool: |
| 270 |
|
| 271 |
.. code-block:: shell |
| 272 |
|
| 273 |
ethtool -T <interface> |
| 274 |
|
| 275 |
**PHC timestamp** |
| 276 |
|
| 277 |
To retrieve PHC timestamp, use `ptp-userspace-api`_, usage example using `testptp`_: |
| 278 |
|
| 279 |
.. code-block:: shell |
| 280 |
|
| 281 |
testptp -d /dev/ptp$(ethtool -T <interface> | awk '/PTP Hardware Clock:/ {print $NF}') -k 1 |
| 282 |
|
| 283 |
PHC get time requests should be within reasonable bounds, |
| 284 |
avoid excessive utilization to ensure optimal performance and efficiency. |
| 285 |
The ENA device restricts the frequency of PHC get time requests to a maximum |
| 286 |
of 125 requests per second. If this limit is surpassed, the get time request |
| 287 |
will fail, leading to an increment in the phc_err_ts statistic. |
| 288 |
|
| 289 |
**PHC statistics** |
| 290 |
|
| 291 |
PHC can be monitored using debugfs (if mounted): |
| 292 |
|
| 293 |
.. code-block:: shell |
| 294 |
|
| 295 |
sudo cat /sys/kernel/debug/<domain:bus:slot.function>/phc_stats |
| 296 |
|
| 297 |
# for example: |
| 298 |
sudo cat /sys/kernel/debug/0000:00:06.0/phc_stats |
| 299 |
|
| 300 |
PHC errors must remain below 1% of all PHC requests to maintain the desired level of accuracy and reliability |
| 301 |
|
| 302 |
================= ====================================================== |
| 303 |
**phc_cnt** | Number of successful retrieved timestamps (below expire timeout). |
| 304 |
**phc_exp** | Number of expired retrieved timestamps (above expire timeout). |
| 305 |
**phc_skp** | Number of skipped get time attempts (during block period). |
| 306 |
**phc_err_dv** | Number of failed get time attempts due to device errors (entering into block state). |
| 307 |
**phc_err_ts** | Number of failed get time attempts due to timestamp errors (entering into block state), |
| 308 |
| This occurs if driver exceeded the request limit or device received an invalid timestamp. |
| 309 |
================= ====================================================== |
| 310 |
|
| 311 |
PHC timeouts: |
| 312 |
|
| 313 |
================= ====================================================== |
| 314 |
**expire** | Max time for a valid timestamp retrieval, passing this threshold will fail |
| 315 |
| the get time request and block new requests until block timeout. |
| 316 |
**block** | Blocking period starts once get time request expires or fails, |
| 317 |
| all get time requests during block period will be skipped. |
| 318 |
================= ====================================================== |
| 319 |
|
| 320 |
Statistics |
| 321 |
========== |
| 322 |
|
| 323 |
The user can obtain ENA device and driver statistics using `ethtool`. |
| 324 |
The driver can collect regular or extended statistics (including |
| 325 |
per-queue stats) from the device. |
| 326 |
|
| 327 |
In addition the driver logs the stats to syslog upon device reset. |
| 328 |
|
| 329 |
On supported instance types, the statistics will also include the |
| 330 |
ENA Express data (fields prefixed with `ena_srd`). For a complete |
| 331 |
documentation of ENA Express data refer to |
| 332 |
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ena-express.html#ena-express-monitor |
| 333 |
|
| 334 |
MTU |
| 335 |
=== |
| 336 |
|
| 337 |
The driver supports an arbitrarily large MTU with a maximum that is |
| 338 |
negotiated with the device. The driver configures MTU using the |
| 339 |
SetFeature command (ENA_ADMIN_MTU property). The user can change MTU |
| 340 |
via `ip(8)` and similar legacy tools. |
| 341 |
|
| 342 |
Stateless Offloads |
| 343 |
================== |
| 344 |
|
| 345 |
The ENA driver supports: |
| 346 |
|
| 347 |
- IPv4 header checksum offload |
| 348 |
- TCP/UDP over IPv4/IPv6 checksum offloads |
| 349 |
|
| 350 |
RSS |
| 351 |
=== |
| 352 |
|
| 353 |
- The ENA device supports RSS that allows flexible Rx traffic |
| 354 |
steering. |
| 355 |
- Toeplitz and CRC32 hash functions are supported. |
| 356 |
- Different combinations of L2/L3/L4 fields can be configured as |
| 357 |
inputs for hash functions. |
| 358 |
- The driver configures RSS settings using the AQ SetFeature command |
| 359 |
(ENA_ADMIN_RSS_HASH_FUNCTION, ENA_ADMIN_RSS_HASH_INPUT and |
| 360 |
ENA_ADMIN_RSS_INDIRECTION_TABLE_CONFIG properties). |
| 361 |
- If the NETIF_F_RXHASH flag is set, the 32-bit result of the hash |
| 362 |
function delivered in the Rx CQ descriptor is set in the received |
| 363 |
SKB. |
| 364 |
- The user can provide a hash key, hash function, and configure the |
| 365 |
indirection table through `ethtool(8)`. |
| 366 |
|
| 367 |
DEVLINK SUPPORT |
| 368 |
=============== |
| 369 |
.. _`devlink`: https://www.kernel.org/doc/html/latest/networking/devlink/index.html |
| 370 |
|
| 371 |
`devlink`_ supports reloading the driver and initiating re-negotiation with the ENA device |
| 372 |
|
| 373 |
.. code-block:: shell |
| 374 |
|
| 375 |
sudo devlink dev reload pci/<domain:bus:slot.function> |
| 376 |
# for example: |
| 377 |
sudo devlink dev reload pci/0000:00:06.0 |
| 378 |
|
| 379 |
DATA PATH |
| 380 |
========= |
| 381 |
|
| 382 |
Tx |
| 383 |
-- |
| 384 |
|
| 385 |
:code:`ena_start_xmit()` is called by the stack. This function does the following: |
| 386 |
|
| 387 |
- Maps data buffers (``skb->data`` and frags). |
| 388 |
- Populates ``ena_buf`` for the push buffer (if the driver and device are |
| 389 |
in push mode). |
| 390 |
- Prepares ENA bufs for the remaining frags. |
| 391 |
- Allocates a new request ID from the empty ``req_id`` ring. The request |
| 392 |
ID is the index of the packet in the Tx info. This is used for |
| 393 |
out-of-order Tx completions. |
| 394 |
- Adds the packet to the proper place in the Tx ring. |
| 395 |
- Calls :code:`ena_com_prepare_tx()`, an ENA communication layer that converts |
| 396 |
the ``ena_bufs`` to ENA descriptors (and adds meta ENA descriptors as |
| 397 |
needed). |
| 398 |
|
| 399 |
* This function also copies the ENA descriptors and the push buffer |
| 400 |
to the Device memory space (if in push mode). |
| 401 |
|
| 402 |
- Writes a doorbell to the ENA device. |
| 403 |
- When the ENA device finishes sending the packet, a completion |
| 404 |
interrupt is raised. |
| 405 |
- The interrupt handler schedules NAPI. |
| 406 |
- The :code:`ena_clean_tx_irq()` function is called. This function handles the |
| 407 |
completion descriptors generated by the ENA, with a single |
| 408 |
completion descriptor per completed packet. |
| 409 |
|
| 410 |
* ``req_id`` is retrieved from the completion descriptor. The ``tx_info`` of |
| 411 |
the packet is retrieved via the ``req_id``. The data buffers are |
| 412 |
unmapped and ``req_id`` is returned to the empty ``req_id`` ring. |
| 413 |
* The function stops when the completion descriptors are completed or |
| 414 |
the budget is reached. |
| 415 |
|
| 416 |
Rx |
| 417 |
-- |
| 418 |
|
| 419 |
- When a packet is received from the ENA device. |
| 420 |
- The interrupt handler schedules NAPI. |
| 421 |
- The :code:`ena_clean_rx_irq()` function is called. This function calls |
| 422 |
:code:`ena_com_rx_pkt()`, an ENA communication layer function, which returns the |
| 423 |
number of descriptors used for a new packet, and zero if |
| 424 |
no new packet is found. |
| 425 |
- :code:`ena_rx_skb()` checks packet length: |
| 426 |
|
| 427 |
* If the packet is small (len < rx_copybreak), the driver allocates |
| 428 |
a SKB for the new packet, and copies the packet payload into the |
| 429 |
SKB data buffer. |
| 430 |
|
| 431 |
- In this way the original data buffer is not passed to the stack |
| 432 |
and is reused for future Rx packets. |
| 433 |
|
| 434 |
* Otherwise the function unmaps the Rx buffer, sets the first |
| 435 |
descriptor as `skb`'s linear part and the other descriptors as the |
| 436 |
`skb`'s frags. |
| 437 |
|
| 438 |
- The new SKB is updated with the necessary information (protocol, |
| 439 |
checksum hw verify result, etc), and then passed to the network |
| 440 |
stack, using the NAPI interface function :code:`napi_gro_receive()`. |
| 441 |
|
| 442 |
Dynamic RX Buffers (DRB) |
| 443 |
------------------------ |
| 444 |
|
| 445 |
Each RX descriptor in the RX ring is a single memory page (which is either 4KB |
| 446 |
or 16KB long depending on system's configurations). |
| 447 |
To reduce the memory allocations required when dealing with a high rate of small |
| 448 |
packets, the driver tries to reuse the remaining RX descriptor's space if more |
| 449 |
than 2KB of this page remain unused. |
| 450 |
|
| 451 |
A simple example of this mechanism is the following sequence of events: |
| 452 |
|
| 453 |
:: |
| 454 |
|
| 455 |
1. Driver allocates page-sized RX buffer and passes it to hardware |
| 456 |
+----------------------+ |
| 457 |
|4KB RX Buffer | |
| 458 |
+----------------------+ |
| 459 |
|
| 460 |
2. A 300Bytes packet is received on this buffer |
| 461 |
|
| 462 |
3. The driver increases the ref count on this page and returns it back to |
| 463 |
HW as an RX buffer of size 4KB - 300Bytes = 3796 Bytes |
| 464 |
+----+--------------------+ |
| 465 |
|****|3796 Bytes RX Buffer| |
| 466 |
+----+--------------------+ |
| 467 |
|
| 468 |
This mechanism isn't used when an XDP program is loaded, or when the |
| 469 |
RX packet is less than rx_copybreak bytes (in which case the packet is |
| 470 |
copied out of the RX buffer into the linear part of a new skb allocated |
| 471 |
for it and the RX buffer remains the same size, see `RX copybreak`_). |