Debian Codegraph @

Documentation/networking/device_drivers/ethernet/amazon/ena.rst

raw
1 .. SPDX-License-Identifier: GPL-2.0
2
3 ============================================================
4 Linux kernel driver for Elastic Network Adapter (ENA) family
5 ============================================================
6
7 Overview
8 ========
9
10 ENA is a networking interface designed to make good use of modern CPU
11 features and system architectures.
12
13 The ENA device exposes a lightweight management interface with a
14 minimal set of memory mapped registers and extendible command set
15 through an Admin Queue.
16
17 The driver supports a range of ENA devices, is link-speed independent
18 (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc), and has
19 a negotiated and extendible feature set.
20
21 Some ENA devices support SR-IOV. This driver is used for both the
22 SR-IOV Physical Function (PF) and Virtual Function (VF) devices.
23
24 ENA devices enable high speed and low overhead network traffic
25 processing by providing multiple Tx/Rx queue pairs (the maximum number
26 is advertised by the device via the Admin Queue), a dedicated MSI-X
27 interrupt vector per Tx/Rx queue pair, adaptive interrupt moderation,
28 and CPU cacheline optimized data placement.
29
30 The ENA driver supports industry standard TCP/IP offload features such as
31 checksum offload. Receive-side scaling (RSS) is supported for multi-core
32 scaling.
33
34 The ENA driver and its corresponding devices implement health
35 monitoring mechanisms such as watchdog, enabling the device and driver
36 to recover in a manner transparent to the application, as well as
37 debug logs.
38
39 Some of the ENA devices support a working mode called Low-latency
40 Queue (LLQ), which saves several more microseconds.
41
42 ENA Source Code Directory Structure
43 ===================================
44
45 ================= ======================================================
46 ena_com.[ch] Management communication layer. This layer is
47 responsible for the handling all the management
48 (admin) communication between the device and the
49 driver.
50 ena_eth_com.[ch] Tx/Rx data path.
51 ena_admin_defs.h Definition of ENA management interface.
52 ena_eth_io_defs.h Definition of ENA data path interface.
53 ena_common_defs.h Common definitions for ena_com layer.
54 ena_regs_defs.h Definition of ENA PCI memory-mapped (MMIO) registers.
55 ena_netdev.[ch] Main Linux kernel driver.
56 ena_ethtool.c ethtool callbacks.
57 ena_xdp.[ch] XDP files
58 ena_pci_id_tbl.h Supported device IDs.
59 ena_phc.[ch] PTP hardware clock infrastructure (see `PHC`_ for more info)
60 ena_devlink.[ch] devlink files.
61 ena_debugfs.[ch] debugfs files.
62 ================= ======================================================
63
64 Management Interface:
65 =====================
66
67 ENA management interface is exposed by means of:
68
69 - PCIe Configuration Space
70 - Device Registers
71 - Admin Queue (AQ) and Admin Completion Queue (ACQ)
72 - Asynchronous Event Notification Queue (AENQ)
73
74 ENA device MMIO Registers are accessed only during driver
75 initialization and are not used during further normal device
76 operation.
77
78 AQ is used for submitting management commands, and the
79 results/responses are reported asynchronously through ACQ.
80
81 ENA introduces a small set of management commands with room for
82 vendor-specific extensions. Most of the management operations are
83 framed in a generic Get/Set feature command.
84
85 The following admin queue commands are supported:
86
87 - Create I/O submission queue
88 - Create I/O completion queue
89 - Destroy I/O submission queue
90 - Destroy I/O completion queue
91 - Get feature
92 - Set feature
93 - Configure AENQ
94 - Get statistics
95
96 Refer to ena_admin_defs.h for the list of supported Get/Set Feature
97 properties.
98
99 The Asynchronous Event Notification Queue (AENQ) is a uni-directional
100 queue used by the ENA device to send to the driver events that cannot
101 be reported using ACQ. AENQ events are subdivided into groups. Each
102 group may have multiple syndromes, as shown below
103
104 The events are:
105
106 ==================== ===============
107 Group Syndrome
108 ==================== ===============
109 Link state change **X**
110 Fatal error **X**
111 Notification Suspend traffic
112 Notification Resume traffic
113 Keep-Alive **X**
114 ==================== ===============
115
116 ACQ and AENQ share the same MSI-X vector.
117
118 Keep-Alive is a special mechanism that allows monitoring the device's health.
119 A Keep-Alive event is delivered by the device every second.
120 The driver maintains a watchdog (WD) handler which logs the current state and
121 statistics. If the keep-alive events aren't delivered as expected the WD resets
122 the device and the driver.
123
124 Data Path Interface
125 ===================
126
127 I/O operations are based on Tx and Rx Submission Queues (Tx SQ and Rx
128 SQ correspondingly). Each SQ has a completion queue (CQ) associated
129 with it.
130
131 The SQs and CQs are implemented as descriptor rings in contiguous
132 physical memory.
133
134 The ENA driver supports two Queue Operation modes for Tx SQs:
135
136 - **Regular mode:**
137 In this mode the Tx SQs reside in the host's memory. The ENA
138 device fetches the ENA Tx descriptors and packet data from host
139 memory.
140
141 - **Low Latency Queue (LLQ) mode or "push-mode":**
142 In this mode the driver pushes the transmit descriptors and the
143 first 96 bytes of the packet directly to the ENA device memory
144 space. The rest of the packet payload is fetched by the
145 device. For this operation mode, the driver uses a dedicated PCI
146 device memory BAR, which is mapped with write-combine capability.
147
148 **Note that** not all ENA devices support LLQ, and this feature is negotiated
149 with the device upon initialization. If the ENA device does not
150 support LLQ mode, the driver falls back to the regular mode.
151
152 The Rx SQs support only the regular mode.
153
154 The driver supports multi-queue for both Tx and Rx. This has various
155 benefits:
156
157 - Reduced CPU/thread/process contention on a given Ethernet interface.
158 - Cache miss rate on completion is reduced, particularly for data
159 cache lines that hold the sk_buff structures.
160 - Increased process-level parallelism when handling received packets.
161 - Increased data cache hit rate, by steering kernel processing of
162 packets to the CPU, where the application thread consuming the
163 packet is running.
164 - In hardware interrupt re-direction.
165
166 Interrupt Modes
167 ===============
168
169 The driver assigns a single MSI-X vector per queue pair (for both Tx
170 and Rx directions). The driver assigns an additional dedicated MSI-X vector
171 for management (for ACQ and AENQ).
172
173 Management interrupt registration is performed when the Linux kernel
174 probes the adapter, and it is de-registered when the adapter is
175 removed. I/O queue interrupt registration is performed when the Linux
176 interface of the adapter is opened, and it is de-registered when the
177 interface is closed.
178
179 The management interrupt is named::
180
181 ena-mgmnt@pci:<PCI domain:bus:slot.function>
182
183 and for each queue pair, an interrupt is named::
184
185 <interface name>-Tx-Rx-<queue index>
186
187 The ENA device operates in auto-mask and auto-clear interrupt
188 modes. That is, once MSI-X is delivered to the host, its Cause bit is
189 automatically cleared and the interrupt is masked. The interrupt is
190 unmasked by the driver after NAPI processing is complete.
191
192 Interrupt Moderation
193 ====================
194
195 ENA driver and device can operate in conventional or adaptive interrupt
196 moderation mode.
197
198 **In conventional mode** the driver instructs device to postpone interrupt
199 posting according to static interrupt delay value. The interrupt delay
200 value can be configured through `ethtool(8)`. The following `ethtool`
201 parameters are supported by the driver: ``tx-usecs``, ``rx-usecs``
202
203 **In adaptive interrupt** moderation mode the interrupt delay value is
204 updated by the driver dynamically and adjusted every NAPI cycle
205 according to the traffic nature.
206
207 Adaptive coalescing can be switched on/off through `ethtool(8)`'s
208 :code:`adaptive_rx on|off` parameter.
209
210 More information about Adaptive Interrupt Moderation (DIM) can be found in
211 Documentation/networking/net_dim.rst
212
213 .. _`RX copybreak`:
214
215 RX copybreak
216 ============
217
218 The rx_copybreak is initialized by default to ENA_DEFAULT_RX_COPYBREAK
219 and can be configured by the ETHTOOL_STUNABLE command of the
220 SIOCETHTOOL ioctl.
221
222 This option controls the maximum packet length for which the RX
223 descriptor it was received on would be recycled. When a packet smaller
224 than RX copybreak bytes is received, it is copied into a new memory
225 buffer and the RX descriptor is returned to HW.
226
227 .. _`PHC`:
228
229 PTP Hardware Clock (PHC)
230 ========================
231 .. _`ptp-userspace-api`: https://docs.kernel.org/driver-api/ptp.html#ptp-hardware-clock-user-space-api
232 .. _`testptp`: https://elixir.bootlin.com/linux/latest/source/tools/testing/selftests/ptp/testptp.c
233
234 ENA Linux driver supports PTP hardware clock providing timestamp reference to achieve nanosecond resolution.
235
236 **PHC support**
237
238 PHC depends on the PTP module, which needs to be either loaded as a module or compiled into the kernel.
239
240 Verify if the PTP module is present:
241
242 .. code-block:: shell
243
244 grep -w '^CONFIG_PTP_1588_CLOCK=[ym]' /boot/config-`uname -r`
245
246 - If no output is provided, the ENA driver cannot be loaded with PHC support.
247
248 **PHC activation**
249
250 The feature is turned off by default, in order to turn the feature on, the ENA driver
251 can be loaded in the following way:
252
253 - devlink:
254
255 .. code-block:: shell
256
257 sudo devlink dev param set pci/<domain:bus:slot.function> name enable_phc value true cmode driverinit
258 sudo devlink dev reload pci/<domain:bus:slot.function>
259 # for example:
260 sudo devlink dev param set pci/0000:00:06.0 name enable_phc value true cmode driverinit
261 sudo devlink dev reload pci/0000:00:06.0
262
263 All available PTP clock sources can be tracked here:
264
265 .. code-block:: shell
266
267 ls /sys/class/ptp
268
269 PHC support and capabilities can be verified using ethtool:
270
271 .. code-block:: shell
272
273 ethtool -T <interface>
274
275 **PHC timestamp**
276
277 To retrieve PHC timestamp, use `ptp-userspace-api`_, usage example using `testptp`_:
278
279 .. code-block:: shell
280
281 testptp -d /dev/ptp$(ethtool -T <interface> | awk '/PTP Hardware Clock:/ {print $NF}') -k 1
282
283 PHC get time requests should be within reasonable bounds,
284 avoid excessive utilization to ensure optimal performance and efficiency.
285 The ENA device restricts the frequency of PHC get time requests to a maximum
286 of 125 requests per second. If this limit is surpassed, the get time request
287 will fail, leading to an increment in the phc_err_ts statistic.
288
289 **PHC statistics**
290
291 PHC can be monitored using debugfs (if mounted):
292
293 .. code-block:: shell
294
295 sudo cat /sys/kernel/debug/<domain:bus:slot.function>/phc_stats
296
297 # for example:
298 sudo cat /sys/kernel/debug/0000:00:06.0/phc_stats
299
300 PHC errors must remain below 1% of all PHC requests to maintain the desired level of accuracy and reliability
301
302 ================= ======================================================
303 **phc_cnt** | Number of successful retrieved timestamps (below expire timeout).
304 **phc_exp** | Number of expired retrieved timestamps (above expire timeout).
305 **phc_skp** | Number of skipped get time attempts (during block period).
306 **phc_err_dv** | Number of failed get time attempts due to device errors (entering into block state).
307 **phc_err_ts** | Number of failed get time attempts due to timestamp errors (entering into block state),
308 | This occurs if driver exceeded the request limit or device received an invalid timestamp.
309 ================= ======================================================
310
311 PHC timeouts:
312
313 ================= ======================================================
314 **expire** | Max time for a valid timestamp retrieval, passing this threshold will fail
315 | the get time request and block new requests until block timeout.
316 **block** | Blocking period starts once get time request expires or fails,
317 | all get time requests during block period will be skipped.
318 ================= ======================================================
319
320 Statistics
321 ==========
322
323 The user can obtain ENA device and driver statistics using `ethtool`.
324 The driver can collect regular or extended statistics (including
325 per-queue stats) from the device.
326
327 In addition the driver logs the stats to syslog upon device reset.
328
329 On supported instance types, the statistics will also include the
330 ENA Express data (fields prefixed with `ena_srd`). For a complete
331 documentation of ENA Express data refer to
332 https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ena-express.html#ena-express-monitor
333
334 MTU
335 ===
336
337 The driver supports an arbitrarily large MTU with a maximum that is
338 negotiated with the device. The driver configures MTU using the
339 SetFeature command (ENA_ADMIN_MTU property). The user can change MTU
340 via `ip(8)` and similar legacy tools.
341
342 Stateless Offloads
343 ==================
344
345 The ENA driver supports:
346
347 - IPv4 header checksum offload
348 - TCP/UDP over IPv4/IPv6 checksum offloads
349
350 RSS
351 ===
352
353 - The ENA device supports RSS that allows flexible Rx traffic
354 steering.
355 - Toeplitz and CRC32 hash functions are supported.
356 - Different combinations of L2/L3/L4 fields can be configured as
357 inputs for hash functions.
358 - The driver configures RSS settings using the AQ SetFeature command
359 (ENA_ADMIN_RSS_HASH_FUNCTION, ENA_ADMIN_RSS_HASH_INPUT and
360 ENA_ADMIN_RSS_INDIRECTION_TABLE_CONFIG properties).
361 - If the NETIF_F_RXHASH flag is set, the 32-bit result of the hash
362 function delivered in the Rx CQ descriptor is set in the received
363 SKB.
364 - The user can provide a hash key, hash function, and configure the
365 indirection table through `ethtool(8)`.
366
367 DEVLINK SUPPORT
368 ===============
369 .. _`devlink`: https://www.kernel.org/doc/html/latest/networking/devlink/index.html
370
371 `devlink`_ supports reloading the driver and initiating re-negotiation with the ENA device
372
373 .. code-block:: shell
374
375 sudo devlink dev reload pci/<domain:bus:slot.function>
376 # for example:
377 sudo devlink dev reload pci/0000:00:06.0
378
379 DATA PATH
380 =========
381
382 Tx
383 --
384
385 :code:`ena_start_xmit()` is called by the stack. This function does the following:
386
387 - Maps data buffers (``skb->data`` and frags).
388 - Populates ``ena_buf`` for the push buffer (if the driver and device are
389 in push mode).
390 - Prepares ENA bufs for the remaining frags.
391 - Allocates a new request ID from the empty ``req_id`` ring. The request
392 ID is the index of the packet in the Tx info. This is used for
393 out-of-order Tx completions.
394 - Adds the packet to the proper place in the Tx ring.
395 - Calls :code:`ena_com_prepare_tx()`, an ENA communication layer that converts
396 the ``ena_bufs`` to ENA descriptors (and adds meta ENA descriptors as
397 needed).
398
399 * This function also copies the ENA descriptors and the push buffer
400 to the Device memory space (if in push mode).
401
402 - Writes a doorbell to the ENA device.
403 - When the ENA device finishes sending the packet, a completion
404 interrupt is raised.
405 - The interrupt handler schedules NAPI.
406 - The :code:`ena_clean_tx_irq()` function is called. This function handles the
407 completion descriptors generated by the ENA, with a single
408 completion descriptor per completed packet.
409
410 * ``req_id`` is retrieved from the completion descriptor. The ``tx_info`` of
411 the packet is retrieved via the ``req_id``. The data buffers are
412 unmapped and ``req_id`` is returned to the empty ``req_id`` ring.
413 * The function stops when the completion descriptors are completed or
414 the budget is reached.
415
416 Rx
417 --
418
419 - When a packet is received from the ENA device.
420 - The interrupt handler schedules NAPI.
421 - The :code:`ena_clean_rx_irq()` function is called. This function calls
422 :code:`ena_com_rx_pkt()`, an ENA communication layer function, which returns the
423 number of descriptors used for a new packet, and zero if
424 no new packet is found.
425 - :code:`ena_rx_skb()` checks packet length:
426
427 * If the packet is small (len < rx_copybreak), the driver allocates
428 a SKB for the new packet, and copies the packet payload into the
429 SKB data buffer.
430
431 - In this way the original data buffer is not passed to the stack
432 and is reused for future Rx packets.
433
434 * Otherwise the function unmaps the Rx buffer, sets the first
435 descriptor as `skb`'s linear part and the other descriptors as the
436 `skb`'s frags.
437
438 - The new SKB is updated with the necessary information (protocol,
439 checksum hw verify result, etc), and then passed to the network
440 stack, using the NAPI interface function :code:`napi_gro_receive()`.
441
442 Dynamic RX Buffers (DRB)
443 ------------------------
444
445 Each RX descriptor in the RX ring is a single memory page (which is either 4KB
446 or 16KB long depending on system's configurations).
447 To reduce the memory allocations required when dealing with a high rate of small
448 packets, the driver tries to reuse the remaining RX descriptor's space if more
449 than 2KB of this page remain unused.
450
451 A simple example of this mechanism is the following sequence of events:
452
453 ::
454
455 1. Driver allocates page-sized RX buffer and passes it to hardware
456 +----------------------+
457 |4KB RX Buffer |
458 +----------------------+
459
460 2. A 300Bytes packet is received on this buffer
461
462 3. The driver increases the ref count on this page and returns it back to
463 HW as an RX buffer of size 4KB - 300Bytes = 3796 Bytes
464 +----+--------------------+
465 |****|3796 Bytes RX Buffer|
466 +----+--------------------+
467
468 This mechanism isn't used when an XDP program is loaded, or when the
469 RX packet is less than rx_copybreak bytes (in which case the packet is
470 copied out of the RX buffer into the linear part of a new skb allocated
471 for it and the RX buffer remains the same size, see `RX copybreak`_).