Debian Codegraph @

Documentation/networking/device_drivers/ethernet/intel/i40e.rst

raw
1 .. SPDX-License-Identifier: GPL-2.0+
2
3 =================================================================
4 Linux Base Driver for the Intel(R) Ethernet Controller 700 Series
5 =================================================================
6
7 Intel 40 Gigabit Linux driver.
8 Copyright(c) 1999-2018 Intel Corporation.
9
10 Contents
11 ========
12
13 - Overview
14 - Identifying Your Adapter
15 - Intel(R) Ethernet Flow Director
16 - Additional Configurations
17 - Known Issues
18 - Support
19
20
21 Driver information can be obtained using ethtool, lspci, and ifconfig.
22 Instructions on updating ethtool can be found in the section Additional
23 Configurations later in this document.
24
25 For questions related to hardware requirements, refer to the documentation
26 supplied with your Intel adapter. All hardware requirements listed apply to use
27 with Linux.
28
29
30 Identifying Your Adapter
31 ========================
32 The driver is compatible with devices based on the following:
33
34 * Intel(R) Ethernet Controller X710
35 * Intel(R) Ethernet Controller XL710
36 * Intel(R) Ethernet Network Connection X722
37 * Intel(R) Ethernet Controller XXV710
38
39 For the best performance, make sure the latest NVM/FW is installed on your
40 device.
41
42 For information on how to identify your adapter, and for the latest NVM/FW
43 images and Intel network drivers, refer to the Intel Support website:
44 https://www.intel.com/support
45
46 SFP+ and QSFP+ Devices
47 ----------------------
48 For information about supported media, refer to this document:
49 https://www.intel.com/content/dam/www/public/us/en/documents/release-notes/xl710-ethernet-controller-feature-matrix.pdf
50
51 NOTE: Some adapters based on the Intel(R) Ethernet Controller 700 Series only
52 support Intel Ethernet Optics modules. On these adapters, other modules are not
53 supported and will not function. In all cases Intel recommends using Intel
54 Ethernet Optics; other modules may function but are not validated by Intel.
55 Contact Intel for supported media types.
56
57 NOTE: For connections based on Intel(R) Ethernet Controller 700 Series, support
58 is dependent on your system board. Please see your vendor for details.
59
60 NOTE: In systems that do not have adequate airflow to cool the adapter and
61 optical modules, you must use high temperature optical modules.
62
63 Virtual Functions (VFs)
64 -----------------------
65 Use sysfs to enable VFs. For example::
66
67 #echo $num_vf_enabled > /sys/class/net/$dev/device/sriov_numvfs #enable VFs
68 #echo 0 > /sys/class/net/$dev/device/sriov_numvfs #disable VFs
69
70 For example, the following instructions will configure PF eth0 and the first VF
71 on VLAN 10::
72
73 $ ip link set dev eth0 vf 0 vlan 10
74
75 VLAN Tag Packet Steering
76 ------------------------
77 Allows you to send all packets with a specific VLAN tag to a particular SR-IOV
78 virtual function (VF). Further, this feature allows you to designate a
79 particular VF as trusted, and allows that trusted VF to request selective
80 promiscuous mode on the Physical Function (PF).
81
82 To set a VF as trusted or untrusted, enter the following command in the
83 Hypervisor::
84
85 # ip link set dev eth0 vf 1 trust [on|off]
86
87 Once the VF is designated as trusted, use the following commands in the VM to
88 set the VF to promiscuous mode.
89
90 ::
91
92 For promiscuous all:
93 #ip link set eth2 promisc on
94 Where eth2 is a VF interface in the VM
95
96 For promiscuous Multicast:
97 #ip link set eth2 allmulticast on
98 Where eth2 is a VF interface in the VM
99
100 NOTE: By default, the ethtool priv-flag vf-true-promisc-support is set to
101 "off",meaning that promiscuous mode for the VF will be limited. To set the
102 promiscuous mode for the VF to true promiscuous and allow the VF to see all
103 ingress traffic, use the following command::
104
105 #ethtool -set-priv-flags p261p1 vf-true-promisc-support on
106
107 The vf-true-promisc-support priv-flag does not enable promiscuous mode; rather,
108 it designates which type of promiscuous mode (limited or true) you will get
109 when you enable promiscuous mode using the ip link commands above. Note that
110 this is a global setting that affects the entire device. However,the
111 vf-true-promisc-support priv-flag is only exposed to the first PF of the
112 device. The PF remains in limited promiscuous mode (unless it is in MFP mode)
113 regardless of the vf-true-promisc-support setting.
114
115 Now add a VLAN interface on the VF interface::
116
117 #ip link add link eth2 name eth2.100 type vlan id 100
118
119 Note that the order in which you set the VF to promiscuous mode and add the
120 VLAN interface does not matter (you can do either first). The end result in
121 this example is that the VF will get all traffic that is tagged with VLAN 100.
122
123 Intel(R) Ethernet Flow Director
124 -------------------------------
125 The Intel Ethernet Flow Director performs the following tasks:
126
127 - Directs receive packets according to their flows to different queues.
128 - Enables tight control on routing a flow in the platform.
129 - Matches flows and CPU cores for flow affinity.
130 - Supports multiple parameters for flexible flow classification and load
131 balancing (in SFP mode only).
132
133 NOTE: The Linux i40e driver supports the following flow types: IPv4, TCPv4, and
134 UDPv4. For a given flow type, it supports valid combinations of IP addresses
135 (source or destination) and UDP/TCP ports (source and destination). For
136 example, you can supply only a source IP address, a source IP address and a
137 destination port, or any combination of one or more of these four parameters.
138
139 NOTE: The Linux i40e driver allows you to filter traffic based on a
140 user-defined flexible two-byte pattern and offset by using the ethtool user-def
141 and mask fields. Only L3 and L4 flow types are supported for user-defined
142 flexible filters. For a given flow type, you must clear all Intel Ethernet Flow
143 Director filters before changing the input set (for that flow type).
144
145 To enable or disable the Intel Ethernet Flow Director::
146
147 # ethtool -K ethX ntuple <on|off>
148
149 When disabling ntuple filters, all the user programmed filters are flushed from
150 the driver cache and hardware. All needed filters must be re-added when ntuple
151 is re-enabled.
152
153 To add a filter that directs packet to queue 2, use -U or -N switch::
154
155 # ethtool -N ethX flow-type tcp4 src-ip 192.168.10.1 dst-ip \
156 192.168.10.2 src-port 2000 dst-port 2001 action 2 [loc 1]
157
158 To set a filter using only the source and destination IP address::
159
160 # ethtool -N ethX flow-type tcp4 src-ip 192.168.10.1 dst-ip \
161 192.168.10.2 action 2 [loc 1]
162
163 To see the list of filters currently present::
164
165 # ethtool <-u|-n> ethX
166
167 Application Targeted Routing (ATR) Perfect Filters
168 --------------------------------------------------
169 ATR is enabled by default when the kernel is in multiple transmit queue mode.
170 An ATR Intel Ethernet Flow Director filter rule is added when a TCP-IP flow
171 starts and is deleted when the flow ends. When a TCP-IP Intel Ethernet Flow
172 Director rule is added from ethtool (Sideband filter), ATR is turned off by the
173 driver. To re-enable ATR, the sideband can be disabled with the ethtool -K
174 option. For example::
175
176 ethtool -K [adapter] ntuple [off|on]
177
178 If sideband is re-enabled after ATR is re-enabled, ATR remains enabled until a
179 TCP-IP flow is added. When all TCP-IP sideband rules are deleted, ATR is
180 automatically re-enabled.
181
182 Packets that match the ATR rules are counted in fdir_atr_match stats in
183 ethtool, which also can be used to verify whether ATR rules still exist.
184
185 Sideband Perfect Filters
186 ------------------------
187 Sideband Perfect Filters are used to direct traffic that matches specified
188 characteristics. They are enabled through ethtool's ntuple interface. To add a
189 new filter use the following command::
190
191 ethtool -U <device> flow-type <type> src-ip <ip> dst-ip <ip> src-port <port> \
192 dst-port <port> action <queue>
193
194 Where:
195 <device> - the ethernet device to program
196 <type> - can be ip4, tcp4, udp4, or sctp4
197 <ip> - the ip address to match on
198 <port> - the port number to match on
199 <queue> - the queue to direct traffic towards (-1 discards matching traffic)
200
201 Use the following command to display all of the active filters::
202
203 ethtool -u <device>
204
205 Use the following command to delete a filter::
206
207 ethtool -U <device> delete <N>
208
209 Where <N> is the filter id displayed when printing all the active filters, and
210 may also have been specified using "loc <N>" when adding the filter.
211
212 The following example matches TCP traffic sent from 192.168.0.1, port 5300,
213 directed to 192.168.0.5, port 80, and sends it to queue 7::
214
215 ethtool -U enp130s0 flow-type tcp4 src-ip 192.168.0.1 dst-ip 192.168.0.5 \
216 src-port 5300 dst-port 80 action 7
217
218 For each flow-type, the programmed filters must all have the same matching
219 input set. For example, issuing the following two commands is acceptable::
220
221 ethtool -U enp130s0 flow-type ip4 src-ip 192.168.0.1 src-port 5300 action 7
222 ethtool -U enp130s0 flow-type ip4 src-ip 192.168.0.5 src-port 55 action 10
223
224 Issuing the next two commands, however, is not acceptable, since the first
225 specifies src-ip and the second specifies dst-ip::
226
227 ethtool -U enp130s0 flow-type ip4 src-ip 192.168.0.1 src-port 5300 action 7
228 ethtool -U enp130s0 flow-type ip4 dst-ip 192.168.0.5 src-port 55 action 10
229
230 The second command will fail with an error. You may program multiple filters
231 with the same fields, using different values, but, on one device, you may not
232 program two tcp4 filters with different matching fields.
233
234 Matching on a sub-portion of a field is not supported by the i40e driver, thus
235 partial mask fields are not supported.
236
237 The driver also supports matching user-defined data within the packet payload.
238 This flexible data is specified using the "user-def" field of the ethtool
239 command in the following way:
240
241 +----------------------------+--------------------------+
242 | 31 28 24 20 16 | 15 12 8 4 0 |
243 +----------------------------+--------------------------+
244 | offset into packet payload | 2 bytes of flexible data |
245 +----------------------------+--------------------------+
246
247 For example,
248
249 ::
250
251 ... user-def 0x4FFFF ...
252
253 tells the filter to look 4 bytes into the payload and match that value against
254 0xFFFF. The offset is based on the beginning of the payload, and not the
255 beginning of the packet. Thus
256
257 ::
258
259 flow-type tcp4 ... user-def 0x8BEAF ...
260
261 would match TCP/IPv4 packets which have the value 0xBEAF 8 bytes into the
262 TCP/IPv4 payload.
263
264 Note that ICMP headers are parsed as 4 bytes of header and 4 bytes of payload.
265 Thus to match the first byte of the payload, you must actually add 4 bytes to
266 the offset. Also note that ip4 filters match both ICMP frames as well as raw
267 (unknown) ip4 frames, where the payload will be the L3 payload of the IP4 frame.
268
269 The maximum offset is 64. The hardware will only read up to 64 bytes of data
270 from the payload. The offset must be even because the flexible data is 2 bytes
271 long and must be aligned to byte 0 of the packet payload.
272
273 The user-defined flexible offset is also considered part of the input set and
274 cannot be programmed separately for multiple filters of the same type. However,
275 the flexible data is not part of the input set and multiple filters may use the
276 same offset but match against different data.
277
278 To create filters that direct traffic to a specific Virtual Function, use the
279 "action" parameter. Specify the action as a 64 bit value, where the lower 32
280 bits represents the queue number, while the next 8 bits represent which VF.
281 Note that 0 is the PF, so the VF identifier is offset by 1. For example::
282
283 ... action 0x800000002 ...
284
285 specifies to direct traffic to Virtual Function 7 (8 minus 1) into queue 2 of
286 that VF.
287
288 Note that these filters will not break internal routing rules, and will not
289 route traffic that otherwise would not have been sent to the specified Virtual
290 Function.
291
292 Setting the link-down-on-close Private Flag
293 -------------------------------------------
294 When the link-down-on-close private flag is set to "on", the port's link will
295 go down when the interface is brought down using the ifconfig ethX down command.
296
297 Use ethtool to view and set link-down-on-close, as follows::
298
299 ethtool --show-priv-flags ethX
300 ethtool --set-priv-flags ethX link-down-on-close [on|off]
301
302 Setting the mdd-auto-reset-vf Private Flag
303 ------------------------------------------
304
305 When the mdd-auto-reset-vf private flag is set to "on", the problematic VF will
306 be automatically reset if a malformed descriptor is detected. If the flag is
307 set to "off", the problematic VF will be disabled.
308
309 Use ethtool to view and set mdd-auto-reset-vf, as follows::
310
311 ethtool --show-priv-flags ethX
312 ethtool --set-priv-flags ethX mdd-auto-reset-vf [on|off]
313
314 Viewing Link Messages
315 ---------------------
316 Link messages will not be displayed to the console if the distribution is
317 restricting system messages. In order to see network driver link messages on
318 your console, set dmesg to eight by entering the following::
319
320 dmesg -n 8
321
322 NOTE: This setting is not saved across reboots.
323
324 Jumbo Frames
325 ------------
326 Jumbo Frames support is enabled by changing the Maximum Transmission Unit (MTU)
327 to a value larger than the default value of 1500.
328
329 Use the ifconfig command to increase the MTU size. For example, enter the
330 following where <x> is the interface number::
331
332 ifconfig eth<x> mtu 9000 up
333
334 Alternatively, you can use the ip command as follows::
335
336 ip link set mtu 9000 dev eth<x>
337 ip link set up dev eth<x>
338
339 This setting is not saved across reboots. The setting change can be made
340 permanent by adding 'MTU=9000' to the file::
341
342 /etc/sysconfig/network-scripts/ifcfg-eth<x> // for RHEL
343 /etc/sysconfig/network/<config_file> // for SLES
344
345 NOTE: The maximum MTU setting for Jumbo Frames is 9702. This value coincides
346 with the maximum Jumbo Frames size of 9728 bytes.
347
348 NOTE: This driver will attempt to use multiple page sized buffers to receive
349 each jumbo packet. This should help to avoid buffer starvation issues when
350 allocating receive packets.
351
352 ethtool
353 -------
354 The driver utilizes the ethtool interface for driver configuration and
355 diagnostics, as well as displaying statistical information. The latest ethtool
356 version is required for this functionality. Download it at:
357 https://www.kernel.org/pub/software/network/ethtool/
358
359 Supported ethtool Commands and Options for Filtering
360 ----------------------------------------------------
361 -n --show-nfc
362 Retrieves the receive network flow classification configurations.
363
364 rx-flow-hash tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6
365 Retrieves the hash options for the specified network traffic type.
366
367 -N --config-nfc
368 Configures the receive network flow classification.
369
370 rx-flow-hash tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6 m|v|t|s|d|f|n|r...
371 Configures the hash options for the specified network traffic type.
372
373 udp4 UDP over IPv4
374 udp6 UDP over IPv6
375
376 f Hash on bytes 0 and 1 of the Layer 4 header of the Rx packet.
377 n Hash on bytes 2 and 3 of the Layer 4 header of the Rx packet.
378
379 Speed and Duplex Configuration
380 ------------------------------
381 In addressing speed and duplex configuration issues, you need to distinguish
382 between copper-based adapters and fiber-based adapters.
383
384 In the default mode, an Intel(R) Ethernet Network Adapter using copper
385 connections will attempt to auto-negotiate with its link partner to determine
386 the best setting. If the adapter cannot establish link with the link partner
387 using auto-negotiation, you may need to manually configure the adapter and link
388 partner to identical settings to establish link and pass packets. This should
389 only be needed when attempting to link with an older switch that does not
390 support auto-negotiation or one that has been forced to a specific speed or
391 duplex mode. Your link partner must match the setting you choose. 1 Gbps speeds
392 and higher cannot be forced. Use the autonegotiation advertising setting to
393 manually set devices for 1 Gbps and higher.
394
395 NOTE: You cannot set the speed for devices based on the Intel(R) Ethernet
396 Network Adapter XXV710 based devices.
397
398 Speed, duplex, and autonegotiation advertising are configured through the
399 ethtool utility.
400
401 Caution: Only experienced network administrators should force speed and duplex
402 or change autonegotiation advertising manually. The settings at the switch must
403 always match the adapter settings. Adapter performance may suffer or your
404 adapter may not operate if you configure the adapter differently from your
405 switch.
406
407 An Intel(R) Ethernet Network Adapter using fiber-based connections, however,
408 will not attempt to auto-negotiate with its link partner since those adapters
409 operate only in full duplex and only at their native speed.
410
411 NAPI
412 ----
413 NAPI (Rx polling mode) is supported in the i40e driver.
414
415 See :ref:`Documentation/networking/napi.rst <napi>` for more information.
416
417 Flow Control
418 ------------
419 Ethernet Flow Control (IEEE 802.3x) can be configured with ethtool to enable
420 receiving and transmitting pause frames for i40e. When transmit is enabled,
421 pause frames are generated when the receive packet buffer crosses a predefined
422 threshold. When receive is enabled, the transmit unit will halt for the time
423 delay specified when a pause frame is received.
424
425 NOTE: You must have a flow control capable link partner.
426
427 Flow Control is on by default.
428
429 Use ethtool to change the flow control settings.
430
431 To enable or disable Rx or Tx Flow Control::
432
433 ethtool -A eth? rx <on|off> tx <on|off>
434
435 Note: This command only enables or disables Flow Control if auto-negotiation is
436 disabled. If auto-negotiation is enabled, this command changes the parameters
437 used for auto-negotiation with the link partner.
438
439 To enable or disable auto-negotiation::
440
441 ethtool -s eth? autoneg <on|off>
442
443 Note: Flow Control auto-negotiation is part of link auto-negotiation. Depending
444 on your device, you may not be able to change the auto-negotiation setting.
445
446 RSS Hash Flow
447 -------------
448 Allows you to set the hash bytes per flow type and any combination of one or
449 more options for Receive Side Scaling (RSS) hash byte configuration.
450
451 ::
452
453 # ethtool -N <dev> rx-flow-hash <type> <option>
454
455 Where <type> is:
456 tcp4 signifying TCP over IPv4
457 udp4 signifying UDP over IPv4
458 tcp6 signifying TCP over IPv6
459 udp6 signifying UDP over IPv6
460 And <option> is one or more of:
461 s Hash on the IP source address of the Rx packet.
462 d Hash on the IP destination address of the Rx packet.
463 f Hash on bytes 0 and 1 of the Layer 4 header of the Rx packet.
464 n Hash on bytes 2 and 3 of the Layer 4 header of the Rx packet.
465
466 MAC and VLAN anti-spoofing feature
467 ----------------------------------
468 When a malicious driver attempts to send a spoofed packet, it is dropped by the
469 hardware and not transmitted.
470 NOTE: This feature can be disabled for a specific Virtual Function (VF)::
471
472 ip link set <pf dev> vf <vf id> spoofchk {off|on}
473
474 IEEE 1588 Precision Time Protocol (PTP) Hardware Clock (PHC)
475 ------------------------------------------------------------
476 Precision Time Protocol (PTP) is used to synchronize clocks in a computer
477 network. PTP support varies among Intel devices that support this driver. Use
478 "ethtool -T <netdev name>" to get a definitive list of PTP capabilities
479 supported by the device.
480
481 IEEE 802.1ad (QinQ) Support
482 ---------------------------
483 The IEEE 802.1ad standard, informally known as QinQ, allows for multiple VLAN
484 IDs within a single Ethernet frame. VLAN IDs are sometimes referred to as
485 "tags," and multiple VLAN IDs are thus referred to as a "tag stack." Tag stacks
486 allow L2 tunneling and the ability to segregate traffic within a particular
487 VLAN ID, among other uses.
488
489 The following are examples of how to configure 802.1ad (QinQ)::
490
491 ip link add link eth0 eth0.24 type vlan proto 802.1ad id 24
492 ip link add link eth0.24 eth0.24.371 type vlan proto 802.1Q id 371
493
494 Where "24" and "371" are example VLAN IDs.
495
496 NOTES:
497 Receive checksum offloads, cloud filters, and VLAN acceleration are not
498 supported for 802.1ad (QinQ) packets.
499
500 VXLAN and GENEVE Overlay HW Offloading
501 --------------------------------------
502 Virtual Extensible LAN (VXLAN) allows you to extend an L2 network over an L3
503 network, which may be useful in a virtualized or cloud environment. Some
504 Intel(R) Ethernet Network devices perform VXLAN processing, offloading it from
505 the operating system. This reduces CPU utilization.
506
507 VXLAN offloading is controlled by the Tx and Rx checksum offload options
508 provided by ethtool. That is, if Tx checksum offload is enabled, and the
509 adapter has the capability, VXLAN offloading is also enabled.
510
511 Support for VXLAN and GENEVE HW offloading is dependent on kernel support of
512 the HW offloading features.
513
514 Multiple Functions per Port
515 ---------------------------
516 Some adapters based on the Intel Ethernet Controller X710/XL710 support
517 multiple functions on a single physical port. Configure these functions through
518 the System Setup/BIOS.
519
520 Minimum TX Bandwidth is the guaranteed minimum data transmission bandwidth, as
521 a percentage of the full physical port link speed, that the partition will
522 receive. The bandwidth the partition is awarded will never fall below the level
523 you specify.
524
525 The range for the minimum bandwidth values is:
526 1 to ((100 minus # of partitions on the physical port) plus 1)
527 For example, if a physical port has 4 partitions, the range would be:
528 1 to ((100 - 4) + 1 = 97)
529
530 The Maximum Bandwidth percentage represents the maximum transmit bandwidth
531 allocated to the partition as a percentage of the full physical port link
532 speed. The accepted range of values is 1-100. The value is used as a limiter,
533 should you chose that any one particular function not be able to consume 100%
534 of a port's bandwidth (should it be available). The sum of all the values for
535 Maximum Bandwidth is not restricted, because no more than 100% of a port's
536 bandwidth can ever be used.
537
538 NOTE: X710/XXV710 devices fail to enable Max VFs (64) when Multiple Functions
539 per Port (MFP) and SR-IOV are enabled. An error from i40e is logged that says
540 "add vsi failed for VF N, aq_err 16". To workaround the issue, enable less than
541 64 virtual functions (VFs).
542
543 Data Center Bridging (DCB)
544 --------------------------
545 DCB is a configuration Quality of Service implementation in hardware. It uses
546 the VLAN priority tag (802.1p) to filter traffic. That means that there are 8
547 different priorities that traffic can be filtered into. It also enables
548 priority flow control (802.1Qbb) which can limit or eliminate the number of
549 dropped packets during network stress. Bandwidth can be allocated to each of
550 these priorities, which is enforced at the hardware level (802.1Qaz).
551
552 Adapter firmware implements LLDP and DCBX protocol agents as per 802.1AB and
553 802.1Qaz respectively. The firmware based DCBX agent runs in willing mode only
554 and can accept settings from a DCBX capable peer. Software configuration of
555 DCBX parameters via dcbtool/lldptool are not supported.
556
557 NOTE: Firmware LLDP can be disabled by setting the private flag disable-fw-lldp.
558
559 The i40e driver implements the DCB netlink interface layer to allow user-space
560 to communicate with the driver and query DCB configuration for the port.
561
562 NOTE:
563 The kernel assumes that TC0 is available, and will disable Priority Flow
564 Control (PFC) on the device if TC0 is not available. To fix this, ensure TC0 is
565 enabled when setting up DCB on your switch.
566
567 Interrupt Rate Limiting
568 -----------------------
569 :Valid Range: 0-235 (0=no limit)
570
571 The Intel(R) Ethernet Controller XL710 family supports an interrupt rate
572 limiting mechanism. The user can control, via ethtool, the number of
573 microseconds between interrupts.
574
575 Syntax::
576
577 # ethtool -C ethX rx-usecs-high N
578
579 The range of 0-235 microseconds provides an effective range of 4,310 to 250,000
580 interrupts per second. The value of rx-usecs-high can be set independently of
581 rx-usecs and tx-usecs in the same ethtool command, and is also independent of
582 the adaptive interrupt moderation algorithm. The underlying hardware supports
583 granularity in 4-microsecond intervals, so adjacent values may result in the
584 same interrupt rate.
585
586 One possible use case is the following::
587
588 # ethtool -C ethX adaptive-rx off adaptive-tx off rx-usecs-high 20 rx-usecs \
589 5 tx-usecs 5
590
591 The above command would disable adaptive interrupt moderation, and allow a
592 maximum of 5 microseconds before indicating a receive or transmit was complete.
593 However, instead of resulting in as many as 200,000 interrupts per second, it
594 limits total interrupts per second to 50,000 via the rx-usecs-high parameter.
595
596 Performance Optimization
597 ========================
598 Driver defaults are meant to fit a wide variety of workloads, but if further
599 optimization is required we recommend experimenting with the following settings.
600
601 NOTE: For better performance when processing small (64B) frame sizes, try
602 enabling Hyper threading in the BIOS in order to increase the number of logical
603 cores in the system and subsequently increase the number of queues available to
604 the adapter.
605
606 Virtualized Environments
607 ------------------------
608 1. Disable XPS on both ends by using the included virt_perf_default script
609 or by running the following command as root::
610
611 for file in `ls /sys/class/net/<ethX>/queues/tx-*/xps_cpus`;
612 do echo 0 > $file; done
613
614 2. Using the appropriate mechanism (vcpupin) in the vm, pin the cpu's to
615 individual lcpu's, making sure to use a set of cpu's included in the
616 device's local_cpulist: /sys/class/net/<ethX>/device/local_cpulist.
617
618 3. Configure as many Rx/Tx queues in the VM as available. Do not rely on
619 the default setting of 1.
620
621
622 Non-virtualized Environments
623 ----------------------------
624 Pin the adapter's IRQs to specific cores by disabling the irqbalance service
625 and using the included set_irq_affinity script. Please see the script's help
626 text for further options.
627
628 - The following settings will distribute the IRQs across all the cores evenly::
629
630 # scripts/set_irq_affinity -x all <interface1> , [ <interface2>, ... ]
631
632 - The following settings will distribute the IRQs across all the cores that are
633 local to the adapter (same NUMA node)::
634
635 # scripts/set_irq_affinity -x local <interface1> ,[ <interface2>, ... ]
636
637 For very CPU intensive workloads, we recommend pinning the IRQs to all cores.
638
639 For IP Forwarding: Disable Adaptive ITR and lower Rx and Tx interrupts per
640 queue using ethtool.
641
642 - Setting rx-usecs and tx-usecs to 125 will limit interrupts to about 8000
643 interrupts per second per queue.
644
645 ::
646
647 # ethtool -C <interface> adaptive-rx off adaptive-tx off rx-usecs 125 \
648 tx-usecs 125
649
650 For lower CPU utilization: Disable Adaptive ITR and lower Rx and Tx interrupts
651 per queue using ethtool.
652
653 - Setting rx-usecs and tx-usecs to 250 will limit interrupts to about 4000
654 interrupts per second per queue.
655
656 ::
657
658 # ethtool -C <interface> adaptive-rx off adaptive-tx off rx-usecs 250 \
659 tx-usecs 250
660
661 For lower latency: Disable Adaptive ITR and ITR by setting Rx and Tx to 0 using
662 ethtool.
663
664 ::
665
666 # ethtool -C <interface> adaptive-rx off adaptive-tx off rx-usecs 0 \
667 tx-usecs 0
668
669 Application Device Queues (ADq)
670 -------------------------------
671 Application Device Queues (ADq) allows you to dedicate one or more queues to a
672 specific application. This can reduce latency for the specified application,
673 and allow Tx traffic to be rate limited per application. Follow the steps below
674 to set ADq.
675
676 1. Create traffic classes (TCs). Maximum of 8 TCs can be created per interface.
677 The shaper bw_rlimit parameter is optional.
678
679 Example: Sets up two tcs, tc0 and tc1, with 16 queues each and max tx rate set
680 to 1Gbit for tc0 and 3Gbit for tc1.
681
682 ::
683
684 # tc qdisc add dev <interface> root mqprio num_tc 2 map 0 0 0 0 1 1 1 1
685 queues 16@0 16@16 hw 1 mode channel shaper bw_rlimit min_rate 1Gbit 2Gbit
686 max_rate 1Gbit 3Gbit
687
688 map: priority mapping for up to 16 priorities to tcs (e.g. map 0 0 0 0 1 1 1 1
689 sets priorities 0-3 to use tc0 and 4-7 to use tc1)
690
691 queues: for each tc, <num queues>@<offset> (e.g. queues 16@0 16@16 assigns
692 16 queues to tc0 at offset 0 and 16 queues to tc1 at offset 16. Max total
693 number of queues for all tcs is 64 or number of cores, whichever is lower.)
694
695 hw 1 mode channel: ‘channel’ with ‘hw’ set to 1 is a new new hardware
696 offload mode in mqprio that makes full use of the mqprio options, the
697 TCs, the queue configurations, and the QoS parameters.
698
699 shaper bw_rlimit: for each tc, sets minimum and maximum bandwidth rates.
700 Totals must be equal or less than port speed.
701
702 For example: min_rate 1Gbit 3Gbit: Verify bandwidth limit using network
703 monitoring tools such as `ifstat` or `sar -n DEV [interval] [number of samples]`
704
705 2. Enable HW TC offload on interface::
706
707 # ethtool -K <interface> hw-tc-offload on
708
709 3. Apply TCs to ingress (RX) flow of interface::
710
711 # tc qdisc add dev <interface> ingress
712
713 NOTES:
714 - Run all tc commands from the iproute2 <pathtoiproute2>/tc/ directory.
715 - ADq is not compatible with cloud filters.
716 - Setting up channels via ethtool (ethtool -L) is not supported when the
717 TCs are configured using mqprio.
718 - You must have iproute2 latest version
719 - NVM version 6.01 or later is required.
720 - ADq cannot be enabled when any the following features are enabled: Data
721 Center Bridging (DCB), Multiple Functions per Port (MFP), or Sideband
722 Filters.
723 - If another driver (for example, DPDK) has set cloud filters, you cannot
724 enable ADq.
725 - Tunnel filters are not supported in ADq. If encapsulated packets do
726 arrive in non-tunnel mode, filtering will be done on the inner headers.
727 For example, for VXLAN traffic in non-tunnel mode, PCTYPE is identified
728 as a VXLAN encapsulated packet, outer headers are ignored. Therefore,
729 inner headers are matched.
730 - If a TC filter on a PF matches traffic over a VF (on the PF), that
731 traffic will be routed to the appropriate queue of the PF, and will
732 not be passed on the VF. Such traffic will end up getting dropped higher
733 up in the TCP/IP stack as it does not match PF address data.
734 - If traffic matches multiple TC filters that point to different TCs,
735 that traffic will be duplicated and sent to all matching TC queues.
736 The hardware switch mirrors the packet to a VSI list when multiple
737 filters are matched.
738
739
740 Known Issues/Troubleshooting
741 ============================
742
743 NOTE: 1 Gb devices based on the Intel(R) Ethernet Network Connection X722 do
744 not support the following features:
745
746 * Data Center Bridging (DCB)
747 * QOS
748 * VMQ
749 * SR-IOV
750 * Task Encapsulation offload (VXLAN, NVGRE)
751 * Energy Efficient Ethernet (EEE)
752 * Auto-media detect
753
754 Unexpected Issues when the device driver and DPDK share a device
755 ----------------------------------------------------------------
756 Unexpected issues may result when an i40e device is in multi driver mode and
757 the kernel driver and DPDK driver are sharing the device. This is because
758 access to the global NIC resources is not synchronized between multiple
759 drivers. Any change to the global NIC configuration (writing to a global
760 register, setting global configuration by AQ, or changing switch modes) will
761 affect all ports and drivers on the device. Loading DPDK with the
762 "multi-driver" module parameter may mitigate some of the issues.
763
764 TC0 must be enabled when setting up DCB on a switch
765 ---------------------------------------------------
766 The kernel assumes that TC0 is available, and will disable Priority Flow
767 Control (PFC) on the device if TC0 is not available. To fix this, ensure TC0 is
768 enabled when setting up DCB on your switch.
769
770
771 Support
772 =======
773 For general information, go to the Intel support website at:
774 https://www.intel.com/support/
775
776 If an issue is identified with the released source code on a supported kernel
777 with a supported adapter, email the specific information related to the issue
778 to [email protected].