| 1 |
.. SPDX-License-Identifier: GPL-2.0+ |
| 2 |
|
| 3 |
================================================================= |
| 4 |
Linux Base Driver for the Intel(R) Ethernet Controller 700 Series |
| 5 |
================================================================= |
| 6 |
|
| 7 |
Intel 40 Gigabit Linux driver. |
| 8 |
Copyright(c) 1999-2018 Intel Corporation. |
| 9 |
|
| 10 |
Contents |
| 11 |
======== |
| 12 |
|
| 13 |
- Overview |
| 14 |
- Identifying Your Adapter |
| 15 |
- Intel(R) Ethernet Flow Director |
| 16 |
- Additional Configurations |
| 17 |
- Known Issues |
| 18 |
- Support |
| 19 |
|
| 20 |
|
| 21 |
Driver information can be obtained using ethtool, lspci, and ifconfig. |
| 22 |
Instructions on updating ethtool can be found in the section Additional |
| 23 |
Configurations later in this document. |
| 24 |
|
| 25 |
For questions related to hardware requirements, refer to the documentation |
| 26 |
supplied with your Intel adapter. All hardware requirements listed apply to use |
| 27 |
with Linux. |
| 28 |
|
| 29 |
|
| 30 |
Identifying Your Adapter |
| 31 |
======================== |
| 32 |
The driver is compatible with devices based on the following: |
| 33 |
|
| 34 |
* Intel(R) Ethernet Controller X710 |
| 35 |
* Intel(R) Ethernet Controller XL710 |
| 36 |
* Intel(R) Ethernet Network Connection X722 |
| 37 |
* Intel(R) Ethernet Controller XXV710 |
| 38 |
|
| 39 |
For the best performance, make sure the latest NVM/FW is installed on your |
| 40 |
device. |
| 41 |
|
| 42 |
For information on how to identify your adapter, and for the latest NVM/FW |
| 43 |
images and Intel network drivers, refer to the Intel Support website: |
| 44 |
https://www.intel.com/support |
| 45 |
|
| 46 |
SFP+ and QSFP+ Devices |
| 47 |
---------------------- |
| 48 |
For information about supported media, refer to this document: |
| 49 |
https://www.intel.com/content/dam/www/public/us/en/documents/release-notes/xl710-ethernet-controller-feature-matrix.pdf |
| 50 |
|
| 51 |
NOTE: Some adapters based on the Intel(R) Ethernet Controller 700 Series only |
| 52 |
support Intel Ethernet Optics modules. On these adapters, other modules are not |
| 53 |
supported and will not function. In all cases Intel recommends using Intel |
| 54 |
Ethernet Optics; other modules may function but are not validated by Intel. |
| 55 |
Contact Intel for supported media types. |
| 56 |
|
| 57 |
NOTE: For connections based on Intel(R) Ethernet Controller 700 Series, support |
| 58 |
is dependent on your system board. Please see your vendor for details. |
| 59 |
|
| 60 |
NOTE: In systems that do not have adequate airflow to cool the adapter and |
| 61 |
optical modules, you must use high temperature optical modules. |
| 62 |
|
| 63 |
Virtual Functions (VFs) |
| 64 |
----------------------- |
| 65 |
Use sysfs to enable VFs. For example:: |
| 66 |
|
| 67 |
#echo $num_vf_enabled > /sys/class/net/$dev/device/sriov_numvfs #enable VFs |
| 68 |
#echo 0 > /sys/class/net/$dev/device/sriov_numvfs #disable VFs |
| 69 |
|
| 70 |
For example, the following instructions will configure PF eth0 and the first VF |
| 71 |
on VLAN 10:: |
| 72 |
|
| 73 |
$ ip link set dev eth0 vf 0 vlan 10 |
| 74 |
|
| 75 |
VLAN Tag Packet Steering |
| 76 |
------------------------ |
| 77 |
Allows you to send all packets with a specific VLAN tag to a particular SR-IOV |
| 78 |
virtual function (VF). Further, this feature allows you to designate a |
| 79 |
particular VF as trusted, and allows that trusted VF to request selective |
| 80 |
promiscuous mode on the Physical Function (PF). |
| 81 |
|
| 82 |
To set a VF as trusted or untrusted, enter the following command in the |
| 83 |
Hypervisor:: |
| 84 |
|
| 85 |
# ip link set dev eth0 vf 1 trust [on|off] |
| 86 |
|
| 87 |
Once the VF is designated as trusted, use the following commands in the VM to |
| 88 |
set the VF to promiscuous mode. |
| 89 |
|
| 90 |
:: |
| 91 |
|
| 92 |
For promiscuous all: |
| 93 |
#ip link set eth2 promisc on |
| 94 |
Where eth2 is a VF interface in the VM |
| 95 |
|
| 96 |
For promiscuous Multicast: |
| 97 |
#ip link set eth2 allmulticast on |
| 98 |
Where eth2 is a VF interface in the VM |
| 99 |
|
| 100 |
NOTE: By default, the ethtool priv-flag vf-true-promisc-support is set to |
| 101 |
"off",meaning that promiscuous mode for the VF will be limited. To set the |
| 102 |
promiscuous mode for the VF to true promiscuous and allow the VF to see all |
| 103 |
ingress traffic, use the following command:: |
| 104 |
|
| 105 |
#ethtool -set-priv-flags p261p1 vf-true-promisc-support on |
| 106 |
|
| 107 |
The vf-true-promisc-support priv-flag does not enable promiscuous mode; rather, |
| 108 |
it designates which type of promiscuous mode (limited or true) you will get |
| 109 |
when you enable promiscuous mode using the ip link commands above. Note that |
| 110 |
this is a global setting that affects the entire device. However,the |
| 111 |
vf-true-promisc-support priv-flag is only exposed to the first PF of the |
| 112 |
device. The PF remains in limited promiscuous mode (unless it is in MFP mode) |
| 113 |
regardless of the vf-true-promisc-support setting. |
| 114 |
|
| 115 |
Now add a VLAN interface on the VF interface:: |
| 116 |
|
| 117 |
#ip link add link eth2 name eth2.100 type vlan id 100 |
| 118 |
|
| 119 |
Note that the order in which you set the VF to promiscuous mode and add the |
| 120 |
VLAN interface does not matter (you can do either first). The end result in |
| 121 |
this example is that the VF will get all traffic that is tagged with VLAN 100. |
| 122 |
|
| 123 |
Intel(R) Ethernet Flow Director |
| 124 |
------------------------------- |
| 125 |
The Intel Ethernet Flow Director performs the following tasks: |
| 126 |
|
| 127 |
- Directs receive packets according to their flows to different queues. |
| 128 |
- Enables tight control on routing a flow in the platform. |
| 129 |
- Matches flows and CPU cores for flow affinity. |
| 130 |
- Supports multiple parameters for flexible flow classification and load |
| 131 |
balancing (in SFP mode only). |
| 132 |
|
| 133 |
NOTE: The Linux i40e driver supports the following flow types: IPv4, TCPv4, and |
| 134 |
UDPv4. For a given flow type, it supports valid combinations of IP addresses |
| 135 |
(source or destination) and UDP/TCP ports (source and destination). For |
| 136 |
example, you can supply only a source IP address, a source IP address and a |
| 137 |
destination port, or any combination of one or more of these four parameters. |
| 138 |
|
| 139 |
NOTE: The Linux i40e driver allows you to filter traffic based on a |
| 140 |
user-defined flexible two-byte pattern and offset by using the ethtool user-def |
| 141 |
and mask fields. Only L3 and L4 flow types are supported for user-defined |
| 142 |
flexible filters. For a given flow type, you must clear all Intel Ethernet Flow |
| 143 |
Director filters before changing the input set (for that flow type). |
| 144 |
|
| 145 |
To enable or disable the Intel Ethernet Flow Director:: |
| 146 |
|
| 147 |
# ethtool -K ethX ntuple <on|off> |
| 148 |
|
| 149 |
When disabling ntuple filters, all the user programmed filters are flushed from |
| 150 |
the driver cache and hardware. All needed filters must be re-added when ntuple |
| 151 |
is re-enabled. |
| 152 |
|
| 153 |
To add a filter that directs packet to queue 2, use -U or -N switch:: |
| 154 |
|
| 155 |
# ethtool -N ethX flow-type tcp4 src-ip 192.168.10.1 dst-ip \ |
| 156 |
192.168.10.2 src-port 2000 dst-port 2001 action 2 [loc 1] |
| 157 |
|
| 158 |
To set a filter using only the source and destination IP address:: |
| 159 |
|
| 160 |
# ethtool -N ethX flow-type tcp4 src-ip 192.168.10.1 dst-ip \ |
| 161 |
192.168.10.2 action 2 [loc 1] |
| 162 |
|
| 163 |
To see the list of filters currently present:: |
| 164 |
|
| 165 |
# ethtool <-u|-n> ethX |
| 166 |
|
| 167 |
Application Targeted Routing (ATR) Perfect Filters |
| 168 |
-------------------------------------------------- |
| 169 |
ATR is enabled by default when the kernel is in multiple transmit queue mode. |
| 170 |
An ATR Intel Ethernet Flow Director filter rule is added when a TCP-IP flow |
| 171 |
starts and is deleted when the flow ends. When a TCP-IP Intel Ethernet Flow |
| 172 |
Director rule is added from ethtool (Sideband filter), ATR is turned off by the |
| 173 |
driver. To re-enable ATR, the sideband can be disabled with the ethtool -K |
| 174 |
option. For example:: |
| 175 |
|
| 176 |
ethtool -K [adapter] ntuple [off|on] |
| 177 |
|
| 178 |
If sideband is re-enabled after ATR is re-enabled, ATR remains enabled until a |
| 179 |
TCP-IP flow is added. When all TCP-IP sideband rules are deleted, ATR is |
| 180 |
automatically re-enabled. |
| 181 |
|
| 182 |
Packets that match the ATR rules are counted in fdir_atr_match stats in |
| 183 |
ethtool, which also can be used to verify whether ATR rules still exist. |
| 184 |
|
| 185 |
Sideband Perfect Filters |
| 186 |
------------------------ |
| 187 |
Sideband Perfect Filters are used to direct traffic that matches specified |
| 188 |
characteristics. They are enabled through ethtool's ntuple interface. To add a |
| 189 |
new filter use the following command:: |
| 190 |
|
| 191 |
ethtool -U <device> flow-type <type> src-ip <ip> dst-ip <ip> src-port <port> \ |
| 192 |
dst-port <port> action <queue> |
| 193 |
|
| 194 |
Where: |
| 195 |
<device> - the ethernet device to program |
| 196 |
<type> - can be ip4, tcp4, udp4, or sctp4 |
| 197 |
<ip> - the ip address to match on |
| 198 |
<port> - the port number to match on |
| 199 |
<queue> - the queue to direct traffic towards (-1 discards matching traffic) |
| 200 |
|
| 201 |
Use the following command to display all of the active filters:: |
| 202 |
|
| 203 |
ethtool -u <device> |
| 204 |
|
| 205 |
Use the following command to delete a filter:: |
| 206 |
|
| 207 |
ethtool -U <device> delete <N> |
| 208 |
|
| 209 |
Where <N> is the filter id displayed when printing all the active filters, and |
| 210 |
may also have been specified using "loc <N>" when adding the filter. |
| 211 |
|
| 212 |
The following example matches TCP traffic sent from 192.168.0.1, port 5300, |
| 213 |
directed to 192.168.0.5, port 80, and sends it to queue 7:: |
| 214 |
|
| 215 |
ethtool -U enp130s0 flow-type tcp4 src-ip 192.168.0.1 dst-ip 192.168.0.5 \ |
| 216 |
src-port 5300 dst-port 80 action 7 |
| 217 |
|
| 218 |
For each flow-type, the programmed filters must all have the same matching |
| 219 |
input set. For example, issuing the following two commands is acceptable:: |
| 220 |
|
| 221 |
ethtool -U enp130s0 flow-type ip4 src-ip 192.168.0.1 src-port 5300 action 7 |
| 222 |
ethtool -U enp130s0 flow-type ip4 src-ip 192.168.0.5 src-port 55 action 10 |
| 223 |
|
| 224 |
Issuing the next two commands, however, is not acceptable, since the first |
| 225 |
specifies src-ip and the second specifies dst-ip:: |
| 226 |
|
| 227 |
ethtool -U enp130s0 flow-type ip4 src-ip 192.168.0.1 src-port 5300 action 7 |
| 228 |
ethtool -U enp130s0 flow-type ip4 dst-ip 192.168.0.5 src-port 55 action 10 |
| 229 |
|
| 230 |
The second command will fail with an error. You may program multiple filters |
| 231 |
with the same fields, using different values, but, on one device, you may not |
| 232 |
program two tcp4 filters with different matching fields. |
| 233 |
|
| 234 |
Matching on a sub-portion of a field is not supported by the i40e driver, thus |
| 235 |
partial mask fields are not supported. |
| 236 |
|
| 237 |
The driver also supports matching user-defined data within the packet payload. |
| 238 |
This flexible data is specified using the "user-def" field of the ethtool |
| 239 |
command in the following way: |
| 240 |
|
| 241 |
+----------------------------+--------------------------+ |
| 242 |
| 31 28 24 20 16 | 15 12 8 4 0 | |
| 243 |
+----------------------------+--------------------------+ |
| 244 |
| offset into packet payload | 2 bytes of flexible data | |
| 245 |
+----------------------------+--------------------------+ |
| 246 |
|
| 247 |
For example, |
| 248 |
|
| 249 |
:: |
| 250 |
|
| 251 |
... user-def 0x4FFFF ... |
| 252 |
|
| 253 |
tells the filter to look 4 bytes into the payload and match that value against |
| 254 |
0xFFFF. The offset is based on the beginning of the payload, and not the |
| 255 |
beginning of the packet. Thus |
| 256 |
|
| 257 |
:: |
| 258 |
|
| 259 |
flow-type tcp4 ... user-def 0x8BEAF ... |
| 260 |
|
| 261 |
would match TCP/IPv4 packets which have the value 0xBEAF 8 bytes into the |
| 262 |
TCP/IPv4 payload. |
| 263 |
|
| 264 |
Note that ICMP headers are parsed as 4 bytes of header and 4 bytes of payload. |
| 265 |
Thus to match the first byte of the payload, you must actually add 4 bytes to |
| 266 |
the offset. Also note that ip4 filters match both ICMP frames as well as raw |
| 267 |
(unknown) ip4 frames, where the payload will be the L3 payload of the IP4 frame. |
| 268 |
|
| 269 |
The maximum offset is 64. The hardware will only read up to 64 bytes of data |
| 270 |
from the payload. The offset must be even because the flexible data is 2 bytes |
| 271 |
long and must be aligned to byte 0 of the packet payload. |
| 272 |
|
| 273 |
The user-defined flexible offset is also considered part of the input set and |
| 274 |
cannot be programmed separately for multiple filters of the same type. However, |
| 275 |
the flexible data is not part of the input set and multiple filters may use the |
| 276 |
same offset but match against different data. |
| 277 |
|
| 278 |
To create filters that direct traffic to a specific Virtual Function, use the |
| 279 |
"action" parameter. Specify the action as a 64 bit value, where the lower 32 |
| 280 |
bits represents the queue number, while the next 8 bits represent which VF. |
| 281 |
Note that 0 is the PF, so the VF identifier is offset by 1. For example:: |
| 282 |
|
| 283 |
... action 0x800000002 ... |
| 284 |
|
| 285 |
specifies to direct traffic to Virtual Function 7 (8 minus 1) into queue 2 of |
| 286 |
that VF. |
| 287 |
|
| 288 |
Note that these filters will not break internal routing rules, and will not |
| 289 |
route traffic that otherwise would not have been sent to the specified Virtual |
| 290 |
Function. |
| 291 |
|
| 292 |
Setting the link-down-on-close Private Flag |
| 293 |
------------------------------------------- |
| 294 |
When the link-down-on-close private flag is set to "on", the port's link will |
| 295 |
go down when the interface is brought down using the ifconfig ethX down command. |
| 296 |
|
| 297 |
Use ethtool to view and set link-down-on-close, as follows:: |
| 298 |
|
| 299 |
ethtool --show-priv-flags ethX |
| 300 |
ethtool --set-priv-flags ethX link-down-on-close [on|off] |
| 301 |
|
| 302 |
Setting the mdd-auto-reset-vf Private Flag |
| 303 |
------------------------------------------ |
| 304 |
|
| 305 |
When the mdd-auto-reset-vf private flag is set to "on", the problematic VF will |
| 306 |
be automatically reset if a malformed descriptor is detected. If the flag is |
| 307 |
set to "off", the problematic VF will be disabled. |
| 308 |
|
| 309 |
Use ethtool to view and set mdd-auto-reset-vf, as follows:: |
| 310 |
|
| 311 |
ethtool --show-priv-flags ethX |
| 312 |
ethtool --set-priv-flags ethX mdd-auto-reset-vf [on|off] |
| 313 |
|
| 314 |
Viewing Link Messages |
| 315 |
--------------------- |
| 316 |
Link messages will not be displayed to the console if the distribution is |
| 317 |
restricting system messages. In order to see network driver link messages on |
| 318 |
your console, set dmesg to eight by entering the following:: |
| 319 |
|
| 320 |
dmesg -n 8 |
| 321 |
|
| 322 |
NOTE: This setting is not saved across reboots. |
| 323 |
|
| 324 |
Jumbo Frames |
| 325 |
------------ |
| 326 |
Jumbo Frames support is enabled by changing the Maximum Transmission Unit (MTU) |
| 327 |
to a value larger than the default value of 1500. |
| 328 |
|
| 329 |
Use the ifconfig command to increase the MTU size. For example, enter the |
| 330 |
following where <x> is the interface number:: |
| 331 |
|
| 332 |
ifconfig eth<x> mtu 9000 up |
| 333 |
|
| 334 |
Alternatively, you can use the ip command as follows:: |
| 335 |
|
| 336 |
ip link set mtu 9000 dev eth<x> |
| 337 |
ip link set up dev eth<x> |
| 338 |
|
| 339 |
This setting is not saved across reboots. The setting change can be made |
| 340 |
permanent by adding 'MTU=9000' to the file:: |
| 341 |
|
| 342 |
/etc/sysconfig/network-scripts/ifcfg-eth<x> // for RHEL |
| 343 |
/etc/sysconfig/network/<config_file> // for SLES |
| 344 |
|
| 345 |
NOTE: The maximum MTU setting for Jumbo Frames is 9702. This value coincides |
| 346 |
with the maximum Jumbo Frames size of 9728 bytes. |
| 347 |
|
| 348 |
NOTE: This driver will attempt to use multiple page sized buffers to receive |
| 349 |
each jumbo packet. This should help to avoid buffer starvation issues when |
| 350 |
allocating receive packets. |
| 351 |
|
| 352 |
ethtool |
| 353 |
------- |
| 354 |
The driver utilizes the ethtool interface for driver configuration and |
| 355 |
diagnostics, as well as displaying statistical information. The latest ethtool |
| 356 |
version is required for this functionality. Download it at: |
| 357 |
https://www.kernel.org/pub/software/network/ethtool/ |
| 358 |
|
| 359 |
Supported ethtool Commands and Options for Filtering |
| 360 |
---------------------------------------------------- |
| 361 |
-n --show-nfc |
| 362 |
Retrieves the receive network flow classification configurations. |
| 363 |
|
| 364 |
rx-flow-hash tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6 |
| 365 |
Retrieves the hash options for the specified network traffic type. |
| 366 |
|
| 367 |
-N --config-nfc |
| 368 |
Configures the receive network flow classification. |
| 369 |
|
| 370 |
rx-flow-hash tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6 m|v|t|s|d|f|n|r... |
| 371 |
Configures the hash options for the specified network traffic type. |
| 372 |
|
| 373 |
udp4 UDP over IPv4 |
| 374 |
udp6 UDP over IPv6 |
| 375 |
|
| 376 |
f Hash on bytes 0 and 1 of the Layer 4 header of the Rx packet. |
| 377 |
n Hash on bytes 2 and 3 of the Layer 4 header of the Rx packet. |
| 378 |
|
| 379 |
Speed and Duplex Configuration |
| 380 |
------------------------------ |
| 381 |
In addressing speed and duplex configuration issues, you need to distinguish |
| 382 |
between copper-based adapters and fiber-based adapters. |
| 383 |
|
| 384 |
In the default mode, an Intel(R) Ethernet Network Adapter using copper |
| 385 |
connections will attempt to auto-negotiate with its link partner to determine |
| 386 |
the best setting. If the adapter cannot establish link with the link partner |
| 387 |
using auto-negotiation, you may need to manually configure the adapter and link |
| 388 |
partner to identical settings to establish link and pass packets. This should |
| 389 |
only be needed when attempting to link with an older switch that does not |
| 390 |
support auto-negotiation or one that has been forced to a specific speed or |
| 391 |
duplex mode. Your link partner must match the setting you choose. 1 Gbps speeds |
| 392 |
and higher cannot be forced. Use the autonegotiation advertising setting to |
| 393 |
manually set devices for 1 Gbps and higher. |
| 394 |
|
| 395 |
NOTE: You cannot set the speed for devices based on the Intel(R) Ethernet |
| 396 |
Network Adapter XXV710 based devices. |
| 397 |
|
| 398 |
Speed, duplex, and autonegotiation advertising are configured through the |
| 399 |
ethtool utility. |
| 400 |
|
| 401 |
Caution: Only experienced network administrators should force speed and duplex |
| 402 |
or change autonegotiation advertising manually. The settings at the switch must |
| 403 |
always match the adapter settings. Adapter performance may suffer or your |
| 404 |
adapter may not operate if you configure the adapter differently from your |
| 405 |
switch. |
| 406 |
|
| 407 |
An Intel(R) Ethernet Network Adapter using fiber-based connections, however, |
| 408 |
will not attempt to auto-negotiate with its link partner since those adapters |
| 409 |
operate only in full duplex and only at their native speed. |
| 410 |
|
| 411 |
NAPI |
| 412 |
---- |
| 413 |
NAPI (Rx polling mode) is supported in the i40e driver. |
| 414 |
|
| 415 |
See :ref:`Documentation/networking/napi.rst <napi>` for more information. |
| 416 |
|
| 417 |
Flow Control |
| 418 |
------------ |
| 419 |
Ethernet Flow Control (IEEE 802.3x) can be configured with ethtool to enable |
| 420 |
receiving and transmitting pause frames for i40e. When transmit is enabled, |
| 421 |
pause frames are generated when the receive packet buffer crosses a predefined |
| 422 |
threshold. When receive is enabled, the transmit unit will halt for the time |
| 423 |
delay specified when a pause frame is received. |
| 424 |
|
| 425 |
NOTE: You must have a flow control capable link partner. |
| 426 |
|
| 427 |
Flow Control is on by default. |
| 428 |
|
| 429 |
Use ethtool to change the flow control settings. |
| 430 |
|
| 431 |
To enable or disable Rx or Tx Flow Control:: |
| 432 |
|
| 433 |
ethtool -A eth? rx <on|off> tx <on|off> |
| 434 |
|
| 435 |
Note: This command only enables or disables Flow Control if auto-negotiation is |
| 436 |
disabled. If auto-negotiation is enabled, this command changes the parameters |
| 437 |
used for auto-negotiation with the link partner. |
| 438 |
|
| 439 |
To enable or disable auto-negotiation:: |
| 440 |
|
| 441 |
ethtool -s eth? autoneg <on|off> |
| 442 |
|
| 443 |
Note: Flow Control auto-negotiation is part of link auto-negotiation. Depending |
| 444 |
on your device, you may not be able to change the auto-negotiation setting. |
| 445 |
|
| 446 |
RSS Hash Flow |
| 447 |
------------- |
| 448 |
Allows you to set the hash bytes per flow type and any combination of one or |
| 449 |
more options for Receive Side Scaling (RSS) hash byte configuration. |
| 450 |
|
| 451 |
:: |
| 452 |
|
| 453 |
# ethtool -N <dev> rx-flow-hash <type> <option> |
| 454 |
|
| 455 |
Where <type> is: |
| 456 |
tcp4 signifying TCP over IPv4 |
| 457 |
udp4 signifying UDP over IPv4 |
| 458 |
tcp6 signifying TCP over IPv6 |
| 459 |
udp6 signifying UDP over IPv6 |
| 460 |
And <option> is one or more of: |
| 461 |
s Hash on the IP source address of the Rx packet. |
| 462 |
d Hash on the IP destination address of the Rx packet. |
| 463 |
f Hash on bytes 0 and 1 of the Layer 4 header of the Rx packet. |
| 464 |
n Hash on bytes 2 and 3 of the Layer 4 header of the Rx packet. |
| 465 |
|
| 466 |
MAC and VLAN anti-spoofing feature |
| 467 |
---------------------------------- |
| 468 |
When a malicious driver attempts to send a spoofed packet, it is dropped by the |
| 469 |
hardware and not transmitted. |
| 470 |
NOTE: This feature can be disabled for a specific Virtual Function (VF):: |
| 471 |
|
| 472 |
ip link set <pf dev> vf <vf id> spoofchk {off|on} |
| 473 |
|
| 474 |
IEEE 1588 Precision Time Protocol (PTP) Hardware Clock (PHC) |
| 475 |
------------------------------------------------------------ |
| 476 |
Precision Time Protocol (PTP) is used to synchronize clocks in a computer |
| 477 |
network. PTP support varies among Intel devices that support this driver. Use |
| 478 |
"ethtool -T <netdev name>" to get a definitive list of PTP capabilities |
| 479 |
supported by the device. |
| 480 |
|
| 481 |
IEEE 802.1ad (QinQ) Support |
| 482 |
--------------------------- |
| 483 |
The IEEE 802.1ad standard, informally known as QinQ, allows for multiple VLAN |
| 484 |
IDs within a single Ethernet frame. VLAN IDs are sometimes referred to as |
| 485 |
"tags," and multiple VLAN IDs are thus referred to as a "tag stack." Tag stacks |
| 486 |
allow L2 tunneling and the ability to segregate traffic within a particular |
| 487 |
VLAN ID, among other uses. |
| 488 |
|
| 489 |
The following are examples of how to configure 802.1ad (QinQ):: |
| 490 |
|
| 491 |
ip link add link eth0 eth0.24 type vlan proto 802.1ad id 24 |
| 492 |
ip link add link eth0.24 eth0.24.371 type vlan proto 802.1Q id 371 |
| 493 |
|
| 494 |
Where "24" and "371" are example VLAN IDs. |
| 495 |
|
| 496 |
NOTES: |
| 497 |
Receive checksum offloads, cloud filters, and VLAN acceleration are not |
| 498 |
supported for 802.1ad (QinQ) packets. |
| 499 |
|
| 500 |
VXLAN and GENEVE Overlay HW Offloading |
| 501 |
-------------------------------------- |
| 502 |
Virtual Extensible LAN (VXLAN) allows you to extend an L2 network over an L3 |
| 503 |
network, which may be useful in a virtualized or cloud environment. Some |
| 504 |
Intel(R) Ethernet Network devices perform VXLAN processing, offloading it from |
| 505 |
the operating system. This reduces CPU utilization. |
| 506 |
|
| 507 |
VXLAN offloading is controlled by the Tx and Rx checksum offload options |
| 508 |
provided by ethtool. That is, if Tx checksum offload is enabled, and the |
| 509 |
adapter has the capability, VXLAN offloading is also enabled. |
| 510 |
|
| 511 |
Support for VXLAN and GENEVE HW offloading is dependent on kernel support of |
| 512 |
the HW offloading features. |
| 513 |
|
| 514 |
Multiple Functions per Port |
| 515 |
--------------------------- |
| 516 |
Some adapters based on the Intel Ethernet Controller X710/XL710 support |
| 517 |
multiple functions on a single physical port. Configure these functions through |
| 518 |
the System Setup/BIOS. |
| 519 |
|
| 520 |
Minimum TX Bandwidth is the guaranteed minimum data transmission bandwidth, as |
| 521 |
a percentage of the full physical port link speed, that the partition will |
| 522 |
receive. The bandwidth the partition is awarded will never fall below the level |
| 523 |
you specify. |
| 524 |
|
| 525 |
The range for the minimum bandwidth values is: |
| 526 |
1 to ((100 minus # of partitions on the physical port) plus 1) |
| 527 |
For example, if a physical port has 4 partitions, the range would be: |
| 528 |
1 to ((100 - 4) + 1 = 97) |
| 529 |
|
| 530 |
The Maximum Bandwidth percentage represents the maximum transmit bandwidth |
| 531 |
allocated to the partition as a percentage of the full physical port link |
| 532 |
speed. The accepted range of values is 1-100. The value is used as a limiter, |
| 533 |
should you chose that any one particular function not be able to consume 100% |
| 534 |
of a port's bandwidth (should it be available). The sum of all the values for |
| 535 |
Maximum Bandwidth is not restricted, because no more than 100% of a port's |
| 536 |
bandwidth can ever be used. |
| 537 |
|
| 538 |
NOTE: X710/XXV710 devices fail to enable Max VFs (64) when Multiple Functions |
| 539 |
per Port (MFP) and SR-IOV are enabled. An error from i40e is logged that says |
| 540 |
"add vsi failed for VF N, aq_err 16". To workaround the issue, enable less than |
| 541 |
64 virtual functions (VFs). |
| 542 |
|
| 543 |
Data Center Bridging (DCB) |
| 544 |
-------------------------- |
| 545 |
DCB is a configuration Quality of Service implementation in hardware. It uses |
| 546 |
the VLAN priority tag (802.1p) to filter traffic. That means that there are 8 |
| 547 |
different priorities that traffic can be filtered into. It also enables |
| 548 |
priority flow control (802.1Qbb) which can limit or eliminate the number of |
| 549 |
dropped packets during network stress. Bandwidth can be allocated to each of |
| 550 |
these priorities, which is enforced at the hardware level (802.1Qaz). |
| 551 |
|
| 552 |
Adapter firmware implements LLDP and DCBX protocol agents as per 802.1AB and |
| 553 |
802.1Qaz respectively. The firmware based DCBX agent runs in willing mode only |
| 554 |
and can accept settings from a DCBX capable peer. Software configuration of |
| 555 |
DCBX parameters via dcbtool/lldptool are not supported. |
| 556 |
|
| 557 |
NOTE: Firmware LLDP can be disabled by setting the private flag disable-fw-lldp. |
| 558 |
|
| 559 |
The i40e driver implements the DCB netlink interface layer to allow user-space |
| 560 |
to communicate with the driver and query DCB configuration for the port. |
| 561 |
|
| 562 |
NOTE: |
| 563 |
The kernel assumes that TC0 is available, and will disable Priority Flow |
| 564 |
Control (PFC) on the device if TC0 is not available. To fix this, ensure TC0 is |
| 565 |
enabled when setting up DCB on your switch. |
| 566 |
|
| 567 |
Interrupt Rate Limiting |
| 568 |
----------------------- |
| 569 |
:Valid Range: 0-235 (0=no limit) |
| 570 |
|
| 571 |
The Intel(R) Ethernet Controller XL710 family supports an interrupt rate |
| 572 |
limiting mechanism. The user can control, via ethtool, the number of |
| 573 |
microseconds between interrupts. |
| 574 |
|
| 575 |
Syntax:: |
| 576 |
|
| 577 |
# ethtool -C ethX rx-usecs-high N |
| 578 |
|
| 579 |
The range of 0-235 microseconds provides an effective range of 4,310 to 250,000 |
| 580 |
interrupts per second. The value of rx-usecs-high can be set independently of |
| 581 |
rx-usecs and tx-usecs in the same ethtool command, and is also independent of |
| 582 |
the adaptive interrupt moderation algorithm. The underlying hardware supports |
| 583 |
granularity in 4-microsecond intervals, so adjacent values may result in the |
| 584 |
same interrupt rate. |
| 585 |
|
| 586 |
One possible use case is the following:: |
| 587 |
|
| 588 |
# ethtool -C ethX adaptive-rx off adaptive-tx off rx-usecs-high 20 rx-usecs \ |
| 589 |
5 tx-usecs 5 |
| 590 |
|
| 591 |
The above command would disable adaptive interrupt moderation, and allow a |
| 592 |
maximum of 5 microseconds before indicating a receive or transmit was complete. |
| 593 |
However, instead of resulting in as many as 200,000 interrupts per second, it |
| 594 |
limits total interrupts per second to 50,000 via the rx-usecs-high parameter. |
| 595 |
|
| 596 |
Performance Optimization |
| 597 |
======================== |
| 598 |
Driver defaults are meant to fit a wide variety of workloads, but if further |
| 599 |
optimization is required we recommend experimenting with the following settings. |
| 600 |
|
| 601 |
NOTE: For better performance when processing small (64B) frame sizes, try |
| 602 |
enabling Hyper threading in the BIOS in order to increase the number of logical |
| 603 |
cores in the system and subsequently increase the number of queues available to |
| 604 |
the adapter. |
| 605 |
|
| 606 |
Virtualized Environments |
| 607 |
------------------------ |
| 608 |
1. Disable XPS on both ends by using the included virt_perf_default script |
| 609 |
or by running the following command as root:: |
| 610 |
|
| 611 |
for file in `ls /sys/class/net/<ethX>/queues/tx-*/xps_cpus`; |
| 612 |
do echo 0 > $file; done |
| 613 |
|
| 614 |
2. Using the appropriate mechanism (vcpupin) in the vm, pin the cpu's to |
| 615 |
individual lcpu's, making sure to use a set of cpu's included in the |
| 616 |
device's local_cpulist: /sys/class/net/<ethX>/device/local_cpulist. |
| 617 |
|
| 618 |
3. Configure as many Rx/Tx queues in the VM as available. Do not rely on |
| 619 |
the default setting of 1. |
| 620 |
|
| 621 |
|
| 622 |
Non-virtualized Environments |
| 623 |
---------------------------- |
| 624 |
Pin the adapter's IRQs to specific cores by disabling the irqbalance service |
| 625 |
and using the included set_irq_affinity script. Please see the script's help |
| 626 |
text for further options. |
| 627 |
|
| 628 |
- The following settings will distribute the IRQs across all the cores evenly:: |
| 629 |
|
| 630 |
# scripts/set_irq_affinity -x all <interface1> , [ <interface2>, ... ] |
| 631 |
|
| 632 |
- The following settings will distribute the IRQs across all the cores that are |
| 633 |
local to the adapter (same NUMA node):: |
| 634 |
|
| 635 |
# scripts/set_irq_affinity -x local <interface1> ,[ <interface2>, ... ] |
| 636 |
|
| 637 |
For very CPU intensive workloads, we recommend pinning the IRQs to all cores. |
| 638 |
|
| 639 |
For IP Forwarding: Disable Adaptive ITR and lower Rx and Tx interrupts per |
| 640 |
queue using ethtool. |
| 641 |
|
| 642 |
- Setting rx-usecs and tx-usecs to 125 will limit interrupts to about 8000 |
| 643 |
interrupts per second per queue. |
| 644 |
|
| 645 |
:: |
| 646 |
|
| 647 |
# ethtool -C <interface> adaptive-rx off adaptive-tx off rx-usecs 125 \ |
| 648 |
tx-usecs 125 |
| 649 |
|
| 650 |
For lower CPU utilization: Disable Adaptive ITR and lower Rx and Tx interrupts |
| 651 |
per queue using ethtool. |
| 652 |
|
| 653 |
- Setting rx-usecs and tx-usecs to 250 will limit interrupts to about 4000 |
| 654 |
interrupts per second per queue. |
| 655 |
|
| 656 |
:: |
| 657 |
|
| 658 |
# ethtool -C <interface> adaptive-rx off adaptive-tx off rx-usecs 250 \ |
| 659 |
tx-usecs 250 |
| 660 |
|
| 661 |
For lower latency: Disable Adaptive ITR and ITR by setting Rx and Tx to 0 using |
| 662 |
ethtool. |
| 663 |
|
| 664 |
:: |
| 665 |
|
| 666 |
# ethtool -C <interface> adaptive-rx off adaptive-tx off rx-usecs 0 \ |
| 667 |
tx-usecs 0 |
| 668 |
|
| 669 |
Application Device Queues (ADq) |
| 670 |
------------------------------- |
| 671 |
Application Device Queues (ADq) allows you to dedicate one or more queues to a |
| 672 |
specific application. This can reduce latency for the specified application, |
| 673 |
and allow Tx traffic to be rate limited per application. Follow the steps below |
| 674 |
to set ADq. |
| 675 |
|
| 676 |
1. Create traffic classes (TCs). Maximum of 8 TCs can be created per interface. |
| 677 |
The shaper bw_rlimit parameter is optional. |
| 678 |
|
| 679 |
Example: Sets up two tcs, tc0 and tc1, with 16 queues each and max tx rate set |
| 680 |
to 1Gbit for tc0 and 3Gbit for tc1. |
| 681 |
|
| 682 |
:: |
| 683 |
|
| 684 |
# tc qdisc add dev <interface> root mqprio num_tc 2 map 0 0 0 0 1 1 1 1 |
| 685 |
queues 16@0 16@16 hw 1 mode channel shaper bw_rlimit min_rate 1Gbit 2Gbit |
| 686 |
max_rate 1Gbit 3Gbit |
| 687 |
|
| 688 |
map: priority mapping for up to 16 priorities to tcs (e.g. map 0 0 0 0 1 1 1 1 |
| 689 |
sets priorities 0-3 to use tc0 and 4-7 to use tc1) |
| 690 |
|
| 691 |
queues: for each tc, <num queues>@<offset> (e.g. queues 16@0 16@16 assigns |
| 692 |
16 queues to tc0 at offset 0 and 16 queues to tc1 at offset 16. Max total |
| 693 |
number of queues for all tcs is 64 or number of cores, whichever is lower.) |
| 694 |
|
| 695 |
hw 1 mode channel: ‘channel’ with ‘hw’ set to 1 is a new new hardware |
| 696 |
offload mode in mqprio that makes full use of the mqprio options, the |
| 697 |
TCs, the queue configurations, and the QoS parameters. |
| 698 |
|
| 699 |
shaper bw_rlimit: for each tc, sets minimum and maximum bandwidth rates. |
| 700 |
Totals must be equal or less than port speed. |
| 701 |
|
| 702 |
For example: min_rate 1Gbit 3Gbit: Verify bandwidth limit using network |
| 703 |
monitoring tools such as `ifstat` or `sar -n DEV [interval] [number of samples]` |
| 704 |
|
| 705 |
2. Enable HW TC offload on interface:: |
| 706 |
|
| 707 |
# ethtool -K <interface> hw-tc-offload on |
| 708 |
|
| 709 |
3. Apply TCs to ingress (RX) flow of interface:: |
| 710 |
|
| 711 |
# tc qdisc add dev <interface> ingress |
| 712 |
|
| 713 |
NOTES: |
| 714 |
- Run all tc commands from the iproute2 <pathtoiproute2>/tc/ directory. |
| 715 |
- ADq is not compatible with cloud filters. |
| 716 |
- Setting up channels via ethtool (ethtool -L) is not supported when the |
| 717 |
TCs are configured using mqprio. |
| 718 |
- You must have iproute2 latest version |
| 719 |
- NVM version 6.01 or later is required. |
| 720 |
- ADq cannot be enabled when any the following features are enabled: Data |
| 721 |
Center Bridging (DCB), Multiple Functions per Port (MFP), or Sideband |
| 722 |
Filters. |
| 723 |
- If another driver (for example, DPDK) has set cloud filters, you cannot |
| 724 |
enable ADq. |
| 725 |
- Tunnel filters are not supported in ADq. If encapsulated packets do |
| 726 |
arrive in non-tunnel mode, filtering will be done on the inner headers. |
| 727 |
For example, for VXLAN traffic in non-tunnel mode, PCTYPE is identified |
| 728 |
as a VXLAN encapsulated packet, outer headers are ignored. Therefore, |
| 729 |
inner headers are matched. |
| 730 |
- If a TC filter on a PF matches traffic over a VF (on the PF), that |
| 731 |
traffic will be routed to the appropriate queue of the PF, and will |
| 732 |
not be passed on the VF. Such traffic will end up getting dropped higher |
| 733 |
up in the TCP/IP stack as it does not match PF address data. |
| 734 |
- If traffic matches multiple TC filters that point to different TCs, |
| 735 |
that traffic will be duplicated and sent to all matching TC queues. |
| 736 |
The hardware switch mirrors the packet to a VSI list when multiple |
| 737 |
filters are matched. |
| 738 |
|
| 739 |
|
| 740 |
Known Issues/Troubleshooting |
| 741 |
============================ |
| 742 |
|
| 743 |
NOTE: 1 Gb devices based on the Intel(R) Ethernet Network Connection X722 do |
| 744 |
not support the following features: |
| 745 |
|
| 746 |
* Data Center Bridging (DCB) |
| 747 |
* QOS |
| 748 |
* VMQ |
| 749 |
* SR-IOV |
| 750 |
* Task Encapsulation offload (VXLAN, NVGRE) |
| 751 |
* Energy Efficient Ethernet (EEE) |
| 752 |
* Auto-media detect |
| 753 |
|
| 754 |
Unexpected Issues when the device driver and DPDK share a device |
| 755 |
---------------------------------------------------------------- |
| 756 |
Unexpected issues may result when an i40e device is in multi driver mode and |
| 757 |
the kernel driver and DPDK driver are sharing the device. This is because |
| 758 |
access to the global NIC resources is not synchronized between multiple |
| 759 |
drivers. Any change to the global NIC configuration (writing to a global |
| 760 |
register, setting global configuration by AQ, or changing switch modes) will |
| 761 |
affect all ports and drivers on the device. Loading DPDK with the |
| 762 |
"multi-driver" module parameter may mitigate some of the issues. |
| 763 |
|
| 764 |
TC0 must be enabled when setting up DCB on a switch |
| 765 |
--------------------------------------------------- |
| 766 |
The kernel assumes that TC0 is available, and will disable Priority Flow |
| 767 |
Control (PFC) on the device if TC0 is not available. To fix this, ensure TC0 is |
| 768 |
enabled when setting up DCB on your switch. |
| 769 |
|
| 770 |
|
| 771 |
Support |
| 772 |
======= |
| 773 |
For general information, go to the Intel support website at: |
| 774 |
https://www.intel.com/support/ |
| 775 |
|
| 776 |
If an issue is identified with the released source code on a supported kernel |
| 777 |
with a supported adapter, email the specific information related to the issue |
| 778 |
to [email protected]. |