| 1 |
.. SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) |
| 2 |
.. include:: <isonum.txt> |
| 3 |
|
| 4 |
=========================================== |
| 5 |
Network Flow Processor (NFP) Kernel Drivers |
| 6 |
=========================================== |
| 7 |
|
| 8 |
:Copyright: |copy| 2019, Netronome Systems, Inc. |
| 9 |
:Copyright: |copy| 2022, Corigine, Inc. |
| 10 |
|
| 11 |
Contents |
| 12 |
======== |
| 13 |
|
| 14 |
- `Overview`_ |
| 15 |
- `Acquiring Firmware`_ |
| 16 |
- `Devlink Info`_ |
| 17 |
- `Configure Device`_ |
| 18 |
- `Statistics`_ |
| 19 |
|
| 20 |
Overview |
| 21 |
======== |
| 22 |
|
| 23 |
This driver supports Netronome and Corigine's line of Network Flow Processor |
| 24 |
devices, including the NFP3800, NFP4000, NFP5000, and NFP6000 models, which |
| 25 |
are also incorporated in the companies' family of Agilio SmartNICs. The SR-IOV |
| 26 |
physical and virtual functions for these devices are supported by the driver. |
| 27 |
|
| 28 |
Acquiring Firmware |
| 29 |
================== |
| 30 |
|
| 31 |
The NFP3800, NFP4000 and NFP6000 devices require application specific firmware |
| 32 |
to function. Application firmware can be located either on the host file system |
| 33 |
or in the device flash (if supported by management firmware). |
| 34 |
|
| 35 |
Firmware files on the host filesystem contain card type (`AMDA-*` string), media |
| 36 |
config etc. They should be placed in `/lib/firmware/netronome` directory to |
| 37 |
load firmware from the host file system. |
| 38 |
|
| 39 |
Firmware for basic NIC operation is available in the upstream |
| 40 |
`linux-firmware.git` repository. |
| 41 |
|
| 42 |
A more comprehensive list of firmware can be downloaded from the |
| 43 |
`Corigine Support site <https://www.corigine.com/DPUDownload.html>`_. |
| 44 |
|
| 45 |
Firmware in NVRAM |
| 46 |
----------------- |
| 47 |
|
| 48 |
Recent versions of management firmware supports loading application |
| 49 |
firmware from flash when the host driver gets probed. The firmware loading |
| 50 |
policy configuration may be used to configure this feature appropriately. |
| 51 |
|
| 52 |
Devlink or ethtool can be used to update the application firmware on the device |
| 53 |
flash by providing the appropriate `nic_AMDA*.nffw` file to the respective |
| 54 |
command. Users need to take care to write the correct firmware image for the |
| 55 |
card and media configuration to flash. |
| 56 |
|
| 57 |
Available storage space in flash depends on the card being used. |
| 58 |
|
| 59 |
Dealing with multiple projects |
| 60 |
------------------------------ |
| 61 |
|
| 62 |
NFP hardware is fully programmable therefore there can be different |
| 63 |
firmware images targeting different applications. |
| 64 |
|
| 65 |
When using application firmware from host, we recommend placing |
| 66 |
actual firmware files in application-named subdirectories in |
| 67 |
`/lib/firmware/netronome` and linking the desired files, e.g.:: |
| 68 |
|
| 69 |
$ tree /lib/firmware/netronome/ |
| 70 |
/lib/firmware/netronome/ |
| 71 |
├── bpf |
| 72 |
│ ├── nic_AMDA0081-0001_1x40.nffw |
| 73 |
│ └── nic_AMDA0081-0001_4x10.nffw |
| 74 |
├── flower |
| 75 |
│ ├── nic_AMDA0081-0001_1x40.nffw |
| 76 |
│ └── nic_AMDA0081-0001_4x10.nffw |
| 77 |
├── nic |
| 78 |
│ ├── nic_AMDA0081-0001_1x40.nffw |
| 79 |
│ └── nic_AMDA0081-0001_4x10.nffw |
| 80 |
├── nic_AMDA0081-0001_1x40.nffw -> bpf/nic_AMDA0081-0001_1x40.nffw |
| 81 |
└── nic_AMDA0081-0001_4x10.nffw -> bpf/nic_AMDA0081-0001_4x10.nffw |
| 82 |
|
| 83 |
3 directories, 8 files |
| 84 |
|
| 85 |
You may need to use hard instead of symbolic links on distributions |
| 86 |
which use old `mkinitrd` command instead of `dracut` (e.g. Ubuntu). |
| 87 |
|
| 88 |
After changing firmware files you may need to regenerate the initramfs |
| 89 |
image. Initramfs contains drivers and firmware files your system may |
| 90 |
need to boot. Refer to the documentation of your distribution to find |
| 91 |
out how to update initramfs. Good indication of stale initramfs |
| 92 |
is system loading wrong driver or firmware on boot, but when driver is |
| 93 |
later reloaded manually everything works correctly. |
| 94 |
|
| 95 |
Selecting firmware per device |
| 96 |
----------------------------- |
| 97 |
|
| 98 |
Most commonly all cards on the system use the same type of firmware. |
| 99 |
If you want to load a specific firmware image for a specific card, you |
| 100 |
can use either the PCI bus address or serial number. The driver will |
| 101 |
print which files it's looking for when it recognizes a NFP device:: |
| 102 |
|
| 103 |
nfp: Looking for firmware file in order of priority: |
| 104 |
nfp: netronome/serial-00-12-34-aa-bb-cc-10-ff.nffw: not found |
| 105 |
nfp: netronome/pci-0000:02:00.0.nffw: not found |
| 106 |
nfp: netronome/nic_AMDA0081-0001_1x40.nffw: found, loading... |
| 107 |
|
| 108 |
In this case if file (or link) called *serial-00-12-34-aa-bb-5d-10-ff.nffw* |
| 109 |
or *pci-0000:02:00.0.nffw* is present in `/lib/firmware/netronome` this |
| 110 |
firmware file will take precedence over `nic_AMDA*` files. |
| 111 |
|
| 112 |
Note that `serial-*` and `pci-*` files are **not** automatically included |
| 113 |
in initramfs, you will have to refer to documentation of appropriate tools |
| 114 |
to find out how to include them. |
| 115 |
|
| 116 |
Running firmware version |
| 117 |
------------------------ |
| 118 |
|
| 119 |
The version of the loaded firmware for a particular <netdev> interface, |
| 120 |
(e.g. enp4s0), or an interface's port <netdev port> (e.g. enp4s0np0) can |
| 121 |
be displayed with the ethtool command:: |
| 122 |
|
| 123 |
$ ethtool -i <netdev> |
| 124 |
|
| 125 |
Firmware loading policy |
| 126 |
----------------------- |
| 127 |
|
| 128 |
Firmware loading policy is controlled via three HWinfo parameters |
| 129 |
stored as key value pairs in the device flash: |
| 130 |
|
| 131 |
app_fw_from_flash |
| 132 |
Defines which firmware should take precedence, 'Disk' (0), 'Flash' (1) or |
| 133 |
the 'Preferred' (2) firmware. When 'Preferred' is selected, the management |
| 134 |
firmware makes the decision over which firmware will be loaded by comparing |
| 135 |
versions of the flash firmware and the host supplied firmware. |
| 136 |
This variable is configurable using the 'fw_load_policy' |
| 137 |
devlink parameter. |
| 138 |
|
| 139 |
abi_drv_reset |
| 140 |
Defines if the driver should reset the firmware when |
| 141 |
the driver is probed, either 'Disk' (0) if firmware was found on disk, |
| 142 |
'Always' (1) reset or 'Never' (2) reset. Note that the device is always |
| 143 |
reset on driver unload if firmware was loaded when the driver was probed. |
| 144 |
This variable is configurable using the 'reset_dev_on_drv_probe' |
| 145 |
devlink parameter. |
| 146 |
|
| 147 |
abi_drv_load_ifc |
| 148 |
Defines a list of PF devices allowed to load FW on the device. |
| 149 |
This variable is not currently user configurable. |
| 150 |
|
| 151 |
Devlink Info |
| 152 |
============ |
| 153 |
|
| 154 |
The devlink info command displays the running and stored firmware versions |
| 155 |
on the device, serial number and board information. |
| 156 |
|
| 157 |
Devlink info command example (replace PCI address):: |
| 158 |
|
| 159 |
$ devlink dev info pci/0000:03:00.0 |
| 160 |
pci/0000:03:00.0: |
| 161 |
driver nfp |
| 162 |
serial_number CSAAMDA2001-1003000111 |
| 163 |
versions: |
| 164 |
fixed: |
| 165 |
board.id AMDA2001-1003 |
| 166 |
board.rev 01 |
| 167 |
board.manufacture CSA |
| 168 |
board.model mozart |
| 169 |
running: |
| 170 |
fw.mgmt 22.10.0-rc3 |
| 171 |
fw.cpld 0x1000003 |
| 172 |
fw.app nic-22.09.0 |
| 173 |
chip.init AMDA-2001-1003 1003000111 |
| 174 |
stored: |
| 175 |
fw.bundle_id bspbundle_1003000111 |
| 176 |
fw.mgmt 22.10.0-rc3 |
| 177 |
fw.cpld 0x0 |
| 178 |
chip.init AMDA-2001-1003 1003000111 |
| 179 |
|
| 180 |
Configure Device |
| 181 |
================ |
| 182 |
|
| 183 |
This section explains how to use Agilio SmartNICs running basic NIC firmware. |
| 184 |
|
| 185 |
Configure interface link-speed |
| 186 |
------------------------------ |
| 187 |
The following steps explains how to change between 10G mode and 25G mode on |
| 188 |
Agilio CX 2x25GbE cards. The changing of port speed must be done in order, |
| 189 |
port 0 (p0) must be set to 10G before port 1 (p1) may be set to 10G. |
| 190 |
|
| 191 |
Down the respective interface(s):: |
| 192 |
|
| 193 |
$ ip link set dev <netdev port 0> down |
| 194 |
$ ip link set dev <netdev port 1> down |
| 195 |
|
| 196 |
Set interface link-speed to 10G:: |
| 197 |
|
| 198 |
$ ethtool -s <netdev port 0> speed 10000 |
| 199 |
$ ethtool -s <netdev port 1> speed 10000 |
| 200 |
|
| 201 |
Set interface link-speed to 25G:: |
| 202 |
|
| 203 |
$ ethtool -s <netdev port 0> speed 25000 |
| 204 |
$ ethtool -s <netdev port 1> speed 25000 |
| 205 |
|
| 206 |
Reload driver for changes to take effect:: |
| 207 |
|
| 208 |
$ rmmod nfp; modprobe nfp |
| 209 |
|
| 210 |
Configure interface Maximum Transmission Unit (MTU) |
| 211 |
--------------------------------------------------- |
| 212 |
|
| 213 |
The MTU of interfaces can temporarily be set using the iproute2, ip link or |
| 214 |
ifconfig tools. Note that this change will not persist. Setting this via |
| 215 |
Network Manager, or another appropriate OS configuration tool, is |
| 216 |
recommended as changes to the MTU using Network Manager can be made to |
| 217 |
persist. |
| 218 |
|
| 219 |
Set interface MTU to 9000 bytes:: |
| 220 |
|
| 221 |
$ ip link set dev <netdev port> mtu 9000 |
| 222 |
|
| 223 |
It is the responsibility of the user or the orchestration layer to set |
| 224 |
appropriate MTU values when handling jumbo frames or utilizing tunnels. For |
| 225 |
example, if packets sent from a VM are to be encapsulated on the card and |
| 226 |
egress a physical port, then the MTU of the VF should be set to lower than |
| 227 |
that of the physical port to account for the extra bytes added by the |
| 228 |
additional header. If a setup is expected to see fallback traffic between |
| 229 |
the SmartNIC and the kernel then the user should also ensure that the PF MTU |
| 230 |
is appropriately set to avoid unexpected drops on this path. |
| 231 |
|
| 232 |
Configure Forward Error Correction (FEC) modes |
| 233 |
---------------------------------------------- |
| 234 |
|
| 235 |
Agilio SmartNICs support FEC mode configuration, e.g. Auto, Firecode Base-R, |
| 236 |
ReedSolomon and Off modes. Each physical port's FEC mode can be set |
| 237 |
independently using ethtool. The supported FEC modes for an interface can |
| 238 |
be viewed using:: |
| 239 |
|
| 240 |
$ ethtool <netdev> |
| 241 |
|
| 242 |
The currently configured FEC mode can be viewed using:: |
| 243 |
|
| 244 |
$ ethtool --show-fec <netdev> |
| 245 |
|
| 246 |
To force the FEC mode for a particular port, auto-negotiation must be disabled |
| 247 |
(see the `Auto-negotiation`_ section). An example of how to set the FEC mode |
| 248 |
to Reed-Solomon is:: |
| 249 |
|
| 250 |
$ ethtool --set-fec <netdev> encoding rs |
| 251 |
|
| 252 |
Auto-negotiation |
| 253 |
---------------- |
| 254 |
|
| 255 |
To change auto-negotiation settings, the link must first be put down. After the |
| 256 |
link is down, auto-negotiation can be enabled or disabled using:: |
| 257 |
|
| 258 |
ethtool -s <netdev> autoneg <on|off> |
| 259 |
|
| 260 |
Statistics |
| 261 |
========== |
| 262 |
|
| 263 |
Following device statistics are available through the ``ethtool -S`` interface: |
| 264 |
|
| 265 |
.. flat-table:: NFP device statistics |
| 266 |
:header-rows: 1 |
| 267 |
:widths: 3 1 11 |
| 268 |
|
| 269 |
* - Name |
| 270 |
- ID |
| 271 |
- Meaning |
| 272 |
|
| 273 |
* - dev_rx_discards |
| 274 |
- 1 |
| 275 |
- Packet can be discarded on the RX path for one of the following reasons: |
| 276 |
|
| 277 |
* The NIC is not in promisc mode, and the destination MAC address |
| 278 |
doesn't match the interfaces' MAC address. |
| 279 |
* The received packet is larger than the max buffer size on the host. |
| 280 |
I.e. it exceeds the Layer 3 MRU. |
| 281 |
* There is no freelist descriptor available on the host for the packet. |
| 282 |
It is likely that the NIC couldn't cache one in time. |
| 283 |
* A BPF program discarded the packet. |
| 284 |
* The datapath drop action was executed. |
| 285 |
* The MAC discarded the packet due to lack of ingress buffer space |
| 286 |
on the NIC. |
| 287 |
|
| 288 |
* - dev_rx_errors |
| 289 |
- 2 |
| 290 |
- A packet can be counted (and dropped) as RX error for the following |
| 291 |
reasons: |
| 292 |
|
| 293 |
* A problem with the VEB lookup (only when SR-IOV is used). |
| 294 |
* A physical layer problem that causes Ethernet errors, like FCS or |
| 295 |
alignment errors. The cause is usually faulty cables or SFPs. |
| 296 |
|
| 297 |
* - dev_rx_bytes |
| 298 |
- 3 |
| 299 |
- Total number of bytes received. |
| 300 |
|
| 301 |
* - dev_rx_uc_bytes |
| 302 |
- 4 |
| 303 |
- Unicast bytes received. |
| 304 |
|
| 305 |
* - dev_rx_mc_bytes |
| 306 |
- 5 |
| 307 |
- Multicast bytes received. |
| 308 |
|
| 309 |
* - dev_rx_bc_bytes |
| 310 |
- 6 |
| 311 |
- Broadcast bytes received. |
| 312 |
|
| 313 |
* - dev_rx_pkts |
| 314 |
- 7 |
| 315 |
- Total number of packets received. |
| 316 |
|
| 317 |
* - dev_rx_mc_pkts |
| 318 |
- 8 |
| 319 |
- Multicast packets received. |
| 320 |
|
| 321 |
* - dev_rx_bc_pkts |
| 322 |
- 9 |
| 323 |
- Broadcast packets received. |
| 324 |
|
| 325 |
* - dev_tx_discards |
| 326 |
- 10 |
| 327 |
- A packet can be discarded in the TX direction if the MAC is |
| 328 |
being flow controlled and the NIC runs out of TX queue space. |
| 329 |
|
| 330 |
* - dev_tx_errors |
| 331 |
- 11 |
| 332 |
- A packet can be counted as TX error (and dropped) for one for the |
| 333 |
following reasons: |
| 334 |
|
| 335 |
* The packet is an LSO segment, but the Layer 3 or Layer 4 offset |
| 336 |
could not be determined. Therefore LSO could not continue. |
| 337 |
* An invalid packet descriptor was received over PCIe. |
| 338 |
* The packet Layer 3 length exceeds the device MTU. |
| 339 |
* An error on the MAC/physical layer. Usually due to faulty cables or |
| 340 |
SFPs. |
| 341 |
* A CTM buffer could not be allocated. |
| 342 |
* The packet offset was incorrect and could not be fixed by the NIC. |
| 343 |
|
| 344 |
* - dev_tx_bytes |
| 345 |
- 12 |
| 346 |
- Total number of bytes transmitted. |
| 347 |
|
| 348 |
* - dev_tx_uc_bytes |
| 349 |
- 13 |
| 350 |
- Unicast bytes transmitted. |
| 351 |
|
| 352 |
* - dev_tx_mc_bytes |
| 353 |
- 14 |
| 354 |
- Multicast bytes transmitted. |
| 355 |
|
| 356 |
* - dev_tx_bc_bytes |
| 357 |
- 15 |
| 358 |
- Broadcast bytes transmitted. |
| 359 |
|
| 360 |
* - dev_tx_pkts |
| 361 |
- 16 |
| 362 |
- Total number of packets transmitted. |
| 363 |
|
| 364 |
* - dev_tx_mc_pkts |
| 365 |
- 17 |
| 366 |
- Multicast packets transmitted. |
| 367 |
|
| 368 |
* - dev_tx_bc_pkts |
| 369 |
- 18 |
| 370 |
- Broadcast packets transmitted. |
| 371 |
|
| 372 |
Note that statistics unknown to the driver will be displayed as |
| 373 |
``dev_unknown_stat$ID``, where ``$ID`` refers to the second column |
| 374 |
above. |