| 1 |
.. SPDX-License-Identifier: GPL-2.0 |
| 2 |
|
| 3 |
==================== |
| 4 |
mlx5 devlink support |
| 5 |
==================== |
| 6 |
|
| 7 |
This document describes the devlink features implemented by the ``mlx5`` |
| 8 |
device driver. |
| 9 |
|
| 10 |
Parameters |
| 11 |
========== |
| 12 |
|
| 13 |
.. list-table:: Generic parameters implemented |
| 14 |
|
| 15 |
* - Name |
| 16 |
- Mode |
| 17 |
- Validation |
| 18 |
- Notes |
| 19 |
* - ``enable_roce`` |
| 20 |
- driverinit |
| 21 |
- Boolean |
| 22 |
- If the device supports RoCE disablement, RoCE enablement state controls |
| 23 |
device support for RoCE capability. Otherwise, the control occurs in the |
| 24 |
driver stack. When RoCE is disabled at the driver level, only raw |
| 25 |
ethernet QPs are supported. |
| 26 |
* - ``io_eq_size`` |
| 27 |
- driverinit |
| 28 |
- The range is between 64 and 4096. |
| 29 |
- |
| 30 |
* - ``event_eq_size`` |
| 31 |
- driverinit |
| 32 |
- The range is between 64 and 4096. |
| 33 |
- |
| 34 |
* - ``max_macs`` |
| 35 |
- driverinit |
| 36 |
- The range is between 1 and 2^31. Only power of 2 values are supported. |
| 37 |
- |
| 38 |
* - ``enable_sriov`` |
| 39 |
- permanent |
| 40 |
- Boolean |
| 41 |
- Applies to each physical function (PF) independently, if the device |
| 42 |
supports it. Otherwise, it applies symmetrically to all PFs. |
| 43 |
* - ``total_vfs`` |
| 44 |
- permanent |
| 45 |
- The range is between 1 and a device-specific max. |
| 46 |
- Applies to each physical function (PF) independently, if the device |
| 47 |
supports it. Otherwise, it applies symmetrically to all PFs. |
| 48 |
|
| 49 |
Note: permanent parameters such as ``enable_sriov`` and ``total_vfs`` require FW reset to take effect |
| 50 |
|
| 51 |
.. code-block:: bash |
| 52 |
|
| 53 |
# setup parameters |
| 54 |
devlink dev param set pci/0000:01:00.0 name enable_sriov value true cmode permanent |
| 55 |
devlink dev param set pci/0000:01:00.0 name total_vfs value 8 cmode permanent |
| 56 |
|
| 57 |
# Fw reset |
| 58 |
devlink dev reload pci/0000:01:00.0 action fw_activate |
| 59 |
|
| 60 |
# for PCI related config such as sriov PCI reset/rescan is required: |
| 61 |
echo 1 >/sys/bus/pci/devices/0000:01:00.0/remove |
| 62 |
echo 1 >/sys/bus/pci/rescan |
| 63 |
grep ^ /sys/bus/pci/devices/0000:01:00.0/sriov_* |
| 64 |
|
| 65 |
* - ``num_doorbells`` |
| 66 |
- driverinit |
| 67 |
- This controls the number of channel doorbells used by the netdev. In all |
| 68 |
cases, an additional doorbell is allocated and used for non-channel |
| 69 |
communication (e.g. for PTP, HWS, etc.). Supported values are: |
| 70 |
|
| 71 |
- 0: No channel-specific doorbells, use the global one for everything. |
| 72 |
- [1, max_num_channels]: Spread netdev channels equally across these |
| 73 |
doorbells. |
| 74 |
|
| 75 |
The ``mlx5`` driver also implements the following driver-specific |
| 76 |
parameters. |
| 77 |
|
| 78 |
.. list-table:: Driver-specific parameters implemented |
| 79 |
:widths: 5 5 5 85 |
| 80 |
|
| 81 |
* - Name |
| 82 |
- Type |
| 83 |
- Mode |
| 84 |
- Description |
| 85 |
* - ``flow_steering_mode`` |
| 86 |
- string |
| 87 |
- runtime |
| 88 |
- Controls the flow steering mode of the driver |
| 89 |
|
| 90 |
* ``dmfs`` Device managed flow steering. In DMFS mode, the HW |
| 91 |
steering entities are created and managed through firmware. |
| 92 |
* ``smfs`` Software managed flow steering. In SMFS mode, the HW |
| 93 |
steering entities are created and manage through the driver without |
| 94 |
firmware intervention. |
| 95 |
* ``hmfs`` Hardware managed flow steering. In HMFS mode, the driver |
| 96 |
is configuring steering rules directly to the HW using Work Queues with |
| 97 |
a special new type of WQE (Work Queue Element). |
| 98 |
|
| 99 |
SMFS mode is faster and provides better rule insertion rate compared to |
| 100 |
default DMFS mode. |
| 101 |
* - ``fdb_large_groups`` |
| 102 |
- u32 |
| 103 |
- driverinit |
| 104 |
- Control the number of large groups (size > 1) in the FDB table. |
| 105 |
|
| 106 |
* The default value is 15, and the range is between 1 and 1024. |
| 107 |
* - ``esw_multiport`` |
| 108 |
- Boolean |
| 109 |
- runtime |
| 110 |
- Control MultiPort E-Switch shared fdb mode. |
| 111 |
|
| 112 |
An experimental mode where a single E-Switch is used and all the vports |
| 113 |
and physical ports on the NIC are connected to it. |
| 114 |
|
| 115 |
An example is to send traffic from a VF that is created on PF0 to an |
| 116 |
uplink that is natively associated with the uplink of PF1 |
| 117 |
|
| 118 |
Note: Future devices, ConnectX-8 and onward, will eventually have this |
| 119 |
as the default to allow forwarding between all NIC ports in a single |
| 120 |
E-switch environment and the dual E-switch mode will likely get |
| 121 |
deprecated. |
| 122 |
|
| 123 |
Default: disabled |
| 124 |
* - ``esw_port_metadata`` |
| 125 |
- Boolean |
| 126 |
- runtime |
| 127 |
- When applicable, disabling eswitch metadata can increase packet rate up |
| 128 |
to 20% depending on the use case and packet sizes. |
| 129 |
|
| 130 |
Eswitch port metadata state controls whether to internally tag packets |
| 131 |
with metadata. Metadata tagging must be enabled for multi-port RoCE, |
| 132 |
failover between representors and stacked devices. By default metadata is |
| 133 |
enabled on the supported devices in E-switch. Metadata is applicable only |
| 134 |
for E-switch in switchdev mode and users may disable it when NONE of the |
| 135 |
below use cases will be in use: |
| 136 |
1. HCA is in Dual/multi-port RoCE mode. |
| 137 |
2. VF/SF representor bonding (Usually used for Live migration) |
| 138 |
3. Stacked devices |
| 139 |
|
| 140 |
When metadata is disabled, the above use cases will fail to initialize if |
| 141 |
users try to enable them. |
| 142 |
|
| 143 |
Note: Setting this parameter does not take effect immediately. Setting |
| 144 |
must happen in legacy mode and eswitch port metadata takes effect after |
| 145 |
enabling switchdev mode. |
| 146 |
* - ``hairpin_num_queues`` |
| 147 |
- u32 |
| 148 |
- driverinit |
| 149 |
- We refer to a TC NIC rule that involves forwarding as "hairpin". |
| 150 |
Hairpin queues are mlx5 hardware specific implementation for hardware |
| 151 |
forwarding of such packets. |
| 152 |
|
| 153 |
Control the number of hairpin queues. |
| 154 |
* - ``hairpin_queue_size`` |
| 155 |
- u32 |
| 156 |
- driverinit |
| 157 |
- Control the size (in packets) of the hairpin queues. |
| 158 |
* - ``pcie_cong_inbound_high`` |
| 159 |
- u16 |
| 160 |
- driverinit |
| 161 |
- High threshold configuration for PCIe congestion events. The firmware |
| 162 |
will send an event once device side inbound PCIe traffic went |
| 163 |
above the configured high threshold for a long enough period (at least |
| 164 |
200ms). |
| 165 |
|
| 166 |
See pci_bw_inbound_high ethtool stat. |
| 167 |
|
| 168 |
Units are 0.01 %. Accepted values are in range [0, 10000]. |
| 169 |
pcie_cong_inbound_low < pcie_cong_inbound_high. |
| 170 |
Default value: 9000 (Corresponds to 90%). |
| 171 |
* - ``pcie_cong_inbound_low`` |
| 172 |
- u16 |
| 173 |
- driverinit |
| 174 |
- Low threshold configuration for PCIe congestion events. The firmware |
| 175 |
will send an event once device side inbound PCIe traffic went |
| 176 |
below the configured low threshold, only after having been previously in |
| 177 |
a congested state. |
| 178 |
|
| 179 |
See pci_bw_inbound_low ethtool stat. |
| 180 |
|
| 181 |
Units are 0.01 %. Accepted values are in range [0, 10000]. |
| 182 |
pcie_cong_inbound_low < pcie_cong_inbound_high. |
| 183 |
Default value: 7500. |
| 184 |
* - ``pcie_cong_outbound_high`` |
| 185 |
- u16 |
| 186 |
- driverinit |
| 187 |
- High threshold configuration for PCIe congestion events. The firmware |
| 188 |
will send an event once device side outbound PCIe traffic went |
| 189 |
above the configured high threshold for a long enough period (at least |
| 190 |
200ms). |
| 191 |
|
| 192 |
See pci_bw_outbound_high ethtool stat. |
| 193 |
|
| 194 |
Units are 0.01 %. Accepted values are in range [0, 10000]. |
| 195 |
pcie_cong_outbound_low < pcie_cong_outbound_high. |
| 196 |
Default value: 9000 (Corresponds to 90%). |
| 197 |
* - ``pcie_cong_outbound_low`` |
| 198 |
- u16 |
| 199 |
- driverinit |
| 200 |
- Low threshold configuration for PCIe congestion events. The firmware |
| 201 |
will send an event once device side outbound PCIe traffic went |
| 202 |
below the configured low threshold, only after having been previously in |
| 203 |
a congested state. |
| 204 |
|
| 205 |
See pci_bw_outbound_low ethtool stat. |
| 206 |
|
| 207 |
Units are 0.01 %. Accepted values are in range [0, 10000]. |
| 208 |
pcie_cong_outbound_low < pcie_cong_outbound_high. |
| 209 |
Default value: 7500. |
| 210 |
|
| 211 |
* - ``cqe_compress_type`` |
| 212 |
- string |
| 213 |
- permanent |
| 214 |
- Configure which mechanism/algorithm should be used by the NIC that will |
| 215 |
affect the rate (aggressiveness) of compressed CQEs depending on PCIe bus |
| 216 |
conditions and other internal NIC factors. This mode affects all queues |
| 217 |
that enable compression. |
| 218 |
* ``balanced`` : Merges fewer CQEs, resulting in a moderate compression ratio but maintaining a balance between bandwidth savings and performance |
| 219 |
* ``aggressive`` : Merges more CQEs into a single entry, achieving a higher compression rate and maximizing performance, particularly under high traffic loads |
| 220 |
|
| 221 |
* - ``swp_l4_csum_mode`` |
| 222 |
- string |
| 223 |
- permanent |
| 224 |
- Configure how the L4 checksum is calculated by the device when using |
| 225 |
Software Parser (SWP) hints for header locations. |
| 226 |
|
| 227 |
* ``default`` : Use the device's default checksum calculation |
| 228 |
mode. The driver will discover during init whether or |
| 229 |
full_csum or l4_only is in use. Setting this value explicitly |
| 230 |
from userspace is not allowed, but some firmware versions may |
| 231 |
return this value on param read. |
| 232 |
* ``full_csum`` : Calculate full checksum including the pseudo-header |
| 233 |
* ``l4_only`` : Calculate L4-only checksum, excluding the pseudo-header |
| 234 |
|
| 235 |
The ``mlx5`` driver supports reloading via ``DEVLINK_CMD_RELOAD`` |
| 236 |
|
| 237 |
Info versions |
| 238 |
============= |
| 239 |
|
| 240 |
The ``mlx5`` driver reports the following versions |
| 241 |
|
| 242 |
.. list-table:: devlink info versions implemented |
| 243 |
:widths: 5 5 90 |
| 244 |
|
| 245 |
* - Name |
| 246 |
- Type |
| 247 |
- Description |
| 248 |
* - ``fw.psid`` |
| 249 |
- fixed |
| 250 |
- Used to represent the board id of the device. |
| 251 |
* - ``fw.version`` |
| 252 |
- stored, running |
| 253 |
- Three digit major.minor.subminor firmware version number. |
| 254 |
|
| 255 |
Health reporters |
| 256 |
================ |
| 257 |
|
| 258 |
tx reporter |
| 259 |
----------- |
| 260 |
The tx reporter is responsible for reporting and recovering of the following three error scenarios: |
| 261 |
|
| 262 |
- tx timeout |
| 263 |
Report on kernel tx timeout detection. |
| 264 |
Recover by searching lost interrupts. |
| 265 |
- tx error completion |
| 266 |
Report on error tx completion. |
| 267 |
Recover by flushing the tx queue and reset it. |
| 268 |
- tx PTP port timestamping CQ unhealthy |
| 269 |
Report too many CQEs never delivered on port ts CQ. |
| 270 |
Recover by flushing and re-creating all PTP channels. |
| 271 |
|
| 272 |
tx reporter also support on demand diagnose callback, on which it provides |
| 273 |
real time information of its send queues status. |
| 274 |
|
| 275 |
User commands examples: |
| 276 |
|
| 277 |
- Diagnose send queues status:: |
| 278 |
|
| 279 |
$ devlink health diagnose pci/0000:82:00.0 reporter tx |
| 280 |
|
| 281 |
.. note:: |
| 282 |
This command has valid output only when interface is up, otherwise the command has empty output. |
| 283 |
|
| 284 |
- Show number of tx errors indicated, number of recover flows ended successfully, |
| 285 |
is autorecover enabled and graceful period from last recover:: |
| 286 |
|
| 287 |
$ devlink health show pci/0000:82:00.0 reporter tx |
| 288 |
|
| 289 |
rx reporter |
| 290 |
----------- |
| 291 |
The rx reporter is responsible for reporting and recovering of the following two error scenarios: |
| 292 |
|
| 293 |
- rx queues' initialization (population) timeout |
| 294 |
Population of rx queues' descriptors on ring initialization is done |
| 295 |
in napi context via triggering an irq. In case of a failure to get |
| 296 |
the minimum amount of descriptors, a timeout would occur, and |
| 297 |
descriptors could be recovered by polling the EQ (Event Queue). |
| 298 |
- rx completions with errors (reported by HW on interrupt context) |
| 299 |
Report on rx completion error. |
| 300 |
Recover (if needed) by flushing the related queue and reset it. |
| 301 |
|
| 302 |
rx reporter also supports on demand diagnose callback, on which it |
| 303 |
provides real time information of its receive queues' status. |
| 304 |
|
| 305 |
- Diagnose rx queues' status and corresponding completion queue:: |
| 306 |
|
| 307 |
$ devlink health diagnose pci/0000:82:00.0 reporter rx |
| 308 |
|
| 309 |
.. note:: |
| 310 |
This command has valid output only when interface is up. Otherwise, the command has empty output. |
| 311 |
|
| 312 |
- Show number of rx errors indicated, number of recover flows ended successfully, |
| 313 |
is autorecover enabled, and graceful period from last recover:: |
| 314 |
|
| 315 |
$ devlink health show pci/0000:82:00.0 reporter rx |
| 316 |
|
| 317 |
fw reporter |
| 318 |
----------- |
| 319 |
The fw reporter implements `diagnose` and `dump` callbacks. |
| 320 |
It follows symptoms of fw error such as fw syndrome by triggering |
| 321 |
fw core dump and storing it into the dump buffer. |
| 322 |
The fw reporter diagnose command can be triggered any time by the user to check |
| 323 |
current fw status. |
| 324 |
|
| 325 |
User commands examples: |
| 326 |
|
| 327 |
- Check fw heath status:: |
| 328 |
|
| 329 |
$ devlink health diagnose pci/0000:82:00.0 reporter fw |
| 330 |
|
| 331 |
- Read FW core dump if already stored or trigger new one:: |
| 332 |
|
| 333 |
$ devlink health dump show pci/0000:82:00.0 reporter fw |
| 334 |
|
| 335 |
.. note:: |
| 336 |
This command can run only on the PF which has fw tracer ownership, |
| 337 |
running it on other PF or any VF will return "Operation not permitted". |
| 338 |
|
| 339 |
fw fatal reporter |
| 340 |
----------------- |
| 341 |
The fw fatal reporter implements `dump` and `recover` callbacks. |
| 342 |
It follows fatal errors indications by CR-space dump and recover flow. |
| 343 |
The CR-space dump uses vsc interface which is valid even if the FW command |
| 344 |
interface is not functional, which is the case in most FW fatal errors. |
| 345 |
The recover function runs recover flow which reloads the driver and triggers fw |
| 346 |
reset if needed. |
| 347 |
On firmware error, the health buffer is dumped into the dmesg. The log |
| 348 |
level is derived from the error's severity (given in health buffer). |
| 349 |
|
| 350 |
User commands examples: |
| 351 |
|
| 352 |
- Run fw recover flow manually:: |
| 353 |
|
| 354 |
$ devlink health recover pci/0000:82:00.0 reporter fw_fatal |
| 355 |
|
| 356 |
- Read FW CR-space dump if already stored or trigger new one:: |
| 357 |
|
| 358 |
$ devlink health dump show pci/0000:82:00.1 reporter fw_fatal |
| 359 |
|
| 360 |
.. note:: |
| 361 |
This command can run only on PF. |
| 362 |
|
| 363 |
vnic reporter |
| 364 |
------------- |
| 365 |
The vnic reporter implements only the `diagnose` callback. |
| 366 |
It is responsible for querying the vnic diagnostic counters from fw and displaying |
| 367 |
them in realtime. |
| 368 |
|
| 369 |
Description of the vnic counters: |
| 370 |
|
| 371 |
- total_error_queues |
| 372 |
number of queues in an error state due to |
| 373 |
an async error or errored command. |
| 374 |
- send_queue_priority_update_flow |
| 375 |
number of QP/SQ priority/SL update events. |
| 376 |
- cq_overrun |
| 377 |
number of times CQ entered an error state due to an overflow. |
| 378 |
- async_eq_overrun |
| 379 |
number of times an EQ mapped to async events was overrun. |
| 380 |
- comp_eq_overrun |
| 381 |
number of times an EQ mapped to completion events was |
| 382 |
overrun. |
| 383 |
- quota_exceeded_command |
| 384 |
number of commands issued and failed due to quota exceeded. |
| 385 |
- invalid_command |
| 386 |
number of commands issued and failed dues to any reason other than quota |
| 387 |
exceeded. |
| 388 |
- nic_receive_steering_discard |
| 389 |
number of packets that completed RX flow |
| 390 |
steering but were discarded due to a mismatch in flow table. |
| 391 |
- generated_pkt_steering_fail |
| 392 |
number of packets generated by the VNIC experiencing unexpected steering |
| 393 |
failure (at any point in steering flow). |
| 394 |
- handled_pkt_steering_fail |
| 395 |
number of packets handled by the VNIC experiencing unexpected steering |
| 396 |
failure (at any point in steering flow owned by the VNIC, including the FDB |
| 397 |
for the eswitch owner). |
| 398 |
- icm_consumption |
| 399 |
amount of Interconnect Host Memory (ICM) consumed by the vnic in |
| 400 |
granularity of 4KB. ICM is host memory allocated by SW upon HCA request |
| 401 |
and is used for storing data structures that control HCA operation. |
| 402 |
- bar_uar_access |
| 403 |
number of WRITE or READ access operations to the UAR on the PCIe BAR. |
| 404 |
- odp_local_triggered_page_fault |
| 405 |
number of locally-triggered page-faults due to ODP. |
| 406 |
- odp_remote_triggered_page_fault |
| 407 |
number of remotly-triggered page-faults due to ODP. |
| 408 |
|
| 409 |
User commands examples: |
| 410 |
|
| 411 |
- Diagnose PF/VF vnic counters:: |
| 412 |
|
| 413 |
$ devlink health diagnose pci/0000:82:00.1 reporter vnic |
| 414 |
|
| 415 |
- Diagnose representor vnic counters (performed by supplying devlink port of the |
| 416 |
representor, which can be obtained via devlink port command):: |
| 417 |
|
| 418 |
$ devlink health diagnose pci/0000:82:00.1/65537 reporter vnic |
| 419 |
|
| 420 |
.. note:: |
| 421 |
This command can run over all interfaces such as PF/VF and representor ports. |