| 1 |
.. SPDX-License-Identifier: GPL-2.0 |
| 2 |
|
| 3 |
============== |
| 4 |
Devlink Health |
| 5 |
============== |
| 6 |
|
| 7 |
Background |
| 8 |
========== |
| 9 |
|
| 10 |
The ``devlink`` health mechanism is targeted for Real Time Alerting, in |
| 11 |
order to know when something bad happened to a PCI device. |
| 12 |
|
| 13 |
* Provide alert debug information. |
| 14 |
* Self healing. |
| 15 |
* If problem needs vendor support, provide a way to gather all needed |
| 16 |
debugging information. |
| 17 |
|
| 18 |
Overview |
| 19 |
======== |
| 20 |
|
| 21 |
The main idea is to unify and centralize driver health reports in the |
| 22 |
generic ``devlink`` instance and allow the user to set different |
| 23 |
attributes of the health reporting and recovery procedures. |
| 24 |
|
| 25 |
The ``devlink`` health reporter: |
| 26 |
Device driver creates a "health reporter" per each error/health type. |
| 27 |
Error/Health type can be a known/generic (e.g. PCI error, fw error, rx/tx error) |
| 28 |
or unknown (driver specific). |
| 29 |
For each registered health reporter a driver can issue error/health reports |
| 30 |
asynchronously. All health reports handling is done by ``devlink``. |
| 31 |
Device driver can provide specific callbacks for each "health reporter", e.g.: |
| 32 |
|
| 33 |
* Recovery procedures |
| 34 |
* Diagnostics procedures |
| 35 |
* Object dump procedures |
| 36 |
* Out Of Box initial parameters |
| 37 |
|
| 38 |
Different parts of the driver can register different types of health reporters |
| 39 |
with different handlers. |
| 40 |
|
| 41 |
Actions |
| 42 |
======= |
| 43 |
|
| 44 |
Once an error is reported, devlink health will perform the following actions: |
| 45 |
|
| 46 |
* A log is being send to the kernel trace events buffer |
| 47 |
* Health status and statistics are being updated for the reporter instance |
| 48 |
* Object dump is being taken and saved at the reporter instance (as long as |
| 49 |
auto-dump is set and there is no other dump which is already stored) |
| 50 |
* Auto recovery attempt is being done. Depends on: |
| 51 |
|
| 52 |
- Auto-recovery configuration |
| 53 |
- Grace period (and burst period) vs. time passed since last recover |
| 54 |
|
| 55 |
Devlink formatted message |
| 56 |
========================= |
| 57 |
|
| 58 |
To handle devlink health diagnose and health dump requests, devlink creates a |
| 59 |
formatted message structure ``devlink_fmsg`` and send it to the driver's callback |
| 60 |
to fill the data in using the devlink fmsg API. |
| 61 |
|
| 62 |
Devlink fmsg is a mechanism to pass descriptors between drivers and devlink, in |
| 63 |
json-like format. The API allows the driver to add nested attributes such as |
| 64 |
object, object pair and value array, in addition to attributes such as name and |
| 65 |
value. |
| 66 |
|
| 67 |
Driver should use this API to fill the fmsg context in a format which will be |
| 68 |
translated by the devlink to the netlink message later. When it needs to send |
| 69 |
the data using SKBs to the netlink layer, it fragments the data between |
| 70 |
different SKBs. In order to do this fragmentation, it uses virtual nests |
| 71 |
attributes, to avoid actual nesting use which cannot be divided between |
| 72 |
different SKBs. |
| 73 |
|
| 74 |
User Interface |
| 75 |
============== |
| 76 |
|
| 77 |
User can access/change each reporter's parameters and driver specific callbacks |
| 78 |
via ``devlink``, e.g per error type (per health reporter): |
| 79 |
|
| 80 |
* Configure reporter's generic parameters (like: disable/enable auto recovery) |
| 81 |
* Invoke recovery procedure |
| 82 |
* Run diagnostics |
| 83 |
* Object dump |
| 84 |
|
| 85 |
.. list-table:: List of devlink health interfaces |
| 86 |
:widths: 10 90 |
| 87 |
|
| 88 |
* - Name |
| 89 |
- Description |
| 90 |
* - ``DEVLINK_CMD_HEALTH_REPORTER_GET`` |
| 91 |
- Retrieves status and configuration info per DEV and reporter. |
| 92 |
* - ``DEVLINK_CMD_HEALTH_REPORTER_SET`` |
| 93 |
- Allows reporter-related configuration setting. |
| 94 |
* - ``DEVLINK_CMD_HEALTH_REPORTER_RECOVER`` |
| 95 |
- Triggers reporter's recovery procedure. |
| 96 |
* - ``DEVLINK_CMD_HEALTH_REPORTER_TEST`` |
| 97 |
- Triggers a fake health event on the reporter. The effects of the test |
| 98 |
event in terms of recovery flow should follow closely that of a real |
| 99 |
event. |
| 100 |
* - ``DEVLINK_CMD_HEALTH_REPORTER_DIAGNOSE`` |
| 101 |
- Retrieves current device state related to the reporter. |
| 102 |
* - ``DEVLINK_CMD_HEALTH_REPORTER_DUMP_GET`` |
| 103 |
- Retrieves the last stored dump. Devlink health |
| 104 |
saves a single dump. If an dump is not already stored by devlink |
| 105 |
for this reporter, devlink generates a new dump. |
| 106 |
Dump output is defined by the reporter. |
| 107 |
* - ``DEVLINK_CMD_HEALTH_REPORTER_DUMP_CLEAR`` |
| 108 |
- Clears the last saved dump file for the specified reporter. |
| 109 |
|
| 110 |
The following diagram provides a general overview of ``devlink-health``:: |
| 111 |
|
| 112 |
netlink |
| 113 |
+--------------------------+ |
| 114 |
| | |
| 115 |
| + | |
| 116 |
| | | |
| 117 |
+--------------------------+ |
| 118 |
|request for ops |
| 119 |
|(diagnose, |
| 120 |
driver devlink |recover, |
| 121 |
|dump) |
| 122 |
+--------+ +--------------------------+ |
| 123 |
| | | reporter| | |
| 124 |
| | | +---------v----------+ | |
| 125 |
| | ops execution | | | | |
| 126 |
| <----------------------------------+ | | |
| 127 |
| | | | | | |
| 128 |
| | | + ^------------------+ | |
| 129 |
| | | | request for ops | |
| 130 |
| | | | (recover, dump) | |
| 131 |
| | | | | |
| 132 |
| | | +-+------------------+ | |
| 133 |
| | health report | | health handler | | |
| 134 |
| +-------------------------------> | | |
| 135 |
| | | +--------------------+ | |
| 136 |
| | health reporter create | | |
| 137 |
| +----------------------------> | |
| 138 |
+--------+ +--------------------------+ |