| 1 |
.. SPDX-License-Identifier: GPL-2.0+ |
| 2 |
|
| 3 |
============================================================== |
| 4 |
Linux kernel driver for Compute Engine Virtual Ethernet (gve): |
| 5 |
============================================================== |
| 6 |
|
| 7 |
Supported Hardware |
| 8 |
=================== |
| 9 |
The GVE driver binds to a single PCI device id used by the virtual |
| 10 |
Ethernet device found in some Compute Engine VMs. |
| 11 |
|
| 12 |
+--------------+----------+---------+ |
| 13 |
|Field | Value | Comments| |
| 14 |
+==============+==========+=========+ |
| 15 |
|Vendor ID | `0x1AE0` | Google | |
| 16 |
+--------------+----------+---------+ |
| 17 |
|Device ID | `0x0042` | | |
| 18 |
+--------------+----------+---------+ |
| 19 |
|Sub-vendor ID | `0x1AE0` | Google | |
| 20 |
+--------------+----------+---------+ |
| 21 |
|Sub-device ID | `0x0058` | | |
| 22 |
+--------------+----------+---------+ |
| 23 |
|Revision ID | `0x0` | | |
| 24 |
+--------------+----------+---------+ |
| 25 |
|Device Class | `0x200` | Ethernet| |
| 26 |
+--------------+----------+---------+ |
| 27 |
|
| 28 |
PCI Bars |
| 29 |
======== |
| 30 |
The gVNIC PCI device exposes three 32-bit memory BARS: |
| 31 |
- Bar0 - Device configuration and status registers. |
| 32 |
- Bar1 - MSI-X vector table |
| 33 |
- Bar2 - IRQ, RX and TX doorbells |
| 34 |
|
| 35 |
Device Interactions |
| 36 |
=================== |
| 37 |
The driver interacts with the device in the following ways: |
| 38 |
- Registers |
| 39 |
- A block of MMIO registers |
| 40 |
- See gve_register.h for more detail |
| 41 |
- Admin Queue |
| 42 |
- See description below |
| 43 |
- Reset |
| 44 |
- At any time the device can be reset |
| 45 |
- Interrupts |
| 46 |
- See supported interrupts below |
| 47 |
- Transmit and Receive Queues |
| 48 |
- See description below |
| 49 |
|
| 50 |
Descriptor Formats |
| 51 |
------------------ |
| 52 |
GVE supports two descriptor formats: GQI and DQO. These two formats have |
| 53 |
entirely different descriptors, which will be described below. |
| 54 |
|
| 55 |
Addressing Mode |
| 56 |
------------------ |
| 57 |
GVE supports two addressing modes: QPL and RDA. |
| 58 |
QPL ("queue-page-list") mode communicates data through a set of |
| 59 |
pre-registered pages. |
| 60 |
|
| 61 |
For RDA ("raw DMA addressing") mode, the set of pages is dynamic. |
| 62 |
Therefore, the packet buffers can be anywhere in guest memory. |
| 63 |
|
| 64 |
Registers |
| 65 |
--------- |
| 66 |
All registers are MMIO. |
| 67 |
|
| 68 |
The registers are used for initializing and configuring the device as well as |
| 69 |
querying device status in response to management interrupts. |
| 70 |
|
| 71 |
Endianness |
| 72 |
---------- |
| 73 |
- Admin Queue messages and registers are all Big Endian. |
| 74 |
- GQI descriptors and datapath registers are Big Endian. |
| 75 |
- DQO descriptors and datapath registers are Little Endian. |
| 76 |
|
| 77 |
Admin Queue (AQ) |
| 78 |
---------------- |
| 79 |
The Admin Queue is a PAGE_SIZE memory block, treated as an array of AQ |
| 80 |
commands, used by the driver to issue commands to the device and set up |
| 81 |
resources.The driver and the device maintain a count of how many commands |
| 82 |
have been submitted and executed. To issue AQ commands, the driver must do |
| 83 |
the following (with proper locking): |
| 84 |
|
| 85 |
1) Copy new commands into next available slots in the AQ array |
| 86 |
2) Increment its counter by he number of new commands |
| 87 |
3) Write the counter into the GVE_ADMIN_QUEUE_DOORBELL register |
| 88 |
4) Poll the ADMIN_QUEUE_EVENT_COUNTER register until it equals |
| 89 |
the value written to the doorbell, or until a timeout. |
| 90 |
|
| 91 |
The device will update the status field in each AQ command reported as |
| 92 |
executed through the ADMIN_QUEUE_EVENT_COUNTER register. |
| 93 |
|
| 94 |
Device Resets |
| 95 |
------------- |
| 96 |
A device reset is triggered by writing 0x0 to the AQ PFN register. |
| 97 |
This causes the device to release all resources allocated by the |
| 98 |
driver, including the AQ itself. |
| 99 |
|
| 100 |
Interrupts |
| 101 |
---------- |
| 102 |
The following interrupts are supported by the driver: |
| 103 |
|
| 104 |
Management Interrupt |
| 105 |
~~~~~~~~~~~~~~~~~~~~ |
| 106 |
The management interrupt is used by the device to tell the driver to |
| 107 |
look at the GVE_DEVICE_STATUS register. |
| 108 |
|
| 109 |
The handler for the management irq simply queues the service task in |
| 110 |
the workqueue to check the register and acks the irq. |
| 111 |
|
| 112 |
Notification Block Interrupts |
| 113 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 114 |
The notification block interrupts are used to tell the driver to poll |
| 115 |
the queues associated with that interrupt. |
| 116 |
|
| 117 |
The handler for these irqs schedule the napi for that block to run |
| 118 |
and poll the queues. |
| 119 |
|
| 120 |
GQI Traffic Queues |
| 121 |
------------------ |
| 122 |
GQI queues are composed of a descriptor ring and a buffer and are assigned to a |
| 123 |
notification block. |
| 124 |
|
| 125 |
The descriptor rings are power-of-two-sized ring buffers consisting of |
| 126 |
fixed-size descriptors. They advance their head pointer using a __be32 |
| 127 |
doorbell located in Bar2. The tail pointers are advanced by consuming |
| 128 |
descriptors in-order and updating a __be32 counter. Both the doorbell |
| 129 |
and the counter overflow to zero. |
| 130 |
|
| 131 |
Each queue's buffers must be registered in advance with the device as a |
| 132 |
queue page list, and packet data can only be put in those pages. |
| 133 |
|
| 134 |
Transmit |
| 135 |
~~~~~~~~ |
| 136 |
gve maps the buffers for transmit rings into a FIFO and copies the packets |
| 137 |
into the FIFO before sending them to the NIC. |
| 138 |
|
| 139 |
Receive |
| 140 |
~~~~~~~ |
| 141 |
The buffers for receive rings are put into a data ring that is the same |
| 142 |
length as the descriptor ring and the head and tail pointers advance over |
| 143 |
the rings together. |
| 144 |
|
| 145 |
DQO Traffic Queues |
| 146 |
------------------ |
| 147 |
- Every TX and RX queue is assigned a notification block. |
| 148 |
|
| 149 |
- TX and RX buffers queues, which send descriptors to the device, use MMIO |
| 150 |
doorbells to notify the device of new descriptors. |
| 151 |
|
| 152 |
- RX and TX completion queues, which receive descriptors from the device, use a |
| 153 |
"generation bit" to know when a descriptor was populated by the device. The |
| 154 |
driver initializes all bits with the "current generation". The device will |
| 155 |
populate received descriptors with the "next generation" which is inverted |
| 156 |
from the current generation. When the ring wraps, the current/next generation |
| 157 |
are swapped. |
| 158 |
|
| 159 |
- It's the driver's responsibility to ensure that the RX and TX completion |
| 160 |
queues are not overrun. This can be accomplished by limiting the number of |
| 161 |
descriptors posted to HW. |
| 162 |
|
| 163 |
- TX packets have a 16 bit completion_tag and RX buffers have a 16 bit |
| 164 |
buffer_id. These will be returned on the TX completion and RX queues |
| 165 |
respectively to let the driver know which packet/buffer was completed. |
| 166 |
|
| 167 |
Transmit |
| 168 |
~~~~~~~~ |
| 169 |
A packet's buffers are DMA mapped for the device to access before transmission. |
| 170 |
After the packet was successfully transmitted, the buffers are unmapped. |
| 171 |
|
| 172 |
Receive |
| 173 |
~~~~~~~ |
| 174 |
The driver posts fixed sized buffers to HW on the RX buffer queue. The packet |
| 175 |
received on the associated RX queue may span multiple descriptors. |