| 1 |
# The dpdkstat plugin |
| 2 |
|
| 3 |
This plugin is optional and only has a specific use case: monitoring DPDK applications |
| 4 |
that don't expose stats in any other way than the DPDK xstats API. |
| 5 |
|
| 6 |
**Data Plane Development Kit** (DPDK) is a set of drivers and libraries for fast |
| 7 |
packet processing. Please note that this plugin is a polling based plugin rather |
| 8 |
than an events based plugin (using it will drive up core utilization on a system). |
| 9 |
|
| 10 |
**PLEASE DO NOT USE THIS PLUGIN FOR OVS-DPDK**. dpdkstat is really for DPDK |
| 11 |
applications that have no other way of exposing stats. For OVS or OVS-with-DPDK the |
| 12 |
Open vSwitch plugins available in collectd 5.8.0 should be used for |
| 13 |
collecting stats and events. In addition the OVS plugin is events based rather |
| 14 |
than polling based and will have a smaller footprint on the system. |
| 15 |
|
| 16 |
## Summary |
| 17 |
|
| 18 |
The *dpdkstat plugin* has the following requirements: |
| 19 |
|
| 20 |
* DPDK 16.04 or later |
| 21 |
* GCC 4.9 or later |
| 22 |
|
| 23 |
You can also build with GCC 4.8 (e.g. Ubuntu 14.04) if you specify the SSSE3 |
| 24 |
instruction set manually: |
| 25 |
|
| 26 |
make -j CFLAGS+='-mssse3' |
| 27 |
|
| 28 |
## Building DPDK |
| 29 |
|
| 30 |
* Setup the build environment: |
| 31 |
|
| 32 |
Ensure that you have GCC 4.9 or later. Ubuntu 14.04, for example, has GCC |
| 33 |
4.8 by default and requires an upgrade: |
| 34 |
|
| 35 |
add-apt-repository ppa:ubuntu-toolchain-r/test |
| 36 |
apt-get update |
| 37 |
apt-get install gcc-4.9 |
| 38 |
|
| 39 |
If you know that the platform that you wish to run collectd on supports the |
| 40 |
SSSE3 instruction set, GCC 4.8 also works if you enable SSSE3 manually: |
| 41 |
|
| 42 |
make -j CFLAGS+='-mssse3' |
| 43 |
|
| 44 |
* Clone DPDK: |
| 45 |
|
| 46 |
git clone git://dpdk.org/dpdk |
| 47 |
|
| 48 |
* Checkout the [DPDK system |
| 49 |
requirements](http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html) and make |
| 50 |
sure you have the required tools and hugepage setup as specified there. |
| 51 |
|
| 52 |
**Note:** It's recommended to use the 1GB hugepage setup for best |
| 53 |
performance, please follow the instruction for "Reserving Hugepages for DPDK |
| 54 |
Use" in the link above. |
| 55 |
|
| 56 |
However if you plan on configuring 2MB hugepages on the fly please ensure to |
| 57 |
add appropriate commands to reserve hugepages in a system startup script if |
| 58 |
collectd is booted at system startup time. These commands include: |
| 59 |
|
| 60 |
mkdir -p /mnt/huge |
| 61 |
mount -t hugetlbfs nodev /mnt/huge |
| 62 |
echo 64 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages |
| 63 |
|
| 64 |
* To configure the DPDK build for the combined shared library and enable autoload |
| 65 |
of pmd drivers modify `config/common_base` in your DPDK as follows |
| 66 |
|
| 67 |
# |
| 68 |
# Compile to share library |
| 69 |
# |
| 70 |
-CONFIG_RTE_BUILD_SHARED_LIB=n |
| 71 |
+CONFIG_RTE_BUILD_SHARED_LIB=y |
| 72 |
|
| 73 |
# Default driver path (or "" to disable) |
| 74 |
-CONFIG_RTE_EAL_PMD_PATH="" |
| 75 |
+CONFIG_RTE_EAL_PMD_PATH="/usr/lib/dpdk-pmd/" |
| 76 |
|
| 77 |
* Prepare the configuration for the appropriate target as specified at: |
| 78 |
http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html. |
| 79 |
|
| 80 |
For example: |
| 81 |
|
| 82 |
make config T=x86_64-native-linuxapp-gcc |
| 83 |
|
| 84 |
* Build the target: |
| 85 |
|
| 86 |
make |
| 87 |
|
| 88 |
* Install DPDK to `/usr` |
| 89 |
|
| 90 |
sudo make install prefix=/usr |
| 91 |
|
| 92 |
* Create dpdk-pmd folder |
| 93 |
|
| 94 |
mkdir -p /usr/lib/dpdk-pmd |
| 95 |
|
| 96 |
* Create symlinks to pmd drivers |
| 97 |
|
| 98 |
find /usr/lib -type f -name 'librte_pmd*' | while read path ; do ln -s $path /usr/lib/dpdk-pmd/`echo $path | grep -o 'librte_.*so'` ; done |
| 99 |
|
| 100 |
**Note 1:** You must run make install as the configuration of collectd with |
| 101 |
DPDK expects DPDK to be installed somewhere. |
| 102 |
|
| 103 |
**Note 2:** If you don't specify a prefix then DPDK will be installed in |
| 104 |
`/usr/local/`. |
| 105 |
|
| 106 |
**Note 3:** If you are not root then use sudo to make install DPDK to the |
| 107 |
appropriate location. |
| 108 |
|
| 109 |
**Note 4:** You **MUST** create symlink to a NIC driver lib. This way collectd |
| 110 |
will be able to work with device bound to dpdk. |
| 111 |
|
| 112 |
* Check that the DPDK library has been installed in `/usr/lib` or `/lib`: |
| 113 |
|
| 114 |
ls /usr/lib | grep dpdk |
| 115 |
|
| 116 |
* Bind the interfaces to use with dpdkstat to DPDK: |
| 117 |
|
| 118 |
DPDK devices can be setup with either the VFIO (for DPDK 1.7+) or UIO |
| 119 |
modules. |
| 120 |
|
| 121 |
**Note:** UIO requires inserting an out of tree driver `igb_uio.ko` that is |
| 122 |
available in DPDK. |
| 123 |
|
| 124 |
**UIO Setup:** |
| 125 |
|
| 126 |
* Insert `uio.ko`: |
| 127 |
|
| 128 |
sudo modprobe uio |
| 129 |
|
| 130 |
* Insert `igb_uio.ko`: |
| 131 |
|
| 132 |
sudo insmod $DPDK_BUILD/kmod/igb_uio.ko |
| 133 |
|
| 134 |
* Bind network device to `igb_uio`: |
| 135 |
|
| 136 |
sudo $DPDK_DIR/tools/dpdk_nic_bind.py --bind=igb_uio eth1 |
| 137 |
|
| 138 |
**VFIO Setup:** |
| 139 |
|
| 140 |
* VFIO needs to be supported in the kernel and the BIOS. More information |
| 141 |
can be found at: http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html. |
| 142 |
* Insert the `vfio-pci.ko` module: |
| 143 |
|
| 144 |
modprobe vfio-pci |
| 145 |
|
| 146 |
* Set the correct permissions for the VFIO device: |
| 147 |
|
| 148 |
sudo /usr/bin/chmod a+x /dev/vfio |
| 149 |
sudo /usr/bin/chmod 0666 /dev/vfio/* |
| 150 |
|
| 151 |
* Bind the network device to `vfio-pci`: |
| 152 |
|
| 153 |
sudo $DPDK_DIR/tools/dpdk_nic_bind.py --bind=vfio-pci eth1 |
| 154 |
|
| 155 |
**Note:** Please ensure to add appropriate commands to bind the network |
| 156 |
interfaces to DPDK in a system startup script if collectd is booted at |
| 157 |
system startup time. |
| 158 |
|
| 159 |
* Run `ldconfig` to update the shared library cache. |
| 160 |
|
| 161 |
## Build collectd with DPDK |
| 162 |
|
| 163 |
**Note:** DPDK 16.04 is the minimum version and currently supported version of |
| 164 |
DPDK required for the dpdkstat plugin. This is to allow the plugin to take |
| 165 |
advantage of functions added to detect if the DPDK primary process is alive. |
| 166 |
|
| 167 |
|
| 168 |
**Note:** The *Address-Space Layout Randomization* (ASLR) security feature in |
| 169 |
Linux should be disabled, in order for the same hugepage memory mappings to be |
| 170 |
present in all DPDK multi-process applications. Note that this has security |
| 171 |
implications. |
| 172 |
|
| 173 |
* To disable ASLR: |
| 174 |
|
| 175 |
echo 0 > /proc/sys/kernel/randomize_va_space |
| 176 |
|
| 177 |
* To fully enable ASLR: |
| 178 |
|
| 179 |
echo 2 > /proc/sys/kernel/randomize_va_space |
| 180 |
|
| 181 |
See also: http://dpdk.org/doc/guides/prog_guide/multi_proc_support.html |
| 182 |
|
| 183 |
* Generate the build script as specified below. (i.e. run `build.sh`). |
| 184 |
* Configure collectd with the DPDK shared library. If DPDK is installed in |
| 185 |
custom installation path you can specify headers include path using |
| 186 |
LIBDPDK_CPPFLAGS variable and libraries path with LIBDPDK_LDFLAGS. |
| 187 |
Example: |
| 188 |
|
| 189 |
./configure |
| 190 |
|
| 191 |
or for custom DPKD installation: |
| 192 |
|
| 193 |
./configure LIBDPDK_CPPFLAGS="-I/home/joe/include/dpdk" LIBDPDK_LDFLAGS="-L/home/joe/usr/lib" |
| 194 |
|
| 195 |
* Make sure that libdpdk and dpdkstat are enabled in the *configure* output. |
| 196 |
|
| 197 |
Expected output: |
| 198 |
|
| 199 |
Libraries: |
| 200 |
... |
| 201 |
libdpdk . . . . . . . . yes |
| 202 |
|
| 203 |
Modules: |
| 204 |
... |
| 205 |
dpdkstat . . . . . . .yes |
| 206 |
|
| 207 |
* Build collectd: |
| 208 |
|
| 209 |
make -j && make -j install |
| 210 |
|
| 211 |
**Note:** As mentioned above, if you are building on Ubuntu 14.04 with |
| 212 |
GCC <= 4.8.X, you need to use: |
| 213 |
|
| 214 |
make -j CFLAGS+='-mssse3' && make -j install |
| 215 |
|
| 216 |
## Caveats |
| 217 |
|
| 218 |
* The same PCI device configuration should be passed to the primary process as |
| 219 |
the secondary process uses the same port indexes as the primary. |
| 220 |
* A blacklist / whitelist of NICs isn't supported yet. |
| 221 |
* Plugin initialization time depends on read interval. It requires 5 read |
| 222 |
cycles to set up internal buffers and states. During that time no statistics |
| 223 |
are submitted. |
| 224 |
* If number of DPDK ports is increased while plugin is running, internal |
| 225 |
buffers are resized. That requires 3 read cycles and no port statistics |
| 226 |
are submitted in that time. |
| 227 |
|
| 228 |
## License |
| 229 |
|
| 230 |
The *dpdkstat plugin* is copyright (c) 2016 *Intel Corporation* and licensed |
| 231 |
under the *MIT license*. Full licensing terms can be found in the file |
| 232 |
`COPYING`. |