[[s-using-drbd-proxy]] == Using DRBD Proxy [[s-drbd-proxy-deployment-considerations]] === DRBD Proxy deployment considerations The <> processes can either be located directly on the machines where DRBD is set up, or they can be placed on distinct dedicated servers. A DRBD Proxy instance can serve as a proxy for multiple DRBD devices distributed across multiple nodes. DRBD Proxy is completely transparent to DRBD. Typically you will expect a high number of data packets in flight, therefore the activity log should be reasonably large. Since this may cause longer re-sync runs after the crash of a primary node, it is recommended to enable DRBD's `csums-alg` setting. For more information about the rationale for the DRBD Proxy, please see the feature explanation <>. The DRBD Proxy 3 uses several kernel features that are only available since 2.6.26, so running it on older systems (e.g. RHEL 5) is not possible; here we can still provide DRBD Proxy 1 packages, thoughfootnote:[The v1 uses a different scheduling model and will therefore not reach the same performance as v3; so even if your production setup is still RHEL 5, perhaps you can run one RHEL 6/7 VM in each data center?]. [[s-drbd-proxy-installation]] === Installation To obtain DRBD Proxy, please contact your Linbit sales representative. Unless instructed otherwise, please always use the most recent DRBD Proxy release. To install DRBD Proxy on Debian and Debian-based systems, use the dpkg tool as follows (replace version with your DRBD Proxy version, and architecture with your target architecture): ---------------------------- # dpkg -i drbd-proxy_3.2.2_amd64.deb ---------------------------- To install DRBD Proxy on RPM based systems (like SLES or RHEL) use the rpm tool as follows (replace version with your DRBD Proxy version, and architecture with your target architecture): ---------------------------- # rpm -i drbd-proxy-3.2.2-1.x86_64.rpm ---------------------------- Also install the DRBD administration program drbdadm since it is required to configure DRBD Proxy. This will install the DRBD proxy binaries as well as an init script which usually goes into `/etc/init.d`. Please always use the init script to start/stop DRBD proxy since it also configures DRBD Proxy using the `drbdadm` tool. [[s-drbd-proxy-license]] === License file When obtaining a license from Linbit, you will be sent a DRBD Proxy license file which is required to run DRBD Proxy. The file is called `drbd-proxy.license`, it must be copied into the `/etc` directory of the target machines, and be owned by the user/group `drbdpxy`. ---------------------------- # cp drbd-proxy.license /etc/ ---------------------------- [[s-drbd-proxy-configuration-linstor]] === Configuration using LINSTOR DRBD Proxy can be configured using LINSTOR as described in the LINSTOR USER'S GUIDE. [[s-drbd-proxy-configuration]] === Configuration using resource files DRBD Proxy can also be configured by editing resource files. It is configured by an additional section called `proxy` and additional `proxy on` sections within the host sections. Below is a DRBD configuration example for proxies running directly on the DRBD nodes: [source,drbd] ---------------------------- resource r0 { protocol A; device /dev/drbd15; disk /dev/VG/r0; meta-disk internal; proxy { memlimit 512M; plugin { zlib level 9; } } on alice { address 127.0.0.1:7915; proxy on alice { inside 127.0.0.1:7815; outside 192.168.23.1:7715; } } on bob { address 127.0.0.1:7915; proxy on bob { inside 127.0.0.1:7815; outside 192.168.23.2:7715; } } } ---------------------------- The `inside` IP address is used for communication between DRBD and the DRBD Proxy, whereas the `outside` IP address is used for communication between the proxies. The latter channel might have to be allowed in your firewall setup. [[s-drbd-proxy-controlling]] === Controlling DRBD Proxy `drbdadm` offers the `proxy-up` and `proxy-down` subcommands to configure or delete the connection to the local DRBD Proxy process of the named DRBD resource(s). These commands are used by the `start` and `stop` actions which `/etc/init.d/drbdproxy` implements. The DRBD Proxy has a low level configuration tool, called `drbd-proxy-ctl`. When called without any option it operates in interactive mode. To pass a command directly, avoiding interactive mode, use the `-c` parameter followed by the command. To display the available commands use: ---------------------------- # drbd-proxy-ctl -c "help" ---------------------------- Note the double quotes around the command being passed. Here is a list of commands; while the first few ones are typically only used indirectly (via `drbdadm proxy-up` resp. `drbdadm proxy-down`), the latter ones give various status information. .`add connection __ _lots of arguments_` Creates a communication path. As this is run via `drbdadm proxy-up` the long list of arguments is omitted here. .`del connection __` Removes a communication path. .`set memlimit __ __` Sets the memory limit for a connection; this can only be done when setting it up afresh, changing it during runtime is not possible. + This command understands the usual units `k`, `M`, and `G`. .`show` Shows currently configured communication paths. .`show memusage` Shows memory usage of each connection. + -- As an example, -------- # watch -n 1 'drbd-proxy-ctl -c "show memusage"' -------- monitors memory usage. Please note that the quotes are required as listed above. -- .`show [h]subconnections` Shows currently established individual connections together with some stats. With `h` outputs bytes in human readable format. .`show [h]connections` Shows currently configured connections and their states With `h` outputs bytes in human readable format. + -- The column `Status` will show one of these states: * _Off_: No communication to the remote DRBD Proxy process. * _Half-up_: The connection to the remote DRBD Proxy could be established; the Proxy => DRBD paths are not up yet. * _DRBD-conn_: The first few packets are being pushed across the connection; but still e.g. a Split-Brain situation might serve it again. * _Up_: The DRBD connection is fully established. -- .`shutdown` Shuts down the `drbd-proxy` program. Attention: this unconditionally terminates any DRBD connections using the DRBD proxy. .`quit` Exits the client program (closes the control connection), but leaves the DRBD proxy running. .`print statistics` This prints detailed statistics for the currently active connections, in an easily parseable format. Use this for integration to your monitoring solution! + NOTE: While the commands above are only accepted from UID 0 (i.e., the `root` user), this one can be used by any user (provided that unix permissions allow access on the proxy socket at `/var/run/drbd-proxy/drbd-proxy-ctl.socket`); see the init script at `/etc/init.d/drbdproxy` about setting the rights. [[s-drbd-proxy-plugins]] === About DRBD Proxy plugins Since DRBD Proxy version 3 the proxy allows to enable a few specific plugins for the WAN connection. + The currently available plugins are `zstd`, `lz4`, `zlib` and `lzma` (all software compression), and `aha` (hardware compression support, see http://www.aha.com/data-compression/). `zstd` (Zstandard) is a real-time compression algorithm, providing high compression ratios. It offers a very wide range of compression / speed trade-off, while being backed by a very fast decoder. Compression rates are dependent on "level" parameter which can be arranged between 1 to 22. Over level 20, Drbd Proxy will require more memory. `lz4` is a very fast compression algorithm; the data typically gets compressed down by 1:2 to 1:4, half- to two-thirds of the bandwidth can be saved. The `zlib` plugin uses the GZIP algorithm for compression; it uses a bit more CPU than `lz4`, but gives a ratio of 1:3 to 1:5. The `lzma` plugin uses the `liblzma2` library. It can use dictionaries of several hundred MiB; these allow for very efficient delta-compression of repeated data, even for small changes. `lzma` needs much more CPU and memory, but results in much better compression than `zlib` -- real-world tests with a VM sitting on top of DRBD gave ratios of 1:10 to 1:40. The `lzma` plugin has to be enabled in your license. `aha` uses hardware compression cards, like the AHA367PCIe (10Gbit/sec) or AHA372 (20GBit/sec); this is the fastest compression for contemporary hardware. + You will need a special flag in your license file to enable this plugin. Please contact LINBIT to find the best settings for your environment - it depends on the CPU (speed, number of threads), available memory, input and available output bandwidth, and expected IO spikes. Having a week of `sysstat` data already available helps in determining the configuration, too. Please note that the older `compression on` in the `proxy` section is deprecated, and will be removed in a future release. + Currently it is treated as `zlib level 9`. [[s-drbd-proxy-bwlimit]] ==== Using a WAN Side Bandwidth Limit The experimental `bwlimit` option of DRBD Proxy is broken. Do not use it, as it may cause applications on DRBD to block on IO. It will be removed. Instead use the Linux kernel's traffic control framework to limit bandwidth consumed by proxy on the WAN side. In the following example you would need to replace the interface name, the source port and the ip address of the peer. ---------------------------- # tc qdisc add dev eth0 root handle 1: htb default 1 # tc class add dev eth0 parent 1: classid 1:1 htb rate 1gbit # tc class add dev eth0 parent 1:1 classid 1:10 htb rate 500kbit # tc filter add dev eth0 parent 1: protocol ip prio 16 u32 \ match ip sport 7000 0xffff \ match ip dst 192.168.47.11 flowid 1:10 # tc filter add dev eth0 parent 1: protocol ip prio 16 u32 \ match ip dport 7000 0xffff \ match ip dst 192.168.47.11 flowid 1:10 ---------------------------- You can remove this bandwidth limitation with ---------------------------- # tc qdisc del dev eth0 root handle 1 ---------------------------- [[s-drbd-proxy-troubleshoot]] === Troubleshooting DRBD proxy logs via syslog using the `LOG_DAEMON` facility. Usually you will find DRBD Proxy messages in `/var/log/daemon.log`. Enabling debug mode in DRBD Proxy can be done with the following command. -------------------------- # drbd-proxy-ctl -c 'set loglevel debug' -------------------------- For example, if proxy fails to connect it will log something like `Rejecting connection because I can't connect on the other side`. In that case, please check if DRBD is running (not in StandAlone mode) on both nodes and if both proxies are running. Also double-check your configuration.