| 1 |
[[s-using-drbd-proxy]] |
| 2 |
== Using DRBD Proxy |
| 3 |
|
| 4 |
[[s-drbd-proxy-deployment-considerations]] |
| 5 |
=== DRBD Proxy deployment considerations |
| 6 |
|
| 7 |
The <<s-drbd-proxy,DRBD Proxy>> processes can either be located |
| 8 |
directly on the machines where DRBD is set up, or they can be placed |
| 9 |
on distinct dedicated servers. A DRBD Proxy instance can serve as a |
| 10 |
proxy for multiple DRBD devices distributed across multiple nodes. |
| 11 |
|
| 12 |
DRBD Proxy is completely transparent to DRBD. Typically you will |
| 13 |
expect a high number of data packets in flight, therefore the activity |
| 14 |
log should be reasonably large. Since this may cause longer re-sync |
| 15 |
runs after the crash of a primary node, it is recommended to enable |
| 16 |
DRBD's `csums-alg` setting. |
| 17 |
|
| 18 |
For more information about the rationale for the DRBD Proxy, please |
| 19 |
see the feature explanation <<s-drbd-proxy>>. |
| 20 |
|
| 21 |
The DRBD Proxy 3 uses several kernel features that are only available since |
| 22 |
2.6.26, so running it on older systems (e.g. RHEL 5) is not possible; here |
| 23 |
we can still provide DRBD Proxy 1 packages, thoughfootnote:[The v1 uses |
| 24 |
a different scheduling model and will therefore not reach the same performance |
| 25 |
as v3; so even if your production setup is still RHEL 5, perhaps you can run |
| 26 |
one RHEL 6/7 VM in each data center?]. |
| 27 |
|
| 28 |
|
| 29 |
[[s-drbd-proxy-installation]] |
| 30 |
=== Installation |
| 31 |
|
| 32 |
To obtain DRBD Proxy, please contact your Linbit sales |
| 33 |
representative. Unless instructed otherwise, please always use the |
| 34 |
most recent DRBD Proxy release. |
| 35 |
|
| 36 |
To install DRBD Proxy on Debian and Debian-based systems, use the dpkg |
| 37 |
tool as follows (replace version with your DRBD Proxy version, and |
| 38 |
architecture with your target architecture): |
| 39 |
|
| 40 |
---------------------------- |
| 41 |
# dpkg -i drbd-proxy_3.2.2_amd64.deb |
| 42 |
---------------------------- |
| 43 |
|
| 44 |
To install DRBD Proxy on RPM based systems (like SLES or RHEL) use |
| 45 |
the rpm tool as follows (replace version with your DRBD Proxy version, |
| 46 |
and architecture with your target architecture): |
| 47 |
|
| 48 |
---------------------------- |
| 49 |
# rpm -i drbd-proxy-3.2.2-1.x86_64.rpm |
| 50 |
---------------------------- |
| 51 |
|
| 52 |
Also install the DRBD administration program drbdadm since it is |
| 53 |
required to configure DRBD Proxy. |
| 54 |
|
| 55 |
This will install the DRBD proxy binaries as well as an init script |
| 56 |
which usually goes into `/etc/init.d`. Please always use the init |
| 57 |
script to start/stop DRBD proxy since it also configures DRBD Proxy |
| 58 |
using the `drbdadm` tool. |
| 59 |
|
| 60 |
[[s-drbd-proxy-license]] |
| 61 |
=== License file |
| 62 |
|
| 63 |
When obtaining a license from Linbit, you will be sent a DRBD Proxy |
| 64 |
license file which is required to run DRBD Proxy. The file is called |
| 65 |
`drbd-proxy.license`, it must be copied into the `/etc` directory of the |
| 66 |
target machines, and be owned by the user/group `drbdpxy`. |
| 67 |
|
| 68 |
---------------------------- |
| 69 |
# cp drbd-proxy.license /etc/ |
| 70 |
---------------------------- |
| 71 |
|
| 72 |
[[s-drbd-proxy-configuration-linstor]] |
| 73 |
=== Configuration using LINSTOR |
| 74 |
|
| 75 |
DRBD Proxy can be configured using LINSTOR as described in the LINSTOR |
| 76 |
USER'S GUIDE. |
| 77 |
|
| 78 |
[[s-drbd-proxy-configuration]] |
| 79 |
=== Configuration using resource files |
| 80 |
|
| 81 |
DRBD Proxy can also be configured by editing resource files. |
| 82 |
It is configured by an additional section called `proxy` and |
| 83 |
additional `proxy on` sections within the host sections. |
| 84 |
|
| 85 |
Below is a DRBD configuration example for proxies running directly on |
| 86 |
the DRBD nodes: |
| 87 |
|
| 88 |
[source,drbd] |
| 89 |
---------------------------- |
| 90 |
resource r0 { |
| 91 |
protocol A; |
| 92 |
device /dev/drbd15; |
| 93 |
disk /dev/VG/r0; |
| 94 |
meta-disk internal; |
| 95 |
|
| 96 |
proxy { |
| 97 |
memlimit 512M; |
| 98 |
plugin { |
| 99 |
zlib level 9; |
| 100 |
} |
| 101 |
} |
| 102 |
|
| 103 |
on alice { |
| 104 |
address 127.0.0.1:7915; |
| 105 |
proxy on alice { |
| 106 |
inside 127.0.0.1:7815; |
| 107 |
outside 192.168.23.1:7715; |
| 108 |
} |
| 109 |
} |
| 110 |
|
| 111 |
on bob { |
| 112 |
address 127.0.0.1:7915; |
| 113 |
proxy on bob { |
| 114 |
inside 127.0.0.1:7815; |
| 115 |
outside 192.168.23.2:7715; |
| 116 |
} |
| 117 |
} |
| 118 |
} |
| 119 |
---------------------------- |
| 120 |
|
| 121 |
The `inside` IP address is used for communication between DRBD and the |
| 122 |
DRBD Proxy, whereas the `outside` IP address is used for communication |
| 123 |
between the proxies. The latter channel might have to be allowed in your |
| 124 |
firewall setup. |
| 125 |
|
| 126 |
[[s-drbd-proxy-controlling]] |
| 127 |
=== Controlling DRBD Proxy |
| 128 |
|
| 129 |
`drbdadm` offers the `proxy-up` and `proxy-down` subcommands to |
| 130 |
configure or delete the connection to the local DRBD Proxy process of |
| 131 |
the named DRBD resource(s). These commands are used by the `start` and |
| 132 |
`stop` actions which `/etc/init.d/drbdproxy` implements. |
| 133 |
|
| 134 |
The DRBD Proxy has a low level configuration tool, called |
| 135 |
`drbd-proxy-ctl`. When called without any option it operates in |
| 136 |
interactive mode. |
| 137 |
|
| 138 |
To pass a command directly, avoiding interactive mode, use |
| 139 |
the `-c` parameter followed by the command. |
| 140 |
|
| 141 |
To display the available commands use: |
| 142 |
---------------------------- |
| 143 |
# drbd-proxy-ctl -c "help" |
| 144 |
---------------------------- |
| 145 |
|
| 146 |
Note the double quotes around the command being passed. |
| 147 |
|
| 148 |
|
| 149 |
Here is a list of commands; while the first few ones are typically only used |
| 150 |
indirectly (via `drbdadm proxy-up` resp. `drbdadm proxy-down`), the latter ones |
| 151 |
give various status information. |
| 152 |
|
| 153 |
.`add connection _<name>_ _lots of arguments_` |
| 154 |
Creates a communication path. As this is run via `drbdadm proxy-up` the long list of arguments is omitted here. |
| 155 |
|
| 156 |
.`del connection _<name>_` |
| 157 |
Removes a communication path. |
| 158 |
|
| 159 |
.`set memlimit _<name>_ _<memlimit-in-bytes>_` |
| 160 |
Sets the memory limit for a connection; this can only be done when setting it |
| 161 |
up afresh, changing it during runtime is not possible. + |
| 162 |
This command understands the usual units `k`, `M`, and `G`. |
| 163 |
|
| 164 |
.`show` |
| 165 |
Shows currently configured communication paths. |
| 166 |
|
| 167 |
.`show memusage` |
| 168 |
Shows memory usage of each connection. + |
| 169 |
-- |
| 170 |
As an example, |
| 171 |
|
| 172 |
-------- |
| 173 |
# watch -n 1 'drbd-proxy-ctl -c "show memusage"' |
| 174 |
-------- |
| 175 |
|
| 176 |
monitors memory usage. Please note that the quotes are required as listed above. |
| 177 |
-- |
| 178 |
|
| 179 |
.`show [h]subconnections` |
| 180 |
Shows currently established individual connections |
| 181 |
together with some stats. With `h` outputs bytes in human |
| 182 |
readable format. |
| 183 |
|
| 184 |
.`show [h]connections` |
| 185 |
Shows currently configured connections and their states |
| 186 |
With `h` outputs bytes in human readable format. + |
| 187 |
-- |
| 188 |
The column `Status` will show one of these states: |
| 189 |
|
| 190 |
* _Off_: No communication to the remote DRBD Proxy process. |
| 191 |
* _Half-up_: The connection to the remote DRBD Proxy could be established; the Proxy => DRBD paths are not up yet. |
| 192 |
* _DRBD-conn_: The first few packets are being pushed across the connection; but still e.g. a Split-Brain situation might serve it again. |
| 193 |
* _Up_: The DRBD connection is fully established. |
| 194 |
-- |
| 195 |
|
| 196 |
.`shutdown` |
| 197 |
Shuts down the `drbd-proxy` program. Attention: this |
| 198 |
unconditionally terminates any DRBD connections using the DRBD proxy. |
| 199 |
|
| 200 |
.`quit` |
| 201 |
Exits the client program (closes the control connection), but |
| 202 |
leaves the DRBD proxy running. |
| 203 |
|
| 204 |
|
| 205 |
.`print statistics` |
| 206 |
This prints detailed statistics for the currently active connections, in an |
| 207 |
easily parseable format. Use this for integration to your monitoring solution! + |
| 208 |
|
| 209 |
NOTE: While the commands above are only accepted from UID 0 (i.e., the `root` user), |
| 210 |
this one can be used by any user |
| 211 |
(provided that unix permissions allow access on the proxy socket at |
| 212 |
`/var/run/drbd-proxy/drbd-proxy-ctl.socket`); see the init script at |
| 213 |
`/etc/init.d/drbdproxy` about setting the rights. |
| 214 |
|
| 215 |
|
| 216 |
|
| 217 |
[[s-drbd-proxy-plugins]] |
| 218 |
=== About DRBD Proxy plugins |
| 219 |
|
| 220 |
Since DRBD Proxy version 3 the proxy allows to enable a few specific |
| 221 |
plugins for the WAN connection. + |
| 222 |
The currently available plugins are `zstd`, `lz4`, `zlib` and |
| 223 |
`lzma` (all software compression), and `aha` (hardware compression support, see http://www.aha.com/data-compression/). |
| 224 |
|
| 225 |
`zstd` (Zstandard) is a real-time compression algorithm, providing high compression ratios. It offers a very |
| 226 |
wide range of compression / speed trade-off, while being backed by a very fast decoder. Compression rates are |
| 227 |
dependent on "level" parameter which can be arranged between 1 to 22. Over level 20, Drbd Proxy will require |
| 228 |
more memory. |
| 229 |
|
| 230 |
`lz4` is a very fast compression algorithm; the data typically gets compressed |
| 231 |
down by 1:2 to 1:4, half- to two-thirds of the bandwidth can be saved. |
| 232 |
|
| 233 |
The `zlib` plugin uses the GZIP algorithm for compression; it uses a bit more CPU than `lz4`, but gives a ratio of 1:3 to 1:5. |
| 234 |
|
| 235 |
The `lzma` plugin uses the `liblzma2` library. It can |
| 236 |
use dictionaries of several hundred MiB; these allow for very |
| 237 |
efficient delta-compression of repeated data, even for small changes. |
| 238 |
`lzma` needs much more CPU and memory, but results in much better |
| 239 |
compression than `zlib` -- real-world tests with a VM sitting on top of DRBD |
| 240 |
gave ratios of 1:10 to 1:40. The `lzma` plugin has to be enabled in your |
| 241 |
license. |
| 242 |
|
| 243 |
`aha` uses hardware compression cards, like the AHA367PCIe (10Gbit/sec) or |
| 244 |
AHA372 (20GBit/sec); this is the fastest compression for contemporary hardware. + |
| 245 |
You will need a special flag in your license file to enable this plugin. |
| 246 |
|
| 247 |
|
| 248 |
Please contact LINBIT to find the best settings for your environment - it |
| 249 |
depends on the CPU (speed, number of threads), available memory, input and |
| 250 |
available output bandwidth, and expected IO spikes. Having a week of `sysstat` |
| 251 |
data already available helps in determining the configuration, too. |
| 252 |
|
| 253 |
|
| 254 |
Please note that the older `compression on` in the |
| 255 |
`proxy` section is deprecated, and will be removed in |
| 256 |
a future release. + |
| 257 |
Currently it is treated as `zlib level 9`. |
| 258 |
|
| 259 |
|
| 260 |
[[s-drbd-proxy-bwlimit]] |
| 261 |
==== Using a WAN Side Bandwidth Limit |
| 262 |
|
| 263 |
The experimental `bwlimit` option of DRBD Proxy is broken. Do not use |
| 264 |
it, as it may cause applications on DRBD to block on IO. It will |
| 265 |
be removed. |
| 266 |
|
| 267 |
Instead use the Linux kernel's traffic control framework to |
| 268 |
limit bandwidth consumed by proxy on the WAN side. |
| 269 |
|
| 270 |
In the following example you would need to replace the interface |
| 271 |
name, the source port and the ip address of the peer. |
| 272 |
|
| 273 |
---------------------------- |
| 274 |
# tc qdisc add dev eth0 root handle 1: htb default 1 |
| 275 |
# tc class add dev eth0 parent 1: classid 1:1 htb rate 1gbit |
| 276 |
# tc class add dev eth0 parent 1:1 classid 1:10 htb rate 500kbit |
| 277 |
# tc filter add dev eth0 parent 1: protocol ip prio 16 u32 \ |
| 278 |
match ip sport 7000 0xffff \ |
| 279 |
match ip dst 192.168.47.11 flowid 1:10 |
| 280 |
# tc filter add dev eth0 parent 1: protocol ip prio 16 u32 \ |
| 281 |
match ip dport 7000 0xffff \ |
| 282 |
match ip dst 192.168.47.11 flowid 1:10 |
| 283 |
---------------------------- |
| 284 |
|
| 285 |
You can remove this bandwidth limitation with |
| 286 |
|
| 287 |
---------------------------- |
| 288 |
# tc qdisc del dev eth0 root handle 1 |
| 289 |
---------------------------- |
| 290 |
|
| 291 |
[[s-drbd-proxy-troubleshoot]] |
| 292 |
=== Troubleshooting |
| 293 |
|
| 294 |
DRBD proxy logs via syslog using the `LOG_DAEMON` facility. Usually |
| 295 |
you will find DRBD Proxy messages in `/var/log/daemon.log`. |
| 296 |
|
| 297 |
Enabling debug mode in DRBD Proxy can be done with the following command. |
| 298 |
|
| 299 |
-------------------------- |
| 300 |
# drbd-proxy-ctl -c 'set loglevel debug' |
| 301 |
-------------------------- |
| 302 |
|
| 303 |
For example, if proxy fails to connect it will log something like |
| 304 |
`Rejecting connection because I can't connect on the other side`. In |
| 305 |
that case, please check if DRBD is running (not in StandAlone mode) on |
| 306 |
both nodes and if both proxies are running. Also double-check your |
| 307 |
configuration. |