| 1 |
[[ch-throughput]] |
| 2 |
== Optimizing DRBD throughput |
| 3 |
|
| 4 |
This chapter deals with optimizing DRBD throughput. It examines some |
| 5 |
hardware considerations with regard to throughput optimization, and |
| 6 |
details tuning recommendations for that purpose. |
| 7 |
|
| 8 |
[[s-throughput-hardware]] |
| 9 |
=== Hardware considerations |
| 10 |
|
| 11 |
DRBD throughput is affected by both the bandwidth of the underlying |
| 12 |
I/O subsystem (disks, controllers, and corresponding caches), and the |
| 13 |
bandwidth of the replication network. |
| 14 |
|
| 15 |
.I/O subsystem throughput |
| 16 |
indexterm:[throughput]I/O subsystem throughput is determined, largely, |
| 17 |
by the number and type of storage units (disks, SSDs, other Flash storage [like |
| 18 |
FusionIO], ...) that can be written to in parallel. A single, |
| 19 |
reasonably recent, SCSI or SAS disk will typically allow streaming |
| 20 |
writes of roughly 40MiB/s to the single disk; an SSD will do 300MiB/s; |
| 21 |
one of the recent Flash storages (NVMe) will be at 1GiB/s. When deployed in a |
| 22 |
striping configuration, the I/O subsystem will parallelize writes |
| 23 |
across disks, effectively multiplying a single disk's throughput by |
| 24 |
the number of stripes in the configuration. Thus the same, 40MB/s |
| 25 |
disks will allow effective throughput of 120MB/s in a RAID-0 or |
| 26 |
RAID-1+0 configuration with three stripes, or 200MB/s with five |
| 27 |
stripes; with SSDs and/or NVMe you can easily get to 1GiB/sec. |
| 28 |
|
| 29 |
A RAID-controller with RAM and a <<s-hardware-bbu,BBU>> can speed up short |
| 30 |
spikes (by buffering them), and so too-short benchmark tests might show speeds |
| 31 |
like 1GiB/s too; for sustained writes its buffers will just run full, |
| 32 |
and then not be of much help, though. |
| 33 |
|
| 34 |
|
| 35 |
NOTE: Disk _mirroring_ (RAID-1) in hardware typically has little, if |
| 36 |
any, effect on throughput. Disk _striping with parity_ (RAID-5) does |
| 37 |
have an effect on throughput, usually an adverse one when compared to |
| 38 |
striping; RAID-5 and RAID-6 in software even more so. |
| 39 |
|
| 40 |
.Network throughput |
| 41 |
indexterm:[throughput]Network throughput is usually determined by the |
| 42 |
amount of traffic present on the network, and on the throughput of any |
| 43 |
routing/switching infrastructure present. These concerns are, however, |
| 44 |
largely irrelevant in DRBD replication links which are normally |
| 45 |
dedicated, back-to-back network connections. Thus, network throughput |
| 46 |
may be improved either by switching to a higher-throughput hardware |
| 47 |
(such as 10 Gigabit Ethernet, or 56GiB Infiniband), or by using link aggregation over |
| 48 |
several network links, as one may do using the Linux |
| 49 |
indexterm:[bonding driver]`bonding` network driver. |
| 50 |
|
| 51 |
[[s-throughput-overhead-expectations]] |
| 52 |
=== Throughput overhead expectations |
| 53 |
|
| 54 |
When estimating the throughput overhead associated with DRBD, it is |
| 55 |
important to consider the following natural limitations: |
| 56 |
|
| 57 |
* DRBD throughput is limited by that of the raw I/O subsystem. |
| 58 |
* DRBD throughput is limited by the available network bandwidth. |
| 59 |
|
| 60 |
The _lower_ of these two establishes the theoretical throughput |
| 61 |
_maximum_ available to DRBD. DRBD then reduces that throughput number |
| 62 |
by its additional overhead, which can be expected to be |
| 63 |
less than 3 percent. |
| 64 |
|
| 65 |
* Consider the example of two cluster nodes containing I/O subsystems |
| 66 |
capable of 600 MB/s throughput, with a Gigabit Ethernet link |
| 67 |
available between them. Gigabit Ethernet can be expected to produce |
| 68 |
110 MB/s throughput for TCP connections, thus the network connection |
| 69 |
would be the bottleneck in this configuration and one would |
| 70 |
expect about 110 MB/s maximum DRBD throughput. |
| 71 |
|
| 72 |
* By contrast, if the I/O subsystem is capable of only 80 MB/s for |
| 73 |
sustained writes, then it constitutes the bottleneck, and you should |
| 74 |
expect only about 77 MB/s maximum DRBD throughput. |
| 75 |
|
| 76 |
|
| 77 |
[[s-throughput-tuning]] |
| 78 |
=== Tuning recommendations |
| 79 |
|
| 80 |
DRBD offers a number of configuration options which may have an effect |
| 81 |
on your system's throughput. This section list some recommendations |
| 82 |
for tuning for throughput. However, since throughput is largely |
| 83 |
hardware dependent, the effects of tweaking the options described here |
| 84 |
may vary greatly from system to system. It is important to understand |
| 85 |
that these recommendations should not be interpreted as "silver |
| 86 |
bullets" which would magically remove any and all throughput |
| 87 |
bottlenecks. |
| 88 |
|
| 89 |
[[s-tune-max-buffer-max-epoch-size]] |
| 90 |
==== Setting `max-buffers` and `max-epoch-size` |
| 91 |
|
| 92 |
These options affect write performance on the secondary |
| 93 |
nodes. `max-buffers` is the maximum number of buffers DRBD allocates for |
| 94 |
writing data to disk while `max-epoch-size` is the maximum number of |
| 95 |
write requests permitted between two write barriers. `max-buffers` must be |
| 96 |
equal or bigger to `max-epoch-size` to increase performance. |
| 97 |
The default for both is 2048; setting it to around |
| 98 |
8000 should be fine for most reasonably high-performance hardware RAID |
| 99 |
controllers. |
| 100 |
|
| 101 |
[source,drbd] |
| 102 |
---------------------------- |
| 103 |
resource <resource> { |
| 104 |
net { |
| 105 |
max-buffers 8000; |
| 106 |
max-epoch-size 8000; |
| 107 |
... |
| 108 |
} |
| 109 |
... |
| 110 |
} |
| 111 |
---------------------------- |
| 112 |
|
| 113 |
[[s-tune-sndbuf-size]] |
| 114 |
==== Tuning the TCP send buffer size |
| 115 |
|
| 116 |
The TCP send buffer is a memory buffer for outgoing TCP traffic. By |
| 117 |
default, it is set to a size of 128 KiB. For use in high-throughput |
| 118 |
networks (such as dedicated Gigabit Ethernet or load-balanced bonded |
| 119 |
connections), it may make sense to increase this to a size of 2MiB, |
| 120 |
or perhaps even more. Send buffer sizes of more than 16MiB are |
| 121 |
generally not recommended (and are also unlikely to produce any |
| 122 |
throughput improvement). |
| 123 |
|
| 124 |
[source,drbd] |
| 125 |
---------------------------- |
| 126 |
resource <resource> { |
| 127 |
net { |
| 128 |
sndbuf-size 2M; |
| 129 |
... |
| 130 |
} |
| 131 |
... |
| 132 |
} |
| 133 |
---------------------------- |
| 134 |
|
| 135 |
DRBD also supports TCP send buffer auto-tuning. After enabling this |
| 136 |
feature, DRBD will dynamically select an appropriate TCP send buffer |
| 137 |
size. TCP send buffer auto tuning is enabled by simply setting the |
| 138 |
buffer size to zero: |
| 139 |
|
| 140 |
[source,drbd] |
| 141 |
---------------------------- |
| 142 |
resource <resource> { |
| 143 |
net { |
| 144 |
sndbuf-size 0; |
| 145 |
... |
| 146 |
} |
| 147 |
... |
| 148 |
} |
| 149 |
---------------------------- |
| 150 |
|
| 151 |
Please note that your ``sysctl``'s settings `net.ipv4.tcp_rmem` and |
| 152 |
`net.ipv4.tcp_wmem` will still influence the behaviour; you should check |
| 153 |
these settings, and perhaps set them similar to `131072 1048576 16777216` |
| 154 |
(minimum 128kiB, default 1MiB, max 16MiB). |
| 155 |
|
| 156 |
WARNING: `net.ipv4.tcp_mem` is a different beast, with a different unit - |
| 157 |
do not touch, wrong values can easily push your machine into out-of-memory |
| 158 |
situations! |
| 159 |
|
| 160 |
[[s-tune-al-extents]] |
| 161 |
==== Tuning the Activity Log size |
| 162 |
|
| 163 |
If the application using DRBD is write intensive in the sense that it |
| 164 |
frequently issues small writes scattered across the device, it is |
| 165 |
usually advisable to use a fairly large activity log. Otherwise, |
| 166 |
frequent metadata updates may be detrimental to write performance. |
| 167 |
|
| 168 |
[source,drbd] |
| 169 |
---------------------------- |
| 170 |
resource <resource> { |
| 171 |
disk { |
| 172 |
al-extents 6007; |
| 173 |
... |
| 174 |
} |
| 175 |
... |
| 176 |
} |
| 177 |
---------------------------- |
| 178 |
|
| 179 |
//// |
| 180 |
For tuning on striped RAID setups, please see the <<al-stripes,al-stripes>> and |
| 181 |
<<al-stripe-size,al-stripe-size>> settings, too. |
| 182 |
//// |
| 183 |
|
| 184 |
[[s-tune-disable-barriers]] |
| 185 |
==== Disabling barriers and disk flushes |
| 186 |
|
| 187 |
WARNING: The recommendations outlined in this section should be applied |
| 188 |
_only_ to systems with non-volatile (battery backed) controller caches. |
| 189 |
|
| 190 |
Systems equipped with battery backed write cache come with built-in |
| 191 |
means of protecting data in the face of power failure. In that case, |
| 192 |
it is permissible to disable some of DRBD's own safeguards created for |
| 193 |
the same purpose. This may be beneficial in terms of throughput: |
| 194 |
|
| 195 |
[source,drbd] |
| 196 |
---------------------------- |
| 197 |
resource <resource> { |
| 198 |
disk { |
| 199 |
disk-barrier no; |
| 200 |
disk-flushes no; |
| 201 |
... |
| 202 |
} |
| 203 |
... |
| 204 |
} |
| 205 |
---------------------------- |
| 206 |
|
| 207 |
|
| 208 |
[[s-tune-read-balancing]] |
| 209 |
=== Achieving better Read Performance via increased Redundancy |
| 210 |
|
| 211 |
As detailed in the man page of `drbd.conf` under `read-balancing`, |
| 212 |
you can increase your read performance by adding more copies of your data. |
| 213 |
|
| 214 |
As a ballpark figure: with a single node processing read requests, `fio` on |
| 215 |
a __FusionIO__ card gave us 100k IOPs; after enabling `read-balancing`, the |
| 216 |
performance jumped to 180k IOPs, i.e. +80%! |
| 217 |
|
| 218 |
So, in case you're running a read-mostly workload (big databases with lots of |
| 219 |
random reads), it might be worth a try to turn `read-balancing` on - and, |
| 220 |
perhaps, add another copy for still more read IO throughput. |