| 1 |
[[ch-latency]] |
| 2 |
|
| 3 |
== Optimizing DRBD latency |
| 4 |
|
| 5 |
This chapter deals with optimizing DRBD latency. It examines some |
| 6 |
hardware considerations with regard to latency minimization, and |
| 7 |
details tuning recommendations for that purpose. |
| 8 |
|
| 9 |
[[s-latency-hardware]] |
| 10 |
=== Hardware considerations |
| 11 |
|
| 12 |
DRBD latency is affected by both the latency of the underlying I/O |
| 13 |
subsystem (disks, controllers, and corresponding caches), and the |
| 14 |
latency of the replication network. |
| 15 |
|
| 16 |
.I/O subsystem latency |
| 17 |
indexterm:[latency]For _rotating media_ the I/O subsystem latency is primarily a function of |
| 18 |
disk rotation speed. Thus, using fast-spinning disks is a valid |
| 19 |
approach for reducing I/O subsystem latency. |
| 20 |
|
| 21 |
For _solid state media_ (like SSDs) the Flash storage controller is the |
| 22 |
determining factor; the next most important thing is the amount of unused |
| 23 |
capacity. Using DRBD's <<s-trim-discard>> will help you provide the controller |
| 24 |
with the needed information which blocks it can recycle. That way, when a write |
| 25 |
requests comes in, it can use a block that got cleaned ahead-of-time and |
| 26 |
doesn't have to wait *now* until there's space availablefootnote:[On low-end |
| 27 |
hardware you can help that a bit by reserving some space - just keep |
| 28 |
10% to 20% of the total space unpartitioned.]. |
| 29 |
|
| 30 |
[[s-hardware-bbu]] |
| 31 |
Likewise, the use of a indexterm:[battery-backed write cache] |
| 32 |
indexterm:[BBU] |
| 33 |
battery-backed write cache (BBWC) reduces write completion times, also |
| 34 |
reducing write latency. Most reasonable storage subsystems come with |
| 35 |
some form of battery-backed cache, and allow the administrator to |
| 36 |
configure which portion of this cache is used for read and write |
| 37 |
operations. The recommended approach is to disable the disk read cache |
| 38 |
completely and use all available cache memory for the disk write |
| 39 |
cache. |
| 40 |
|
| 41 |
.Network latency |
| 42 |
indexterm:[latency]Network latency is, in essence, the packet |
| 43 |
indexterm:[round-trip-time]indexterm:[RTT] |
| 44 |
round-trip time (RTT) between hosts. It is influenced by a number of |
| 45 |
factors, most of which are irrelevant on the dedicated, back-to-back |
| 46 |
network connections recommended for use as DRBD replication |
| 47 |
links. Thus, it is sufficient to accept that a certain amount of |
| 48 |
latency always exists in network links, which typically is on |
| 49 |
the order of 100 to 200 microseconds (μs) packet RTT for Gigabit Ethernet. |
| 50 |
|
| 51 |
Network latency may typically be pushed below this limit only by using |
| 52 |
lower-latency network protocols, such as running DRBD over Dolphin |
| 53 |
Express using Dolphin SuperSockets, or a 10GBe direct connection; these are |
| 54 |
typically in the 50µs range. Even better is InfiniBand, which provides |
| 55 |
even lower latency. |
| 56 |
|
| 57 |
|
| 58 |
[[s-latency-overhead-expectations]] |
| 59 |
=== Latency overhead expectations |
| 60 |
|
| 61 |
As for throughput, when estimating the latency overhead associated |
| 62 |
with DRBD, there are some important natural limitations to consider: |
| 63 |
|
| 64 |
* DRBD latency is bound by that of the raw I/O subsystem. |
| 65 |
* DRBD latency is bound by the available network latency. |
| 66 |
|
| 67 |
The _sum_ of the two establishes the theoretical latency _minimum_ |
| 68 |
incurred to DRBDfootnote:[for protocol C, because the other node(s) have to |
| 69 |
write to stable storage, too]. DRBD then adds to that latency a slight |
| 70 |
additional latency overhead, which can be expected to be less than 1 percent. |
| 71 |
|
| 72 |
* Consider the example of a local disk subsystem with a write latency |
| 73 |
of 3ms and a network link with one of 0.2ms. Then the expected DRBD |
| 74 |
latency would be 3.2 ms or a roughly 7-percent latency increase over |
| 75 |
just writing to a local disk. |
| 76 |
|
| 77 |
NOTE: Latency may be influenced by a number of other factors, |
| 78 |
including CPU cache misses, context switches, and others. |
| 79 |
|
| 80 |
|
| 81 |
[[s-latency-iops]] |
| 82 |
=== Latency vs. IOPs |
| 83 |
|
| 84 |
indexterm:[latency]indexterm:[IOPs]_IOPs_ is the abbreviation of "__I/O |
| 85 |
operations per second__". |
| 86 |
|
| 87 |
Marketing typically doesn't like numbers that get smaller; press releases |
| 88 |
aren't written with "__Latency reduced by 10µs, from 50µs to 40µs now!__" in |
| 89 |
mind, they like "__Performance increased by 25%, from 20000 to now 25000 |
| 90 |
IOPs__" much more. Therefore IOPs were invented - to get a number that says |
| 91 |
"higher is better". |
| 92 |
|
| 93 |
So, in other words, IOPs are the reciprocal of latency. What you'll have to |
| 94 |
keep in mind is that the method given in <<s-measure-latency>> gives you |
| 95 |
the latency resp. the number of IOPs for a purely sequential, single-threaded |
| 96 |
IO load, while most other documentation will give numbers for some highly |
| 97 |
parallel loadfootnote:[Like in "__16 threads, IO-depth of 32__" - this means that 512 |
| 98 |
I/O-requests are being done in parallel!], because this gives much "prettier" |
| 99 |
numbers. With that kind of trick DRBD does offer you 100000 IOPs, too! |
| 100 |
|
| 101 |
So, please don't shy away from measuring serialized, single-threaded latency. |
| 102 |
If you want lots of IOPS, run the `fio` utility with `threads=8` and an |
| 103 |
`iodepth=16`, or some similar settings... But please keep in mind that these |
| 104 |
number will not have any meaning to your setup, unless you're driving |
| 105 |
a database with many tens or hundreds of client connections active at the same |
| 106 |
time. |
| 107 |
|
| 108 |
|
| 109 |
[[s-latency-tuning]] |
| 110 |
=== Tuning recommendations |
| 111 |
|
| 112 |
[[s-latency-tuning-cpu-mask]] |
| 113 |
==== Setting DRBD's CPU mask |
| 114 |
|
| 115 |
DRBD allows for setting an explicit CPU mask for its kernel |
| 116 |
threads. This is particularly beneficial for applications which would |
| 117 |
otherwise compete with DRBD for CPU cycles. |
| 118 |
|
| 119 |
The CPU mask is a number in whose binary representation the least |
| 120 |
significant bit represents the first CPU, the second-least significant |
| 121 |
bit the second, and so forth. A set bit in the bitmask implies that |
| 122 |
the corresponding CPU may be used by DRBD, whereas a cleared bit means |
| 123 |
it must not. Thus, for example, a CPU mask of 1 (`00000001`) means |
| 124 |
DRBD may use the first CPU only. A mask of 12 (`00001100`) implies |
| 125 |
DRBD may use the third and fourth CPU. |
| 126 |
|
| 127 |
|
| 128 |
An example CPU mask configuration for a resource may look like this: |
| 129 |
|
| 130 |
[source,drbd] |
| 131 |
---------------------------- |
| 132 |
resource <resource> { |
| 133 |
options { |
| 134 |
cpu-mask 2; |
| 135 |
... |
| 136 |
} |
| 137 |
... |
| 138 |
} |
| 139 |
---------------------------- |
| 140 |
|
| 141 |
IMPORTANT: Of course, in order to minimize CPU competition between |
| 142 |
DRBD and the application using it, you need to configure your |
| 143 |
application to use only those CPUs which DRBD does not use. |
| 144 |
|
| 145 |
Some applications may provide for this via an entry in a configuration |
| 146 |
file, just like DRBD itself. Others include an invocation of the |
| 147 |
`taskset` command in an application init script. |
| 148 |
|
| 149 |
[NOTE] |
| 150 |
==================== |
| 151 |
|
| 152 |
It makes sense to keep the DRBD threads running on the same L2/L3 caches. |
| 153 |
|
| 154 |
But: the numbering of CPUs doesn't have to correlate with the physical partitioning. |
| 155 |
You can try the `lstopo` (or `hwloc-ls`) program for X11 or `hwloc-info -v -p` |
| 156 |
for console output to get an overview of the topology. |
| 157 |
==================== |
| 158 |
|
| 159 |
|
| 160 |
[[s-latency-tuning-mtu-size]] |
| 161 |
==== Modifying the network MTU |
| 162 |
|
| 163 |
It may be beneficial to change the replication network's |
| 164 |
maximum transmission unit (MTU) size to a value higher than the |
| 165 |
default of 1500 bytes. Colloquially, this is referred to as |
| 166 |
indexterm:[Jumbo frames] "enabling Jumbo frames". |
| 167 |
|
| 168 |
The MTU may be changed using the following commands: |
| 169 |
---------------------------- |
| 170 |
# ifconfig <interface> mtu <size> |
| 171 |
---------------------------- |
| 172 |
or |
| 173 |
---------------------------- |
| 174 |
# ip link set <interface> mtu <size> |
| 175 |
---------------------------- |
| 176 |
|
| 177 |
_<interface>_ refers to the network interface used for DRBD |
| 178 |
replication. A typical value for _<size>_ would be 9000 (bytes). |
| 179 |
|
| 180 |
[[s-latency-tuning-deadline-scheduler]] |
| 181 |
==== Enabling the deadline I/O scheduler |
| 182 |
|
| 183 |
indexterm:[io scheduler] |
| 184 |
When used in conjunction with high-performance, write back enabled |
| 185 |
hardware RAID controllers, DRBD latency may benefit greatly from using |
| 186 |
the simple deadline I/O scheduler, rather than the CFQ scheduler. The |
| 187 |
latter is typically enabled by default. |
| 188 |
|
| 189 |
Modifications to the I/O scheduler configuration may be performed via |
| 190 |
the `sysfs` virtual file system, mounted at `/sys`. The scheduler |
| 191 |
configuration is in `/sys/block/__device__`, where _<device>_ is the |
| 192 |
backing device DRBD uses. |
| 193 |
|
| 194 |
Enabling the deadline scheduler works via the following command: |
| 195 |
---------------------------- |
| 196 |
# echo deadline > /sys/block/<device>/queue/scheduler |
| 197 |
---------------------------- |
| 198 |
|
| 199 |
You may then also set the following values, which may provide |
| 200 |
additional latency benefits: |
| 201 |
|
| 202 |
* Disable front merges: |
| 203 |
+ |
| 204 |
---------------------------- |
| 205 |
# echo 0 > /sys/block/<device>/queue/iosched/front_merges |
| 206 |
---------------------------- |
| 207 |
|
| 208 |
* Reduce read I/O deadline to 150 milliseconds (the default is 500ms): |
| 209 |
+ |
| 210 |
---------------------------- |
| 211 |
# echo 150 > /sys/block/<device>/queue/iosched/read_expire |
| 212 |
---------------------------- |
| 213 |
|
| 214 |
* Reduce write I/O deadline to 1500 milliseconds (the default is |
| 215 |
3000ms): |
| 216 |
+ |
| 217 |
---------------------------- |
| 218 |
# echo 1500 > /sys/block/<device>/queue/iosched/write_expire |
| 219 |
---------------------------- |
| 220 |
|
| 221 |
If these values effect a significant latency improvement, you may want |
| 222 |
to make them permanent so they are automatically set at system |
| 223 |
startup. indexterm:[Debian GNU/Linux]Debian and indexterm:[Ubuntu |
| 224 |
Linux]Ubuntu systems provide this functionality via the |
| 225 |
`sysfsutils` package and the `/etc/sysfs.conf` configuration file. |
| 226 |
|
| 227 |
You may also make a global I/O scheduler selection by passing the |
| 228 |
`elevator` option via your kernel command line. To do so, edit your |
| 229 |
boot loader configuration (normally found in `/etc/default/grub` if |
| 230 |
you are using the GRUB bootloader) and add `elevator=deadline` to your |
| 231 |
list of kernel boot options. |