Debian Codegraph @

UG9/en/latency.adoc

raw
1 [[ch-latency]]
2
3 == Optimizing DRBD latency
4
5 This chapter deals with optimizing DRBD latency. It examines some
6 hardware considerations with regard to latency minimization, and
7 details tuning recommendations for that purpose.
8
9 [[s-latency-hardware]]
10 === Hardware considerations
11
12 DRBD latency is affected by both the latency of the underlying I/O
13 subsystem (disks, controllers, and corresponding caches), and the
14 latency of the replication network.
15
16 .I/O subsystem latency
17 indexterm:[latency]For _rotating media_ the I/O subsystem latency is primarily a function of
18 disk rotation speed. Thus, using fast-spinning disks is a valid
19 approach for reducing I/O subsystem latency.
20
21 For _solid state media_ (like SSDs) the Flash storage controller is the
22 determining factor; the next most important thing is the amount of unused
23 capacity. Using DRBD's <<s-trim-discard>> will help you provide the controller
24 with the needed information which blocks it can recycle. That way, when a write
25 requests comes in, it can use a block that got cleaned ahead-of-time and
26 doesn't have to wait *now* until there's space availablefootnote:[On low-end
27 hardware you can help that a bit by reserving some space - just keep
28 10% to 20% of the total space unpartitioned.].
29
30 [[s-hardware-bbu]]
31 Likewise, the use of a indexterm:[battery-backed write cache]
32 indexterm:[BBU]
33 battery-backed write cache (BBWC) reduces write completion times, also
34 reducing write latency. Most reasonable storage subsystems come with
35 some form of battery-backed cache, and allow the administrator to
36 configure which portion of this cache is used for read and write
37 operations. The recommended approach is to disable the disk read cache
38 completely and use all available cache memory for the disk write
39 cache.
40
41 .Network latency
42 indexterm:[latency]Network latency is, in essence, the packet
43 indexterm:[round-trip-time]indexterm:[RTT]
44 round-trip time (RTT) between hosts. It is influenced by a number of
45 factors, most of which are irrelevant on the dedicated, back-to-back
46 network connections recommended for use as DRBD replication
47 links. Thus, it is sufficient to accept that a certain amount of
48 latency always exists in network links, which typically is on
49 the order of 100 to 200 microseconds (μs) packet RTT for Gigabit Ethernet.
50
51 Network latency may typically be pushed below this limit only by using
52 lower-latency network protocols, such as running DRBD over Dolphin
53 Express using Dolphin SuperSockets, or a 10GBe direct connection; these are
54 typically in the 50µs range. Even better is InfiniBand, which provides
55 even lower latency.
56
57
58 [[s-latency-overhead-expectations]]
59 === Latency overhead expectations
60
61 As for throughput, when estimating the latency overhead associated
62 with DRBD, there are some important natural limitations to consider:
63
64 * DRBD latency is bound by that of the raw I/O subsystem.
65 * DRBD latency is bound by the available network latency.
66
67 The _sum_ of the two establishes the theoretical latency _minimum_
68 incurred to DRBDfootnote:[for protocol C, because the other node(s) have to
69 write to stable storage, too]. DRBD then adds to that latency a slight
70 additional latency overhead, which can be expected to be less than 1 percent.
71
72 * Consider the example of a local disk subsystem with a write latency
73 of 3ms and a network link with one of 0.2ms. Then the expected DRBD
74 latency would be 3.2 ms or a roughly 7-percent latency increase over
75 just writing to a local disk.
76
77 NOTE: Latency may be influenced by a number of other factors,
78 including CPU cache misses, context switches, and others.
79
80
81 [[s-latency-iops]]
82 === Latency vs. IOPs
83
84 indexterm:[latency]indexterm:[IOPs]_IOPs_ is the abbreviation of "__I/O
85 operations per second__".
86
87 Marketing typically doesn't like numbers that get smaller; press releases
88 aren't written with "__Latency reduced by 10µs, from 50µs to 40µs now!__" in
89 mind, they like "__Performance increased by 25%, from 20000 to now 25000
90 IOPs__" much more. Therefore IOPs were invented - to get a number that says
91 "higher is better".
92
93 So, in other words, IOPs are the reciprocal of latency. What you'll have to
94 keep in mind is that the method given in <<s-measure-latency>> gives you
95 the latency resp. the number of IOPs for a purely sequential, single-threaded
96 IO load, while most other documentation will give numbers for some highly
97 parallel loadfootnote:[Like in "__16 threads, IO-depth of 32__" - this means that 512
98 I/O-requests are being done in parallel!], because this gives much "prettier"
99 numbers. With that kind of trick DRBD does offer you 100000 IOPs, too!
100
101 So, please don't shy away from measuring serialized, single-threaded latency.
102 If you want lots of IOPS, run the `fio` utility with `threads=8` and an
103 `iodepth=16`, or some similar settings... But please keep in mind that these
104 number will not have any meaning to your setup, unless you're driving
105 a database with many tens or hundreds of client connections active at the same
106 time.
107
108
109 [[s-latency-tuning]]
110 === Tuning recommendations
111
112 [[s-latency-tuning-cpu-mask]]
113 ==== Setting DRBD's CPU mask
114
115 DRBD allows for setting an explicit CPU mask for its kernel
116 threads. This is particularly beneficial for applications which would
117 otherwise compete with DRBD for CPU cycles.
118
119 The CPU mask is a number in whose binary representation the least
120 significant bit represents the first CPU, the second-least significant
121 bit the second, and so forth. A set bit in the bitmask implies that
122 the corresponding CPU may be used by DRBD, whereas a cleared bit means
123 it must not. Thus, for example, a CPU mask of 1 (`00000001`) means
124 DRBD may use the first CPU only. A mask of 12 (`00001100`) implies
125 DRBD may use the third and fourth CPU.
126
127
128 An example CPU mask configuration for a resource may look like this:
129
130 [source,drbd]
131 ----------------------------
132 resource <resource> {
133 options {
134 cpu-mask 2;
135 ...
136 }
137 ...
138 }
139 ----------------------------
140
141 IMPORTANT: Of course, in order to minimize CPU competition between
142 DRBD and the application using it, you need to configure your
143 application to use only those CPUs which DRBD does not use.
144
145 Some applications may provide for this via an entry in a configuration
146 file, just like DRBD itself. Others include an invocation of the
147 `taskset` command in an application init script.
148
149 [NOTE]
150 ====================
151
152 It makes sense to keep the DRBD threads running on the same L2/L3 caches.
153
154 But: the numbering of CPUs doesn't have to correlate with the physical partitioning.
155 You can try the `lstopo` (or `hwloc-ls`) program for X11 or `hwloc-info -v -p`
156 for console output to get an overview of the topology.
157 ====================
158
159
160 [[s-latency-tuning-mtu-size]]
161 ==== Modifying the network MTU
162
163 It may be beneficial to change the replication network's
164 maximum transmission unit (MTU) size to a value higher than the
165 default of 1500 bytes. Colloquially, this is referred to as
166 indexterm:[Jumbo frames] "enabling Jumbo frames".
167
168 The MTU may be changed using the following commands:
169 ----------------------------
170 # ifconfig <interface> mtu <size>
171 ----------------------------
172 or
173 ----------------------------
174 # ip link set <interface> mtu <size>
175 ----------------------------
176
177 _<interface>_ refers to the network interface used for DRBD
178 replication. A typical value for _<size>_ would be 9000 (bytes).
179
180 [[s-latency-tuning-deadline-scheduler]]
181 ==== Enabling the deadline I/O scheduler
182
183 indexterm:[io scheduler]
184 When used in conjunction with high-performance, write back enabled
185 hardware RAID controllers, DRBD latency may benefit greatly from using
186 the simple deadline I/O scheduler, rather than the CFQ scheduler. The
187 latter is typically enabled by default.
188
189 Modifications to the I/O scheduler configuration may be performed via
190 the `sysfs` virtual file system, mounted at `/sys`. The scheduler
191 configuration is in `/sys/block/__device__`, where _<device>_ is the
192 backing device DRBD uses.
193
194 Enabling the deadline scheduler works via the following command:
195 ----------------------------
196 # echo deadline > /sys/block/<device>/queue/scheduler
197 ----------------------------
198
199 You may then also set the following values, which may provide
200 additional latency benefits:
201
202 * Disable front merges:
203 +
204 ----------------------------
205 # echo 0 > /sys/block/<device>/queue/iosched/front_merges
206 ----------------------------
207
208 * Reduce read I/O deadline to 150 milliseconds (the default is 500ms):
209 +
210 ----------------------------
211 # echo 150 > /sys/block/<device>/queue/iosched/read_expire
212 ----------------------------
213
214 * Reduce write I/O deadline to 1500 milliseconds (the default is
215 3000ms):
216 +
217 ----------------------------
218 # echo 1500 > /sys/block/<device>/queue/iosched/write_expire
219 ----------------------------
220
221 If these values effect a significant latency improvement, you may want
222 to make them permanent so they are automatically set at system
223 startup. indexterm:[Debian GNU/Linux]Debian and indexterm:[Ubuntu
224 Linux]Ubuntu systems provide this functionality via the
225 `sysfsutils` package and the `/etc/sysfs.conf` configuration file.
226
227 You may also make a global I/O scheduler selection by passing the
228 `elevator` option via your kernel command line. To do so, edit your
229 boot loader configuration (normally found in `/etc/default/grub` if
230 you are using the GRUB bootloader) and add `elevator=deadline` to your
231 list of kernel boot options.