Debian Codegraph @

UG9/en/throughput.adoc

raw
1 [[ch-throughput]]
2 == Optimizing DRBD throughput
3
4 This chapter deals with optimizing DRBD throughput. It examines some
5 hardware considerations with regard to throughput optimization, and
6 details tuning recommendations for that purpose.
7
8 [[s-throughput-hardware]]
9 === Hardware considerations
10
11 DRBD throughput is affected by both the bandwidth of the underlying
12 I/O subsystem (disks, controllers, and corresponding caches), and the
13 bandwidth of the replication network.
14
15 .I/O subsystem throughput
16 indexterm:[throughput]I/O subsystem throughput is determined, largely,
17 by the number and type of storage units (disks, SSDs, other Flash storage [like
18 FusionIO], ...) that can be written to in parallel. A single,
19 reasonably recent, SCSI or SAS disk will typically allow streaming
20 writes of roughly 40MiB/s to the single disk; an SSD will do 300MiB/s;
21 one of the recent Flash storages (NVMe) will be at 1GiB/s. When deployed in a
22 striping configuration, the I/O subsystem will parallelize writes
23 across disks, effectively multiplying a single disk's throughput by
24 the number of stripes in the configuration. Thus the same, 40MB/s
25 disks will allow effective throughput of 120MB/s in a RAID-0 or
26 RAID-1+0 configuration with three stripes, or 200MB/s with five
27 stripes; with SSDs and/or NVMe you can easily get to 1GiB/sec.
28
29 A RAID-controller with RAM and a <<s-hardware-bbu,BBU>> can speed up short
30 spikes (by buffering them), and so too-short benchmark tests might show speeds
31 like 1GiB/s too; for sustained writes its buffers will just run full,
32 and then not be of much help, though.
33
34
35 NOTE: Disk _mirroring_ (RAID-1) in hardware typically has little, if
36 any, effect on throughput. Disk _striping with parity_ (RAID-5) does
37 have an effect on throughput, usually an adverse one when compared to
38 striping; RAID-5 and RAID-6 in software even more so.
39
40 .Network throughput
41 indexterm:[throughput]Network throughput is usually determined by the
42 amount of traffic present on the network, and on the throughput of any
43 routing/switching infrastructure present. These concerns are, however,
44 largely irrelevant in DRBD replication links which are normally
45 dedicated, back-to-back network connections. Thus, network throughput
46 may be improved either by switching to a higher-throughput hardware
47 (such as 10 Gigabit Ethernet, or 56GiB Infiniband), or by using link aggregation over
48 several network links, as one may do using the Linux
49 indexterm:[bonding driver]`bonding` network driver.
50
51 [[s-throughput-overhead-expectations]]
52 === Throughput overhead expectations
53
54 When estimating the throughput overhead associated with DRBD, it is
55 important to consider the following natural limitations:
56
57 * DRBD throughput is limited by that of the raw I/O subsystem.
58 * DRBD throughput is limited by the available network bandwidth.
59
60 The _lower_ of these two establishes the theoretical throughput
61 _maximum_ available to DRBD. DRBD then reduces that throughput number
62 by its additional overhead, which can be expected to be
63 less than 3 percent.
64
65 * Consider the example of two cluster nodes containing I/O subsystems
66 capable of 600 MB/s throughput, with a Gigabit Ethernet link
67 available between them. Gigabit Ethernet can be expected to produce
68 110 MB/s throughput for TCP connections, thus the network connection
69 would be the bottleneck in this configuration and one would
70 expect about 110 MB/s maximum DRBD throughput.
71
72 * By contrast, if the I/O subsystem is capable of only 80 MB/s for
73 sustained writes, then it constitutes the bottleneck, and you should
74 expect only about 77 MB/s maximum DRBD throughput.
75
76
77 [[s-throughput-tuning]]
78 === Tuning recommendations
79
80 DRBD offers a number of configuration options which may have an effect
81 on your system's throughput. This section list some recommendations
82 for tuning for throughput. However, since throughput is largely
83 hardware dependent, the effects of tweaking the options described here
84 may vary greatly from system to system. It is important to understand
85 that these recommendations should not be interpreted as "silver
86 bullets" which would magically remove any and all throughput
87 bottlenecks.
88
89 [[s-tune-max-buffer-max-epoch-size]]
90 ==== Setting `max-buffers` and `max-epoch-size`
91
92 These options affect write performance on the secondary
93 nodes. `max-buffers` is the maximum number of buffers DRBD allocates for
94 writing data to disk while `max-epoch-size` is the maximum number of
95 write requests permitted between two write barriers. `max-buffers` must be
96 equal or bigger to `max-epoch-size` to increase performance.
97 The default for both is 2048; setting it to around
98 8000 should be fine for most reasonably high-performance hardware RAID
99 controllers.
100
101 [source,drbd]
102 ----------------------------
103 resource <resource> {
104 net {
105 max-buffers 8000;
106 max-epoch-size 8000;
107 ...
108 }
109 ...
110 }
111 ----------------------------
112
113 [[s-tune-sndbuf-size]]
114 ==== Tuning the TCP send buffer size
115
116 The TCP send buffer is a memory buffer for outgoing TCP traffic. By
117 default, it is set to a size of 128 KiB. For use in high-throughput
118 networks (such as dedicated Gigabit Ethernet or load-balanced bonded
119 connections), it may make sense to increase this to a size of 2MiB,
120 or perhaps even more. Send buffer sizes of more than 16MiB are
121 generally not recommended (and are also unlikely to produce any
122 throughput improvement).
123
124 [source,drbd]
125 ----------------------------
126 resource <resource> {
127 net {
128 sndbuf-size 2M;
129 ...
130 }
131 ...
132 }
133 ----------------------------
134
135 DRBD also supports TCP send buffer auto-tuning. After enabling this
136 feature, DRBD will dynamically select an appropriate TCP send buffer
137 size. TCP send buffer auto tuning is enabled by simply setting the
138 buffer size to zero:
139
140 [source,drbd]
141 ----------------------------
142 resource <resource> {
143 net {
144 sndbuf-size 0;
145 ...
146 }
147 ...
148 }
149 ----------------------------
150
151 Please note that your ``sysctl``'s settings `net.ipv4.tcp_rmem` and
152 `net.ipv4.tcp_wmem` will still influence the behaviour; you should check
153 these settings, and perhaps set them similar to `131072 1048576 16777216`
154 (minimum 128kiB, default 1MiB, max 16MiB).
155
156 WARNING: `net.ipv4.tcp_mem` is a different beast, with a different unit -
157 do not touch, wrong values can easily push your machine into out-of-memory
158 situations!
159
160 [[s-tune-al-extents]]
161 ==== Tuning the Activity Log size
162
163 If the application using DRBD is write intensive in the sense that it
164 frequently issues small writes scattered across the device, it is
165 usually advisable to use a fairly large activity log. Otherwise,
166 frequent metadata updates may be detrimental to write performance.
167
168 [source,drbd]
169 ----------------------------
170 resource <resource> {
171 disk {
172 al-extents 6007;
173 ...
174 }
175 ...
176 }
177 ----------------------------
178
179 ////
180 For tuning on striped RAID setups, please see the <<al-stripes,al-stripes>> and
181 <<al-stripe-size,al-stripe-size>> settings, too.
182 ////
183
184 [[s-tune-disable-barriers]]
185 ==== Disabling barriers and disk flushes
186
187 WARNING: The recommendations outlined in this section should be applied
188 _only_ to systems with non-volatile (battery backed) controller caches.
189
190 Systems equipped with battery backed write cache come with built-in
191 means of protecting data in the face of power failure. In that case,
192 it is permissible to disable some of DRBD's own safeguards created for
193 the same purpose. This may be beneficial in terms of throughput:
194
195 [source,drbd]
196 ----------------------------
197 resource <resource> {
198 disk {
199 disk-barrier no;
200 disk-flushes no;
201 ...
202 }
203 ...
204 }
205 ----------------------------
206
207
208 [[s-tune-read-balancing]]
209 === Achieving better Read Performance via increased Redundancy
210
211 As detailed in the man page of `drbd.conf` under `read-balancing`,
212 you can increase your read performance by adding more copies of your data.
213
214 As a ballpark figure: with a single node processing read requests, `fio` on
215 a __FusionIO__ card gave us 100k IOPs; after enabling `read-balancing`, the
216 performance jumped to 180k IOPs, i.e. +80%!
217
218 So, in case you're running a read-mostly workload (big databases with lots of
219 random reads), it might be worth a try to turn `read-balancing` on - and,
220 perhaps, add another copy for still more read IO throughput.