[[ch-throughput]] == Optimizing DRBD throughput This chapter deals with optimizing DRBD throughput. It examines some hardware considerations with regard to throughput optimization, and details tuning recommendations for that purpose. [[s-throughput-hardware]] === Hardware considerations DRBD throughput is affected by both the bandwidth of the underlying I/O subsystem (disks, controllers, and corresponding caches), and the bandwidth of the replication network. .I/O subsystem throughput indexterm:[throughput]I/O subsystem throughput is determined, largely, by the number and type of storage units (disks, SSDs, other Flash storage [like FusionIO], ...) that can be written to in parallel. A single, reasonably recent, SCSI or SAS disk will typically allow streaming writes of roughly 40MiB/s to the single disk; an SSD will do 300MiB/s; one of the recent Flash storages (NVMe) will be at 1GiB/s. When deployed in a striping configuration, the I/O subsystem will parallelize writes across disks, effectively multiplying a single disk's throughput by the number of stripes in the configuration. Thus the same, 40MB/s disks will allow effective throughput of 120MB/s in a RAID-0 or RAID-1+0 configuration with three stripes, or 200MB/s with five stripes; with SSDs and/or NVMe you can easily get to 1GiB/sec. A RAID-controller with RAM and a <> can speed up short spikes (by buffering them), and so too-short benchmark tests might show speeds like 1GiB/s too; for sustained writes its buffers will just run full, and then not be of much help, though. NOTE: Disk _mirroring_ (RAID-1) in hardware typically has little, if any, effect on throughput. Disk _striping with parity_ (RAID-5) does have an effect on throughput, usually an adverse one when compared to striping; RAID-5 and RAID-6 in software even more so. .Network throughput indexterm:[throughput]Network throughput is usually determined by the amount of traffic present on the network, and on the throughput of any routing/switching infrastructure present. These concerns are, however, largely irrelevant in DRBD replication links which are normally dedicated, back-to-back network connections. Thus, network throughput may be improved either by switching to a higher-throughput hardware (such as 10 Gigabit Ethernet, or 56GiB Infiniband), or by using link aggregation over several network links, as one may do using the Linux indexterm:[bonding driver]`bonding` network driver. [[s-throughput-overhead-expectations]] === Throughput overhead expectations When estimating the throughput overhead associated with DRBD, it is important to consider the following natural limitations: * DRBD throughput is limited by that of the raw I/O subsystem. * DRBD throughput is limited by the available network bandwidth. The _lower_ of these two establishes the theoretical throughput _maximum_ available to DRBD. DRBD then reduces that throughput number by its additional overhead, which can be expected to be less than 3 percent. * Consider the example of two cluster nodes containing I/O subsystems capable of 600 MB/s throughput, with a Gigabit Ethernet link available between them. Gigabit Ethernet can be expected to produce 110 MB/s throughput for TCP connections, thus the network connection would be the bottleneck in this configuration and one would expect about 110 MB/s maximum DRBD throughput. * By contrast, if the I/O subsystem is capable of only 80 MB/s for sustained writes, then it constitutes the bottleneck, and you should expect only about 77 MB/s maximum DRBD throughput. [[s-throughput-tuning]] === Tuning recommendations DRBD offers a number of configuration options which may have an effect on your system's throughput. This section list some recommendations for tuning for throughput. However, since throughput is largely hardware dependent, the effects of tweaking the options described here may vary greatly from system to system. It is important to understand that these recommendations should not be interpreted as "silver bullets" which would magically remove any and all throughput bottlenecks. [[s-tune-max-buffer-max-epoch-size]] ==== Setting `max-buffers` and `max-epoch-size` These options affect write performance on the secondary nodes. `max-buffers` is the maximum number of buffers DRBD allocates for writing data to disk while `max-epoch-size` is the maximum number of write requests permitted between two write barriers. `max-buffers` must be equal or bigger to `max-epoch-size` to increase performance. The default for both is 2048; setting it to around 8000 should be fine for most reasonably high-performance hardware RAID controllers. [source,drbd] ---------------------------- resource { net { max-buffers 8000; max-epoch-size 8000; ... } ... } ---------------------------- [[s-tune-sndbuf-size]] ==== Tuning the TCP send buffer size The TCP send buffer is a memory buffer for outgoing TCP traffic. By default, it is set to a size of 128 KiB. For use in high-throughput networks (such as dedicated Gigabit Ethernet or load-balanced bonded connections), it may make sense to increase this to a size of 2MiB, or perhaps even more. Send buffer sizes of more than 16MiB are generally not recommended (and are also unlikely to produce any throughput improvement). [source,drbd] ---------------------------- resource { net { sndbuf-size 2M; ... } ... } ---------------------------- DRBD also supports TCP send buffer auto-tuning. After enabling this feature, DRBD will dynamically select an appropriate TCP send buffer size. TCP send buffer auto tuning is enabled by simply setting the buffer size to zero: [source,drbd] ---------------------------- resource { net { sndbuf-size 0; ... } ... } ---------------------------- Please note that your ``sysctl``'s settings `net.ipv4.tcp_rmem` and `net.ipv4.tcp_wmem` will still influence the behaviour; you should check these settings, and perhaps set them similar to `131072 1048576 16777216` (minimum 128kiB, default 1MiB, max 16MiB). WARNING: `net.ipv4.tcp_mem` is a different beast, with a different unit - do not touch, wrong values can easily push your machine into out-of-memory situations! [[s-tune-al-extents]] ==== Tuning the Activity Log size If the application using DRBD is write intensive in the sense that it frequently issues small writes scattered across the device, it is usually advisable to use a fairly large activity log. Otherwise, frequent metadata updates may be detrimental to write performance. [source,drbd] ---------------------------- resource { disk { al-extents 6007; ... } ... } ---------------------------- //// For tuning on striped RAID setups, please see the <> and <> settings, too. //// [[s-tune-disable-barriers]] ==== Disabling barriers and disk flushes WARNING: The recommendations outlined in this section should be applied _only_ to systems with non-volatile (battery backed) controller caches. Systems equipped with battery backed write cache come with built-in means of protecting data in the face of power failure. In that case, it is permissible to disable some of DRBD's own safeguards created for the same purpose. This may be beneficial in terms of throughput: [source,drbd] ---------------------------- resource { disk { disk-barrier no; disk-flushes no; ... } ... } ---------------------------- [[s-tune-read-balancing]] === Achieving better Read Performance via increased Redundancy As detailed in the man page of `drbd.conf` under `read-balancing`, you can increase your read performance by adding more copies of your data. As a ballpark figure: with a single node processing read requests, `fio` on a __FusionIO__ card gave us 100k IOPs; after enabling `read-balancing`, the performance jumped to 180k IOPs, i.e. +80%! So, in case you're running a read-mostly workload (big databases with lots of random reads), it might be worth a try to turn `read-balancing` on - and, perhaps, add another copy for still more read IO throughput.