Debian Codegraph @

UG9/en/features.adoc

raw
1 [[ch-features]]
2 == DRBD Features
3
4 This chapter discusses various useful DRBD features, and gives some
5 background information about them. Some of these features will be
6 important to most users, some will only be relevant in very specific
7 deployment scenarios. <<p-work>> and <<ch-troubleshooting>> contain
8 instructions on how to enable and use these features in day-to-day
9 operation.
10
11 [[s-single-primary-mode]]
12 === Single-primary mode
13
14 In single-primary mode, a resource is, at any given
15 time, in the primary role on only one cluster member. Since it is
16 guaranteed that only one cluster node manipulates the data at any
17 moment, this mode can be used with any conventional file system (ext3,
18 ext4, XFS etc.).
19
20 Deploying DRBD in single-primary mode is the canonical approach for
21 High-Availability (fail-over capable) clusters.
22
23 [[s-dual-primary-mode]]
24 === Dual-primary mode
25
26 In dual-primary mode a resource can be in the
27 primary role on two nodes at a time. Since concurrent access to the
28 data is thus possible, this mode usually requires the use of a shared cluster
29 file system that utilizes a distributed lock manager.
30 ifndef::drbd-only[]
31 Examples include <<ch-gfs,GFS>> and <<ch-ocfs2,OCFS2>>.
32 endif::drbd-only[]
33
34 Deploying DRBD in dual-primary mode is the preferred approach for
35 load-balancing clusters which require concurrent data access from two
36 nodes, e.g. virtualization environments with a need for live-migration.
37 This mode is disabled by default, and must be enabled
38 explicitly in DRBD's configuration file.
39
40 See <<s-enable-dual-primary>> for information on enabling dual-primary
41 mode for specific resources.
42
43 [[s-replication-protocols]]
44 === Replication modes
45
46 DRBD supports three distinct replication modes, allowing three degrees
47 of replication synchronicity.
48
49 [[fp-protocol-a]]
50 .Protocol A
51 Asynchronous replication protocol. Local write operations on the
52 primary node are considered completed as soon as the local disk write
53 has finished, and the replication packet has been placed in the local
54 TCP send buffer. In the event of forced fail-over, data loss may
55 occur. The data on the standby node is consistent after fail-over;
56 however, the most recent updates performed prior to the crash could be
57 lost. Protocol A is most often used in long distance replication scenarios.
58 ifndef::drbd-only,de-brand[]
59 When used in combination with DRBD Proxy it makes an effective
60 disaster recovery solution. See <<s-drbd-proxy>>, for more information.
61 endif::[]
62
63
64 [[fp-protocol-b]]
65 .Protocol B
66 Memory synchronous (semi-synchronous) replication protocol. Local
67 write operations on the primary node are considered completed as soon
68 as the local disk write has occurred, and the replication packet has
69 reached the peer node. Normally, no writes are lost in case of forced
70 fail-over. However, in the event of simultaneous power failure on both
71 nodes *and* concurrent, irreversible destruction of the primary's data
72 store, the most recent writes completed on the primary may be lost.
73
74 [[fp-protocol-c]]
75 .Protocol C
76 Synchronous replication protocol. Local write operations on the
77 primary node are considered completed only after both the local and
78 the remote disk write(s) have been confirmed. As a result, loss of a
79 single node is guaranteed not to lead to any data loss. Data loss is,
80 of course, inevitable even with this replication protocol if all
81 nodes (resp. their storage subsystems) are irreversibly destroyed at the
82 same time.
83
84 By far, the most commonly used replication protocol in DRBD setups is
85 protocol C.
86
87 The choice of replication protocol influences two factors of your
88 deployment: _protection_ and _latency_. _Throughput_, by contrast, is
89 largely independent of the replication protocol selected.
90
91 See <<s-configure-resource>> for an example resource configuration
92 which demonstrates replication protocol configuration.
93
94 [[s-multi-node]]
95 === More than 2-way redundancy
96
97 With DRBD 9 it's possible to have the data stored simultaneously on more than
98 two cluster nodes.
99
100 While this has been possible before via <<s-three-way-repl,stacking>>, in
101 DRBD 9 this is supported out-of-the-box for (currently) up to 16 nodes.
102 (In practice, using 3-, 4- or perhaps 5-way redundancy via DRBD
103 will make other things the leading cause of downtime.)
104
105 The major difference to the stacking solution is that there's less performance
106 loss, because only one level of data replication is being used.
107
108 // E.g. if availability for a single node is 99%, for two nodes it might
109 // be 99.99%, for three nodes 99.999%
110
111 [[s-automatic-promotion]]
112 === Automatic Promotion of Resources
113
114 Prior to DRBD 9, promoting a resource would be done with the `drbdadm primary` command. With DRBD
115 9, DRBD will automatically promote a resource to primary role when the `auto-promote`
116 option is enabled, and one of its volumes is
117 mounted or opened for writing. As soon as all volumes are unmounted or closed, the role
118 of the resource changes back to secondary.
119
120 Automatic promotion will only succeed if the cluster state allows it (that is, if an
121 explicit `drbdadm primary` command would succeed). Otherwise, mounting or opening
122 the device fails as it did prior to DRBD 9.
123
124
125 [[s-replication-transports]]
126 === Multiple replication transports
127 DRBD supports multiple network transports. As of now two transport
128 implementations are available: TCP and RDMA. Each transport implementation
129 comes as its own kernel module.
130
131 [[s-tcp_transport]]
132 ==== TCP Transport
133 The `drbd_transport_tcp.ko` transport
134 implementation is included with the distribution files of drbd itself.
135 As the name implies, this transport implementation uses the TCP/IP
136 protocol to move data between machines.
137
138 DRBD's replication and synchronization framework socket layer supports
139 multiple low-level transports:
140
141 .TCP over IPv4
142 This is the canonical implementation, and DRBD's default. It may be
143 used on any system that has IPv4 enabled.
144
145 .TCP over IPv6
146 When configured to use standard TCP sockets for replication and
147 synchronization, DRBD can use also IPv6 as its network protocol. This
148 is equivalent in semantics and performance to IPv4, albeit using a
149 different addressing scheme.
150
151 .SDP
152 SDP is an implementation of BSD-style sockets for RDMA capable
153 transports such as InfiniBand. SDP was available as part of the OFED
154 stack of most distributions but is now *considered deprecated*. SDP uses an IPv4-style
155 addressing scheme. Employed over an InfiniBand interconnect, SDP
156 provides a high-throughput, low-latency replication network to DRBD.
157
158 .SuperSockets
159 SuperSockets replace the TCP/IP portions of the stack with a single,
160 monolithic, highly efficient and RDMA capable socket
161 implementation. DRBD can use this socket type for very low latency
162 replication. SuperSockets must run on specific hardware which is
163 currently available from a single vendor, Dolphin Interconnect
164 Solutions.
165
166 [[s-rdma_transport]]
167 ==== RDMA Transport
168 Alternatively the `drbd_transport_rdma.ko` kernel module is available
169 ifndef::de-brand[]
170 from LINBIT
171 endif::de-brand[]
172 .
173 This transport uses the verbs/RDMA API to move data over InfiniBand HCAs, iWARP
174 capable NICs or RoCE capable NICs. In contrast to the BSD sockets API (used by
175 TCP/IP) the verbs/RDMA API allows data movement with very little CPU
176 involvement.
177
178 At high transfer rates it might be possible that the CPU load/memory
179 bandwidth of the tcp transport becomes the limiting factor. You can
180 probably achieve higher transfer rates using the rdma transport with
181 appropriate hardware.
182
183 A transport implementation can be configured for each connection of a
184 resource. See <<s-configuring-transports>> for more details.
185
186 [[s-multiple-paths]]
187 === Multiple paths
188
189 DRBD allows configuring multiple paths per connection. The TCP transport
190 uses only one path at a time for a connection. The RDMA transport is
191 capable of balancing the network traffic over multiple paths of a single
192 connection. see <<s-configuring-multiple-paths>> for more details.
193
194 [[s-resync]]
195 === Efficient synchronization
196
197 (Re-)synchronization is distinct from device replication. While
198 replication occurs on any write event to a resource in the primary
199 role, synchronization is decoupled from incoming writes. Rather, it
200 affects the device as a whole.
201
202 Synchronization is necessary if the replication link has been
203 interrupted for any reason, be it due to failure of the primary node,
204 failure of the secondary node, or interruption of the replication
205 link. Synchronization is efficient in the sense that DRBD does not
206 synchronize modified blocks in the order they were originally written,
207 but in linear order, which has the following consequences:
208
209 * Synchronization is fast, since blocks in which several successive
210 write operations occurred are only synchronized once.
211
212 * Synchronization is also associated with few disk seeks, as blocks
213 are synchronized according to the natural on-disk block layout.
214
215 * During synchronization, the data set on the standby node is partly
216 obsolete and partly already updated. This state of data is called
217 _inconsistent_.
218
219 The service continues to run uninterrupted on the active node, while
220 background synchronization is in progress.
221
222 IMPORTANT: A node with inconsistent data generally cannot be put into
223 operation, thus it is desirable to keep the time period during which a
224 node is inconsistent as short as possible. DRBD does, however, ship
225 with an LVM integration facility that automates the creation of LVM
226 snapshots immediately before synchronization. This ensures that a
227 _consistent_ copy of the data is always available on the peer, even
228 while synchronization is running. See <<s-lvm-snapshots>> for details
229 on using this facility.
230
231 [[s-variable-rate-sync]]
232 ==== Variable-rate synchronization
233
234 In variable-rate synchronization (the default since 8.4), DRBD detects the
235 available bandwidth on the synchronization network, compares it to
236 incoming foreground application I/O, and selects an appropriate
237 synchronization rate based on a fully automatic control loop.
238
239 See <<s-configure-sync-rate-variable>> for configuration suggestions with
240 regard to variable-rate synchronization.
241
242 [[s-fixed_rate_synchronization]]
243 ==== Fixed-rate synchronization
244
245 In fixed-rate synchronization, the amount of data shipped to the
246 synchronizing peer per second (the _synchronization rate_) has a
247 configurable, static upper limit. Based on this limit, you may
248 estimate the expected sync time based on the following simple formula:
249
250 [[eq-resync-time]]
251 [equation]
252 .Synchronization time
253 image::images/resync-time.svg[]
254
255
256 _t~sync~_ is the expected sync time. _D_ is the amount of data to be
257 synchronized, which you are unlikely to have any influence over (this
258 is the amount of data that was modified by your application while the
259 replication link was broken). _R_ is the rate of synchronization,
260 which is configurable -- bounded by the throughput limitations of the
261 replication network and I/O subsystem.
262
263 See <<s-configure-sync-rate>> for configuration suggestions with
264 regard to fixed-rate synchronization.
265
266 [[s-checksum-sync]]
267 ==== Checksum-based synchronization
268
269 [[p-checksum-sync]]
270 The efficiency of DRBD's synchronization algorithm may be further
271 enhanced by using data digests, also known as checksums. When using
272 checksum-based synchronization, then rather than performing a
273 brute-force overwrite of blocks marked out of sync, DRBD _reads_
274 blocks before synchronizing them and computes a hash of the contents
275 currently found on disk. It then compares this hash with one computed
276 from the same sector on the peer, and omits re-writing this block if
277 the hashes match. This can dramatically cut down synchronization times
278 in situation where a filesystem re-writes a sector with identical
279 contents while DRBD is in disconnected mode.
280
281 See <<s-configure-checksum-sync>> for configuration suggestions with
282 regard to synchronization.
283
284
285 [[s-suspended-replication]]
286 === Suspended replication
287
288 If properly configured, DRBD can detect if the
289 replication network is congested, and _suspend_ replication in this
290 case. In this mode, the primary node "pulls ahead" of the secondary --
291 temporarily going out of sync, but still leaving a consistent copy on
292 the secondary. When more bandwidth becomes available, replication
293 automatically resumes and a background synchronization takes place.
294
295 Suspended replication is typically enabled over links with variable
296 bandwidth, such as wide area replication over shared connections
297 between data centers or cloud instances.
298
299 See <<s-configure-congestion-policy>> for details on congestion
300 policies and suspended replication.
301
302 [[s-online-verify]]
303 === On-line device verification
304
305 On-line device verification enables users to do a block-by-block data
306 integrity check between nodes in a very efficient manner.
307
308 Note that _efficient_ refers to efficient use of network bandwidth
309 here, and to the fact that verification does not break redundancy in
310 any way. On-line verification is still a resource-intensive operation,
311 with a noticeable impact on CPU utilization and load average.
312
313 It works by one node (the _verification source_) sequentially
314 calculating a cryptographic digest of every block stored on the
315 lower-level storage device of a particular resource. DRBD then
316 transmits that digest to the peer node(s) (the _verification target(s)_),
317 where it is checked against a digest of the local copy of the affected
318 block. If the digests do not match, the block is marked out-of-sync
319 and may later be synchronized. Because DRBD transmits just the
320 digests, not the full blocks, on-line verification uses network
321 bandwidth very efficiently.
322
323 The process is termed _on-line_ verification because it does not
324 require that the DRBD resource being verified is unused at the time of
325 verification. Thus, though it does carry a slight performance penalty
326 while it is running, on-line verification does not cause service
327 interruption or system down time -- neither during the
328 verification run nor during subsequent synchronization.
329
330 It is a common use case to have on-line verification managed by the
331 local cron daemon, running it, for example, once a week or once a
332 month. See <<s-use-online-verify>> for information on how to enable,
333 invoke, and automate on-line verification.
334
335 [[s-integrity-check]]
336 === Replication traffic integrity checking
337
338 DRBD optionally performs end-to-end message integrity checking using
339 cryptographic message digest algorithms such as MD5, SHA-1, or CRC-32C.
340
341 These message digest algorithms are *not* provided by DRBD, but by the Linux
342 kernel crypto API; DRBD merely uses them. Thus, DRBD is
343 capable of utilizing any message digest algorithm available in a
344 particular system's kernel configuration.
345
346 With this feature enabled, DRBD generates a message digest of every
347 data block it replicates to the peer, which the peer then uses to
348 verify the integrity of the replication packet. If the replicated
349 block can not be verified against the digest, the connection is dropped
350 and immediately re-established; because of the bitmap the typical result is a
351 retransmission. Thus, DRBD replication is protected against several
352 error sources, all of which, if unchecked, would potentially lead to
353 data corruption during the replication process:
354
355 * Bitwise errors ("bit flips") occurring on data in transit between
356 main memory and the network interface on the sending node (which
357 goes undetected by TCP checksumming if it is offloaded to the
358 network card, as is common in recent implementations);
359
360 * Bit flips occurring on data in transit from the network interface to
361 main memory on the receiving node (the same considerations apply for
362 TCP checksum offloading);
363
364 * Any form of corruption due to a race conditions or bugs in network
365 interface firmware or drivers;
366
367 * Bit flips or random corruption injected by some reassembling network
368 component between nodes (if not using direct, back-to-back
369 connections).
370
371 See <<s-configure-integrity-check>> for information on how to enable
372 replication traffic integrity checking.
373
374 [[s-split-brain-notification-and-recovery]]
375 === Split brain notification and automatic recovery
376
377 Split brain is a situation where, due to temporary failure of all
378 network links between cluster nodes, and possibly due to intervention
379 by a cluster management software or human error, both nodes switched
380 to the _Primary_ role while disconnected. This is a potentially harmful
381 state, as it implies that modifications to the data might have been
382 made on either node, without having been replicated to the peer. Thus,
383 it is likely in this situation that two diverging sets of data have
384 been created, which cannot be trivially merged.
385
386 DRBD split brain is distinct from cluster split brain, which is the
387 loss of all connectivity between hosts managed by a distributed
388 cluster management application such as Pacemaker. To avoid confusion,
389 this guide uses the following convention:
390
391 * _Split brain_ refers to DRBD split brain as described in the
392 paragraph above.
393
394 * Loss of all cluster connectivity is referred to as a _cluster
395 partition_, an alternative term for cluster split brain.
396
397 DRBD allows for automatic operator notification (by email or other
398 means) when it detects split brain. See <<s-split-brain-notification>>
399 for details on how to configure this feature.
400
401 While the recommended course of action in this scenario is to
402 <<s-resolve-split-brain,manually resolve>> the split brain and then
403 eliminate its root cause, it may be desirable, in some cases, to
404 automate the process. DRBD has several resolution algorithms available
405 for doing so:
406
407 * *Discarding modifications made on the younger primary.* In this
408 mode, when the network connection is re-established and split brain
409 is discovered, DRBD will discard modifications made, in the
410 meantime, on the node which switched to the primary role _last_.
411
412 * *Discarding modifications made on the older primary.* In this mode,
413 DRBD will discard modifications made, in the meantime, on the node
414 which switched to the primary role _first_.
415
416 * *Discarding modifications on the primary with fewer changes.* In
417 this mode, DRBD will check which of the two nodes has recorded fewer
418 modifications, and will then discard _all_ modifications made on
419 that host.
420
421 * *Graceful recovery from split brain if one host has had no
422 intermediate changes.* In this mode, if one of the hosts has made no
423 modifications at all during split brain, DRBD will simply recover
424 gracefully and declare the split brain resolved. Note that this is a
425 fairly unlikely scenario. Even if both hosts only mounted the file
426 system on the DRBD block device (even read-only), the device
427 contents typically would be modified (e.g. by filesystem journal
428 replay), ruling out the possibility of automatic
429 recovery.
430
431 Whether or not automatic split brain recovery is acceptable depends
432 largely on the individual application. Consider the example of DRBD
433 hosting a database. The discard modifications from host with fewer
434 changes approach may be fine for a web application click-through
435 database. By contrast, it may be totally unacceptable to automatically
436 discard _any_ modifications made to a financial database, requiring
437 manual recovery in any split brain event. Consider your application's
438 requirements carefully before enabling automatic split brain recovery.
439
440 Refer to <<s-automatic-split-brain-recovery-configuration>> for
441 details on configuring DRBD's automatic split brain recovery policies.
442
443 [[s-disk-flush-support]]
444 === Support for disk flushes
445
446 When local block devices such as hard drives or RAID logical disks
447 have write caching enabled, writes to these devices are considered
448 completed as soon as they have reached the volatile cache. Controller
449 manufacturers typically refer to this as write-back mode, the opposite
450 being write-through. If a power outage occurs on a controller in
451 write-back mode, the last writes are never
452 committed to the disk, potentially causing data loss.
453
454 To counteract this, DRBD makes use of disk flushes. A disk flush is a
455 write operation that completes only when the associated data has been
456 committed to stable (non-volatile) storage -- that is to say, it has
457 effectively been written to disk, rather than to the cache. DRBD uses
458 disk flushes for write operations both to its replicated data set and
459 to its meta data. In effect, DRBD circumvents the write cache in
460 situations it deems necessary, as in <<s-activity-log,activity log>>
461 updates or enforcement of implicit write-after-write
462 dependencies. This means additional reliability even in the face of
463 power failure.
464
465 It is important to understand that DRBD can use disk flushes only when
466 layered on top of backing devices that support them. Most reasonably
467 recent kernels support disk flushes for most SCSI and SATA
468 devices. Linux software RAID (md) supports disk flushes for RAID-1
469 provided that all component devices support them too. The same is true for
470 device-mapper devices (LVM2, dm-raid, multipath).
471
472 Controllers with battery-backed write cache (BBWC) use a battery to
473 back up their volatile storage. On such devices, when power is
474 restored after an outage, the controller flushes all pending writes out
475 to disk from the battery-backed cache, ensuring that all
476 writes committed to the volatile cache are actually transferred to
477 stable storage. When running DRBD on top of such devices, it may be
478 acceptable to disable disk flushes, thereby improving DRBD's write
479 performance. See <<s-disable-flushes>> for details.
480
481 [[s-trim-discard]]
482 === Trim/Discard support
483
484 _Trim_/_Discard_ are two names for the same feature: a request to a storage
485 system, telling it that some data range is not being used anymorefootnote:[For
486 example, a deleted file's data.] and can get recycled. +
487 This call originates in Flash-based storages (SSDs, FusionIO cards, etc.),
488 which cannot easily _rewrite_ a sector but instead have to _erase_ and write
489 the (new) data again (incurring some latency cost). For more details, see e.g.
490 the https://en.wikipedia.org/wiki/Trim_%28computing%29[wikipedia page].
491
492
493 Since 8.4.3 DRBD includes support for _Trim_/_Discard_. You don't need to configure
494 or enable anything; if DRBD detects that the local (underlying) storage system
495 allows using these commands, it will transparently enable them and pass such
496 requests through.
497
498 The effect is that e.g. a recent-enough `mkfs.ext4` on a multi-TB volume can
499 shorten the initial sync time to a few seconds to minutes - just by telling
500 DRBD (which will relay that information to all connected nodes) that most/all
501 of the storage is now to be seen as invalidated.
502
503 Nodes that connect to that resource later on will not have seen the
504 _Trim_/_Discard_ requests, and will therefore start a full resync; depending on
505 kernel version and file system a call to `fstrim` might give the wanted result,
506 though.
507
508 NOTE: even if you don't have storage with _Trim_/_Discard_ support, some
509 virtual block devices will provide you with the same feature, for example
510 Thin LVM.
511
512
513 [[s-handling-disk-errors]]
514 === Disk error handling strategies
515
516 If a hard drive fails which is used as a backing block device for DRBD on one
517 of the nodes, DRBD may either pass on the I/O error to the upper
518 layer (usually the file system) or it can mask I/O errors from upper
519 layers.
520
521 [[fp-io-error-pass-on]]
522 .Passing on I/O errors
523 If DRBD is configured to pass on I/O errors, any such errors occurring
524 on the lower-level device are transparently passed to upper I/O
525 layers. Thus, it is left to upper layers to deal with such errors
526 (this may result in a file system being remounted read-only, for
527 example). This strategy does not ensure service continuity, and is
528 hence not recommended for most users.
529
530 [[fp-io-error-detach]]
531 .Masking I/O errors
532 If DRBD is configured to _detach_ on lower-level I/O error, DRBD will
533 do so, automatically, upon occurrence of the first lower-level I/O
534 error. The I/O error is masked from upper layers while DRBD
535 transparently fetches the affected block from a peer node, over the
536 network. From then onwards, DRBD is said to operate in diskless mode,
537 and carries out all subsequent I/O operations, read and write, on the
538 peer node(s) only. Performance in this mode will be reduced,
539 but the service continues without interruption, and can be moved to
540 the peer node in a deliberate fashion at a convenient time.
541
542 See <<s-configure-io-error-behavior>> for information on configuring
543 I/O error handling strategies for DRBD.
544
545 [[s-outdate]]
546 === Strategies for dealing with outdated data
547
548 DRBD distinguishes between _inconsistent_ and _outdated_
549 data. Inconsistent data is data that cannot be expected to be
550 accessible and useful in any manner. The prime example for this is
551 data on a node that is currently the target of an on-going
552 synchronization. Data on such a node is part obsolete, part up to
553 date, and impossible to identify as either. Thus, for example, if the
554 device holds a filesystem (as is commonly the case), that filesystem
555 would be unexpected to mount or even pass an automatic filesystem
556 check.
557
558 Outdated data, by contrast, is data on a secondary node that is
559 consistent, but no longer in sync with the primary node. This would
560 occur in any interruption of the replication link, whether temporary
561 or permanent. Data on an outdated, disconnected secondary node is
562 expected to be clean, but it reflects a state of the peer node some
563 time past. In order to avoid services using outdated data, DRBD
564 disallows <<s-resource-roles,promoting a resource>> that
565 is in the outdated state.
566
567 DRBD has interfaces that allow an external application to outdate a
568 secondary node as soon as a network interruption occurs. DRBD will
569 then refuse to switch the node to the primary role, preventing
570 applications from using the outdated data.
571 ifndef::drbd-only[]
572 A complete implementation
573 of this functionality exists for the <<ch-pacemaker,Pacemaker cluster
574 management framework>> (where it uses a communication channel separate
575 from the DRBD replication link). However, the interfaces are generic
576 and may be easily used by any other cluster management application.
577 endif::drbd-only[]
578
579 Whenever an outdated resource has its replication link re-established,
580 its outdated flag is automatically cleared. A <<s-resync,background
581 synchronization>> then follows.
582
583 [[s-three-way-repl]]
584 === Three-way replication via stacking
585
586 NOTE: Available in DRBD version 8.3.0 and above; deprecated in DRBD version
587 9.x, as more nodes can be implemented on a single level. See
588 <<s-drbdconf-conns>> for details.
589
590 When using three-way replication, DRBD adds a third node to an
591 existing 2-node cluster and replicates data to that node, where it can
592 be used for backup and disaster recovery purposes.
593 ifndef::drbd-only,de-brand[]
594 This type of configuration generally involves <<s-drbd-proxy>>.
595 endif::[]
596
597 Three-way replication works by adding another, _stacked_ DRBD resource
598 on top of the existing resource holding your production data, as seen
599 in this illustration:
600
601 .DRBD resource stacking
602 image::images/drbd-resource-stacking.svg[]
603
604 The stacked resource is replicated using asynchronous replication
605 (DRBD protocol A), whereas the production data would usually make use
606 of synchronous replication (DRBD protocol C).
607
608 Three-way replication can be used permanently, where the third node is
609 continuously updated with data from the production
610 cluster. Alternatively, it may also be employed on demand, where the
611 production cluster is normally disconnected from the backup site, and
612 site-to-site synchronization is performed on a regular basis, for
613 example by running a nightly cron job.
614
615 ifndef::drbd-only,de-brand[]
616 [[s-drbd-proxy]]
617 === Long-distance replication via DRBD Proxy
618
619 DRBD's <<s-replication-protocols,protocol A>> is asynchronous, but the
620 writing application will block as soon as the socket output buffer is
621 full (see the `sndbuf-size` option in the man page of `drbd.conf`). In that
622 event, the writing application has to wait until some of the data written
623 runs off through a possibly small bandwidth network link.
624
625 The average write bandwidth is limited by available bandwidth of the
626 network link. Write bursts can only be handled gracefully if they fit
627 into the limited socket output buffer.
628
629 You can mitigate this by DRBD Proxy's buffering mechanism. DRBD Proxy
630 will place changed data from the DRBD device on the primary node into
631 its buffers. DRBD Proxy's buffer size is freely configurable, only
632 limited by the address room size and available physical RAM.
633
634 Optionally DRBD Proxy can be configured to compress and decompress the
635 data it forwards. Compression and decompression of DRBD's data packets
636 might slightly increase latency. However, when the bandwidth of the network
637 link is the limiting factor, the gain in shortening transmit time
638 outweighs the compression and decompression overhead.
639
640 Compression and decompression were implemented with multi core SMP
641 systems in mind, and can utilize multiple CPU cores.
642
643 The fact that most block I/O data compresses very well and therefore
644 the effective bandwidth increases justifies the use of the DRBD
645 Proxy even with DRBD protocols B and C.
646
647 See <<s-using-drbd-proxy>> for information on configuring DRBD Proxy.
648
649 NOTE: DRBD Proxy is one of the few parts of the DRBD product family that is
650 not published under an open source license. Please contact
651 [email protected] or [email protected] for an evaluation license.
652 endif::[]
653
654 [[s-truck-based-replication]]
655 === Truck based replication
656
657 Truck based replication, also known as disk shipping, is a means of
658 preseeding a remote site with data to be replicated, by physically
659 shipping storage media to the remote site. This is particularly suited
660 for situations where
661
662 * the total amount of data to be replicated is fairly
663 large (more than a few hundreds of gigabytes);
664
665 * the expected rate of change of the data to be replicated is less
666 than enormous;
667
668 * the available network bandwidth between sites is
669 limited.
670
671 In such situations, without truck based replication, DRBD would
672 require a very long initial device synchronization (on the order of
673 weeks, months, or years). Truck based replication allows to ship a data seed
674 to the remote site, and so drastically reduces the initial synchronization
675 time. See <<s-using-truck-based-replication>> for details on this use
676 case.
677
678 [[s-floating-peers]]
679 === Floating peers
680
681 NOTE: This feature is available in DRBD versions 8.3.2 and above.
682
683 A somewhat special use case for DRBD is the _floating peers_
684 configuration. In floating peer setups, DRBD peers are not tied to
685 specific named hosts (as in conventional configurations), but instead
686 have the ability to float between several hosts. In such a
687 configuration, DRBD identifies peers by IP address, rather than by
688 host name.
689
690 ifndef::drbd-only[]
691 For more information about managing floating peer configurations, see
692 <<s-pacemaker-floating-peers>>.
693 endif::drbd-only[]
694
695 [[s-rebalance]]
696 === Data rebalancing (horizontal storage scaling)
697
698 If your company's policy says that 3-way redundancy is needed, you need
699 at least 3 servers for your setup.
700
701 Now, as your storage demands grow, you will encounter the need for
702 additional servers. Rather than having to buy 3 more servers at the same
703 time, you can _rebalance_ your data across a single additional node.
704
705 .DRBD data rebalancing
706 image::images/rebalance.svg[]
707
708 //[scaledwidth="75"]
709
710 In the figure above you can see the _before_ and _after_ states: from
711 3 nodes with three 25TiB volumes each (for a net 75TiB), to 4 nodes,
712 with net 100TiB.
713
714 DRBD 9 makes it possible to do an online, live migration of the data; please see
715 <<s-rebalance-workflow>> for the exact steps needed.
716
717 [[s-drbd-client]]
718 === DRBD client
719
720 With the multiple-peer feature of DRBD a number of interesting use-cases have
721 been added, for example the _DRBD client_.
722
723 The basic idea is that the DRBD _backend_ can consist of 3, 4, or more nodes
724 (depending on the policy of required redundancy); but, as DRBD 9 can connect more
725 nodes than that. DRBD works then as a storage access protocol in addition to
726 storage replication.
727
728 All write requests executed on a primary _DRBD client_ gets shipped to all
729 nodes equipped with storage. Read requests are only shipped to one of the server
730 nodes. The _DRBD client_ will evenly distribute the read requests among all
731 available server nodes.
732
733 See <<s-permanently-diskless-nodes>> for more information.
734
735 [[s-feature-quorum]]
736 === Quorum
737
738 In order to avoid split brain or diverging data of replicas one has to configure
739 fencing. It turns out that in real world deployments node fencing is not popular
740 because often mistakes happen in planning or deploying it.
741
742 In the moment a data-set has 3 replicas one can rely on the quorum implementation
743 within DRBD instead of cluster manager level fencing.
744 ifndef::drbd-only[]
745 Pacemaker gets informed about quorum or loss-of-quorum via the master score of
746 the resource.
747 endif::drbd-only[]
748
749 DRBD's quorum can be used with any kind of Linux based service. In case the
750 service terminates in the moment it is exposed to an IO-error the on quorum
751 loss behavior is very elegant.
752 In case the service does not terminate upon IO-error the systems needs to
753 be configured to reboot a primary node that looses quorum.
754
755 See <<s-configuring-quorum>> for more information.
756
757 [[s-feature-quorum-tiebreaker]]
758 ==== Tiebreaker
759
760 [NOTE]
761 ====================
762 The quorum tiebreaker feature is available in DRBD versions 9.0.18 and above.
763 ====================
764
765 The fundamental problem with two node clusters is that in the moment they lose
766 connectivity we have two partitions and none of them has quorum, which results
767 in the cluster halting the service. This problem can be mitigated by adding a
768 third, diskless node to the cluster which will then act as a quorum tiebreaker.
769
770 See <<s-configuring-quorum-tiebreaker>> for more information.
771
772 ifndef::drbd-only[]
773 [[s-feature-VCS]]
774 === DRBD integration for VCS
775
776 Veritas Cluster Server (or Veritas Infoscale Availability) is a commercial alternative
777 to the Pacemaker open source software. In case you need to integrate DRBD resources
778 into a VCS setup please see the README in
779 https://github.com/LINBIT/drbd-utils/tree/master/scripts/VCS[drbd-utils/scripts/VCS]
780 on github.
781 endif::drbd-only[]