| 1 |
[[ch-features]] |
| 2 |
== DRBD Features |
| 3 |
|
| 4 |
This chapter discusses various useful DRBD features, and gives some |
| 5 |
background information about them. Some of these features will be |
| 6 |
important to most users, some will only be relevant in very specific |
| 7 |
deployment scenarios. <<p-work>> and <<ch-troubleshooting>> contain |
| 8 |
instructions on how to enable and use these features in day-to-day |
| 9 |
operation. |
| 10 |
|
| 11 |
[[s-single-primary-mode]] |
| 12 |
=== Single-primary mode |
| 13 |
|
| 14 |
In single-primary mode, a resource is, at any given |
| 15 |
time, in the primary role on only one cluster member. Since it is |
| 16 |
guaranteed that only one cluster node manipulates the data at any |
| 17 |
moment, this mode can be used with any conventional file system (ext3, |
| 18 |
ext4, XFS etc.). |
| 19 |
|
| 20 |
Deploying DRBD in single-primary mode is the canonical approach for |
| 21 |
High-Availability (fail-over capable) clusters. |
| 22 |
|
| 23 |
[[s-dual-primary-mode]] |
| 24 |
=== Dual-primary mode |
| 25 |
|
| 26 |
In dual-primary mode a resource can be in the |
| 27 |
primary role on two nodes at a time. Since concurrent access to the |
| 28 |
data is thus possible, this mode usually requires the use of a shared cluster |
| 29 |
file system that utilizes a distributed lock manager. |
| 30 |
ifndef::drbd-only[] |
| 31 |
Examples include <<ch-gfs,GFS>> and <<ch-ocfs2,OCFS2>>. |
| 32 |
endif::drbd-only[] |
| 33 |
|
| 34 |
Deploying DRBD in dual-primary mode is the preferred approach for |
| 35 |
load-balancing clusters which require concurrent data access from two |
| 36 |
nodes, e.g. virtualization environments with a need for live-migration. |
| 37 |
This mode is disabled by default, and must be enabled |
| 38 |
explicitly in DRBD's configuration file. |
| 39 |
|
| 40 |
See <<s-enable-dual-primary>> for information on enabling dual-primary |
| 41 |
mode for specific resources. |
| 42 |
|
| 43 |
[[s-replication-protocols]] |
| 44 |
=== Replication modes |
| 45 |
|
| 46 |
DRBD supports three distinct replication modes, allowing three degrees |
| 47 |
of replication synchronicity. |
| 48 |
|
| 49 |
[[fp-protocol-a]] |
| 50 |
.Protocol A |
| 51 |
Asynchronous replication protocol. Local write operations on the |
| 52 |
primary node are considered completed as soon as the local disk write |
| 53 |
has finished, and the replication packet has been placed in the local |
| 54 |
TCP send buffer. In the event of forced fail-over, data loss may |
| 55 |
occur. The data on the standby node is consistent after fail-over; |
| 56 |
however, the most recent updates performed prior to the crash could be |
| 57 |
lost. Protocol A is most often used in long distance replication scenarios. |
| 58 |
ifndef::drbd-only,de-brand[] |
| 59 |
When used in combination with DRBD Proxy it makes an effective |
| 60 |
disaster recovery solution. See <<s-drbd-proxy>>, for more information. |
| 61 |
endif::[] |
| 62 |
|
| 63 |
|
| 64 |
[[fp-protocol-b]] |
| 65 |
.Protocol B |
| 66 |
Memory synchronous (semi-synchronous) replication protocol. Local |
| 67 |
write operations on the primary node are considered completed as soon |
| 68 |
as the local disk write has occurred, and the replication packet has |
| 69 |
reached the peer node. Normally, no writes are lost in case of forced |
| 70 |
fail-over. However, in the event of simultaneous power failure on both |
| 71 |
nodes *and* concurrent, irreversible destruction of the primary's data |
| 72 |
store, the most recent writes completed on the primary may be lost. |
| 73 |
|
| 74 |
[[fp-protocol-c]] |
| 75 |
.Protocol C |
| 76 |
Synchronous replication protocol. Local write operations on the |
| 77 |
primary node are considered completed only after both the local and |
| 78 |
the remote disk write(s) have been confirmed. As a result, loss of a |
| 79 |
single node is guaranteed not to lead to any data loss. Data loss is, |
| 80 |
of course, inevitable even with this replication protocol if all |
| 81 |
nodes (resp. their storage subsystems) are irreversibly destroyed at the |
| 82 |
same time. |
| 83 |
|
| 84 |
By far, the most commonly used replication protocol in DRBD setups is |
| 85 |
protocol C. |
| 86 |
|
| 87 |
The choice of replication protocol influences two factors of your |
| 88 |
deployment: _protection_ and _latency_. _Throughput_, by contrast, is |
| 89 |
largely independent of the replication protocol selected. |
| 90 |
|
| 91 |
See <<s-configure-resource>> for an example resource configuration |
| 92 |
which demonstrates replication protocol configuration. |
| 93 |
|
| 94 |
[[s-multi-node]] |
| 95 |
=== More than 2-way redundancy |
| 96 |
|
| 97 |
With DRBD 9 it's possible to have the data stored simultaneously on more than |
| 98 |
two cluster nodes. |
| 99 |
|
| 100 |
While this has been possible before via <<s-three-way-repl,stacking>>, in |
| 101 |
DRBD 9 this is supported out-of-the-box for (currently) up to 16 nodes. |
| 102 |
(In practice, using 3-, 4- or perhaps 5-way redundancy via DRBD |
| 103 |
will make other things the leading cause of downtime.) |
| 104 |
|
| 105 |
The major difference to the stacking solution is that there's less performance |
| 106 |
loss, because only one level of data replication is being used. |
| 107 |
|
| 108 |
// E.g. if availability for a single node is 99%, for two nodes it might |
| 109 |
// be 99.99%, for three nodes 99.999% |
| 110 |
|
| 111 |
[[s-automatic-promotion]] |
| 112 |
=== Automatic Promotion of Resources |
| 113 |
|
| 114 |
Prior to DRBD 9, promoting a resource would be done with the `drbdadm primary` command. With DRBD |
| 115 |
9, DRBD will automatically promote a resource to primary role when the `auto-promote` |
| 116 |
option is enabled, and one of its volumes is |
| 117 |
mounted or opened for writing. As soon as all volumes are unmounted or closed, the role |
| 118 |
of the resource changes back to secondary. |
| 119 |
|
| 120 |
Automatic promotion will only succeed if the cluster state allows it (that is, if an |
| 121 |
explicit `drbdadm primary` command would succeed). Otherwise, mounting or opening |
| 122 |
the device fails as it did prior to DRBD 9. |
| 123 |
|
| 124 |
|
| 125 |
[[s-replication-transports]] |
| 126 |
=== Multiple replication transports |
| 127 |
DRBD supports multiple network transports. As of now two transport |
| 128 |
implementations are available: TCP and RDMA. Each transport implementation |
| 129 |
comes as its own kernel module. |
| 130 |
|
| 131 |
[[s-tcp_transport]] |
| 132 |
==== TCP Transport |
| 133 |
The `drbd_transport_tcp.ko` transport |
| 134 |
implementation is included with the distribution files of drbd itself. |
| 135 |
As the name implies, this transport implementation uses the TCP/IP |
| 136 |
protocol to move data between machines. |
| 137 |
|
| 138 |
DRBD's replication and synchronization framework socket layer supports |
| 139 |
multiple low-level transports: |
| 140 |
|
| 141 |
.TCP over IPv4 |
| 142 |
This is the canonical implementation, and DRBD's default. It may be |
| 143 |
used on any system that has IPv4 enabled. |
| 144 |
|
| 145 |
.TCP over IPv6 |
| 146 |
When configured to use standard TCP sockets for replication and |
| 147 |
synchronization, DRBD can use also IPv6 as its network protocol. This |
| 148 |
is equivalent in semantics and performance to IPv4, albeit using a |
| 149 |
different addressing scheme. |
| 150 |
|
| 151 |
.SDP |
| 152 |
SDP is an implementation of BSD-style sockets for RDMA capable |
| 153 |
transports such as InfiniBand. SDP was available as part of the OFED |
| 154 |
stack of most distributions but is now *considered deprecated*. SDP uses an IPv4-style |
| 155 |
addressing scheme. Employed over an InfiniBand interconnect, SDP |
| 156 |
provides a high-throughput, low-latency replication network to DRBD. |
| 157 |
|
| 158 |
.SuperSockets |
| 159 |
SuperSockets replace the TCP/IP portions of the stack with a single, |
| 160 |
monolithic, highly efficient and RDMA capable socket |
| 161 |
implementation. DRBD can use this socket type for very low latency |
| 162 |
replication. SuperSockets must run on specific hardware which is |
| 163 |
currently available from a single vendor, Dolphin Interconnect |
| 164 |
Solutions. |
| 165 |
|
| 166 |
[[s-rdma_transport]] |
| 167 |
==== RDMA Transport |
| 168 |
Alternatively the `drbd_transport_rdma.ko` kernel module is available |
| 169 |
ifndef::de-brand[] |
| 170 |
from LINBIT |
| 171 |
endif::de-brand[] |
| 172 |
. |
| 173 |
This transport uses the verbs/RDMA API to move data over InfiniBand HCAs, iWARP |
| 174 |
capable NICs or RoCE capable NICs. In contrast to the BSD sockets API (used by |
| 175 |
TCP/IP) the verbs/RDMA API allows data movement with very little CPU |
| 176 |
involvement. |
| 177 |
|
| 178 |
At high transfer rates it might be possible that the CPU load/memory |
| 179 |
bandwidth of the tcp transport becomes the limiting factor. You can |
| 180 |
probably achieve higher transfer rates using the rdma transport with |
| 181 |
appropriate hardware. |
| 182 |
|
| 183 |
A transport implementation can be configured for each connection of a |
| 184 |
resource. See <<s-configuring-transports>> for more details. |
| 185 |
|
| 186 |
[[s-multiple-paths]] |
| 187 |
=== Multiple paths |
| 188 |
|
| 189 |
DRBD allows configuring multiple paths per connection. The TCP transport |
| 190 |
uses only one path at a time for a connection. The RDMA transport is |
| 191 |
capable of balancing the network traffic over multiple paths of a single |
| 192 |
connection. see <<s-configuring-multiple-paths>> for more details. |
| 193 |
|
| 194 |
[[s-resync]] |
| 195 |
=== Efficient synchronization |
| 196 |
|
| 197 |
(Re-)synchronization is distinct from device replication. While |
| 198 |
replication occurs on any write event to a resource in the primary |
| 199 |
role, synchronization is decoupled from incoming writes. Rather, it |
| 200 |
affects the device as a whole. |
| 201 |
|
| 202 |
Synchronization is necessary if the replication link has been |
| 203 |
interrupted for any reason, be it due to failure of the primary node, |
| 204 |
failure of the secondary node, or interruption of the replication |
| 205 |
link. Synchronization is efficient in the sense that DRBD does not |
| 206 |
synchronize modified blocks in the order they were originally written, |
| 207 |
but in linear order, which has the following consequences: |
| 208 |
|
| 209 |
* Synchronization is fast, since blocks in which several successive |
| 210 |
write operations occurred are only synchronized once. |
| 211 |
|
| 212 |
* Synchronization is also associated with few disk seeks, as blocks |
| 213 |
are synchronized according to the natural on-disk block layout. |
| 214 |
|
| 215 |
* During synchronization, the data set on the standby node is partly |
| 216 |
obsolete and partly already updated. This state of data is called |
| 217 |
_inconsistent_. |
| 218 |
|
| 219 |
The service continues to run uninterrupted on the active node, while |
| 220 |
background synchronization is in progress. |
| 221 |
|
| 222 |
IMPORTANT: A node with inconsistent data generally cannot be put into |
| 223 |
operation, thus it is desirable to keep the time period during which a |
| 224 |
node is inconsistent as short as possible. DRBD does, however, ship |
| 225 |
with an LVM integration facility that automates the creation of LVM |
| 226 |
snapshots immediately before synchronization. This ensures that a |
| 227 |
_consistent_ copy of the data is always available on the peer, even |
| 228 |
while synchronization is running. See <<s-lvm-snapshots>> for details |
| 229 |
on using this facility. |
| 230 |
|
| 231 |
[[s-variable-rate-sync]] |
| 232 |
==== Variable-rate synchronization |
| 233 |
|
| 234 |
In variable-rate synchronization (the default since 8.4), DRBD detects the |
| 235 |
available bandwidth on the synchronization network, compares it to |
| 236 |
incoming foreground application I/O, and selects an appropriate |
| 237 |
synchronization rate based on a fully automatic control loop. |
| 238 |
|
| 239 |
See <<s-configure-sync-rate-variable>> for configuration suggestions with |
| 240 |
regard to variable-rate synchronization. |
| 241 |
|
| 242 |
[[s-fixed_rate_synchronization]] |
| 243 |
==== Fixed-rate synchronization |
| 244 |
|
| 245 |
In fixed-rate synchronization, the amount of data shipped to the |
| 246 |
synchronizing peer per second (the _synchronization rate_) has a |
| 247 |
configurable, static upper limit. Based on this limit, you may |
| 248 |
estimate the expected sync time based on the following simple formula: |
| 249 |
|
| 250 |
[[eq-resync-time]] |
| 251 |
[equation] |
| 252 |
.Synchronization time |
| 253 |
image::images/resync-time.svg[] |
| 254 |
|
| 255 |
|
| 256 |
_t~sync~_ is the expected sync time. _D_ is the amount of data to be |
| 257 |
synchronized, which you are unlikely to have any influence over (this |
| 258 |
is the amount of data that was modified by your application while the |
| 259 |
replication link was broken). _R_ is the rate of synchronization, |
| 260 |
which is configurable -- bounded by the throughput limitations of the |
| 261 |
replication network and I/O subsystem. |
| 262 |
|
| 263 |
See <<s-configure-sync-rate>> for configuration suggestions with |
| 264 |
regard to fixed-rate synchronization. |
| 265 |
|
| 266 |
[[s-checksum-sync]] |
| 267 |
==== Checksum-based synchronization |
| 268 |
|
| 269 |
[[p-checksum-sync]] |
| 270 |
The efficiency of DRBD's synchronization algorithm may be further |
| 271 |
enhanced by using data digests, also known as checksums. When using |
| 272 |
checksum-based synchronization, then rather than performing a |
| 273 |
brute-force overwrite of blocks marked out of sync, DRBD _reads_ |
| 274 |
blocks before synchronizing them and computes a hash of the contents |
| 275 |
currently found on disk. It then compares this hash with one computed |
| 276 |
from the same sector on the peer, and omits re-writing this block if |
| 277 |
the hashes match. This can dramatically cut down synchronization times |
| 278 |
in situation where a filesystem re-writes a sector with identical |
| 279 |
contents while DRBD is in disconnected mode. |
| 280 |
|
| 281 |
See <<s-configure-checksum-sync>> for configuration suggestions with |
| 282 |
regard to synchronization. |
| 283 |
|
| 284 |
|
| 285 |
[[s-suspended-replication]] |
| 286 |
=== Suspended replication |
| 287 |
|
| 288 |
If properly configured, DRBD can detect if the |
| 289 |
replication network is congested, and _suspend_ replication in this |
| 290 |
case. In this mode, the primary node "pulls ahead" of the secondary -- |
| 291 |
temporarily going out of sync, but still leaving a consistent copy on |
| 292 |
the secondary. When more bandwidth becomes available, replication |
| 293 |
automatically resumes and a background synchronization takes place. |
| 294 |
|
| 295 |
Suspended replication is typically enabled over links with variable |
| 296 |
bandwidth, such as wide area replication over shared connections |
| 297 |
between data centers or cloud instances. |
| 298 |
|
| 299 |
See <<s-configure-congestion-policy>> for details on congestion |
| 300 |
policies and suspended replication. |
| 301 |
|
| 302 |
[[s-online-verify]] |
| 303 |
=== On-line device verification |
| 304 |
|
| 305 |
On-line device verification enables users to do a block-by-block data |
| 306 |
integrity check between nodes in a very efficient manner. |
| 307 |
|
| 308 |
Note that _efficient_ refers to efficient use of network bandwidth |
| 309 |
here, and to the fact that verification does not break redundancy in |
| 310 |
any way. On-line verification is still a resource-intensive operation, |
| 311 |
with a noticeable impact on CPU utilization and load average. |
| 312 |
|
| 313 |
It works by one node (the _verification source_) sequentially |
| 314 |
calculating a cryptographic digest of every block stored on the |
| 315 |
lower-level storage device of a particular resource. DRBD then |
| 316 |
transmits that digest to the peer node(s) (the _verification target(s)_), |
| 317 |
where it is checked against a digest of the local copy of the affected |
| 318 |
block. If the digests do not match, the block is marked out-of-sync |
| 319 |
and may later be synchronized. Because DRBD transmits just the |
| 320 |
digests, not the full blocks, on-line verification uses network |
| 321 |
bandwidth very efficiently. |
| 322 |
|
| 323 |
The process is termed _on-line_ verification because it does not |
| 324 |
require that the DRBD resource being verified is unused at the time of |
| 325 |
verification. Thus, though it does carry a slight performance penalty |
| 326 |
while it is running, on-line verification does not cause service |
| 327 |
interruption or system down time -- neither during the |
| 328 |
verification run nor during subsequent synchronization. |
| 329 |
|
| 330 |
It is a common use case to have on-line verification managed by the |
| 331 |
local cron daemon, running it, for example, once a week or once a |
| 332 |
month. See <<s-use-online-verify>> for information on how to enable, |
| 333 |
invoke, and automate on-line verification. |
| 334 |
|
| 335 |
[[s-integrity-check]] |
| 336 |
=== Replication traffic integrity checking |
| 337 |
|
| 338 |
DRBD optionally performs end-to-end message integrity checking using |
| 339 |
cryptographic message digest algorithms such as MD5, SHA-1, or CRC-32C. |
| 340 |
|
| 341 |
These message digest algorithms are *not* provided by DRBD, but by the Linux |
| 342 |
kernel crypto API; DRBD merely uses them. Thus, DRBD is |
| 343 |
capable of utilizing any message digest algorithm available in a |
| 344 |
particular system's kernel configuration. |
| 345 |
|
| 346 |
With this feature enabled, DRBD generates a message digest of every |
| 347 |
data block it replicates to the peer, which the peer then uses to |
| 348 |
verify the integrity of the replication packet. If the replicated |
| 349 |
block can not be verified against the digest, the connection is dropped |
| 350 |
and immediately re-established; because of the bitmap the typical result is a |
| 351 |
retransmission. Thus, DRBD replication is protected against several |
| 352 |
error sources, all of which, if unchecked, would potentially lead to |
| 353 |
data corruption during the replication process: |
| 354 |
|
| 355 |
* Bitwise errors ("bit flips") occurring on data in transit between |
| 356 |
main memory and the network interface on the sending node (which |
| 357 |
goes undetected by TCP checksumming if it is offloaded to the |
| 358 |
network card, as is common in recent implementations); |
| 359 |
|
| 360 |
* Bit flips occurring on data in transit from the network interface to |
| 361 |
main memory on the receiving node (the same considerations apply for |
| 362 |
TCP checksum offloading); |
| 363 |
|
| 364 |
* Any form of corruption due to a race conditions or bugs in network |
| 365 |
interface firmware or drivers; |
| 366 |
|
| 367 |
* Bit flips or random corruption injected by some reassembling network |
| 368 |
component between nodes (if not using direct, back-to-back |
| 369 |
connections). |
| 370 |
|
| 371 |
See <<s-configure-integrity-check>> for information on how to enable |
| 372 |
replication traffic integrity checking. |
| 373 |
|
| 374 |
[[s-split-brain-notification-and-recovery]] |
| 375 |
=== Split brain notification and automatic recovery |
| 376 |
|
| 377 |
Split brain is a situation where, due to temporary failure of all |
| 378 |
network links between cluster nodes, and possibly due to intervention |
| 379 |
by a cluster management software or human error, both nodes switched |
| 380 |
to the _Primary_ role while disconnected. This is a potentially harmful |
| 381 |
state, as it implies that modifications to the data might have been |
| 382 |
made on either node, without having been replicated to the peer. Thus, |
| 383 |
it is likely in this situation that two diverging sets of data have |
| 384 |
been created, which cannot be trivially merged. |
| 385 |
|
| 386 |
DRBD split brain is distinct from cluster split brain, which is the |
| 387 |
loss of all connectivity between hosts managed by a distributed |
| 388 |
cluster management application such as Pacemaker. To avoid confusion, |
| 389 |
this guide uses the following convention: |
| 390 |
|
| 391 |
* _Split brain_ refers to DRBD split brain as described in the |
| 392 |
paragraph above. |
| 393 |
|
| 394 |
* Loss of all cluster connectivity is referred to as a _cluster |
| 395 |
partition_, an alternative term for cluster split brain. |
| 396 |
|
| 397 |
DRBD allows for automatic operator notification (by email or other |
| 398 |
means) when it detects split brain. See <<s-split-brain-notification>> |
| 399 |
for details on how to configure this feature. |
| 400 |
|
| 401 |
While the recommended course of action in this scenario is to |
| 402 |
<<s-resolve-split-brain,manually resolve>> the split brain and then |
| 403 |
eliminate its root cause, it may be desirable, in some cases, to |
| 404 |
automate the process. DRBD has several resolution algorithms available |
| 405 |
for doing so: |
| 406 |
|
| 407 |
* *Discarding modifications made on the younger primary.* In this |
| 408 |
mode, when the network connection is re-established and split brain |
| 409 |
is discovered, DRBD will discard modifications made, in the |
| 410 |
meantime, on the node which switched to the primary role _last_. |
| 411 |
|
| 412 |
* *Discarding modifications made on the older primary.* In this mode, |
| 413 |
DRBD will discard modifications made, in the meantime, on the node |
| 414 |
which switched to the primary role _first_. |
| 415 |
|
| 416 |
* *Discarding modifications on the primary with fewer changes.* In |
| 417 |
this mode, DRBD will check which of the two nodes has recorded fewer |
| 418 |
modifications, and will then discard _all_ modifications made on |
| 419 |
that host. |
| 420 |
|
| 421 |
* *Graceful recovery from split brain if one host has had no |
| 422 |
intermediate changes.* In this mode, if one of the hosts has made no |
| 423 |
modifications at all during split brain, DRBD will simply recover |
| 424 |
gracefully and declare the split brain resolved. Note that this is a |
| 425 |
fairly unlikely scenario. Even if both hosts only mounted the file |
| 426 |
system on the DRBD block device (even read-only), the device |
| 427 |
contents typically would be modified (e.g. by filesystem journal |
| 428 |
replay), ruling out the possibility of automatic |
| 429 |
recovery. |
| 430 |
|
| 431 |
Whether or not automatic split brain recovery is acceptable depends |
| 432 |
largely on the individual application. Consider the example of DRBD |
| 433 |
hosting a database. The discard modifications from host with fewer |
| 434 |
changes approach may be fine for a web application click-through |
| 435 |
database. By contrast, it may be totally unacceptable to automatically |
| 436 |
discard _any_ modifications made to a financial database, requiring |
| 437 |
manual recovery in any split brain event. Consider your application's |
| 438 |
requirements carefully before enabling automatic split brain recovery. |
| 439 |
|
| 440 |
Refer to <<s-automatic-split-brain-recovery-configuration>> for |
| 441 |
details on configuring DRBD's automatic split brain recovery policies. |
| 442 |
|
| 443 |
[[s-disk-flush-support]] |
| 444 |
=== Support for disk flushes |
| 445 |
|
| 446 |
When local block devices such as hard drives or RAID logical disks |
| 447 |
have write caching enabled, writes to these devices are considered |
| 448 |
completed as soon as they have reached the volatile cache. Controller |
| 449 |
manufacturers typically refer to this as write-back mode, the opposite |
| 450 |
being write-through. If a power outage occurs on a controller in |
| 451 |
write-back mode, the last writes are never |
| 452 |
committed to the disk, potentially causing data loss. |
| 453 |
|
| 454 |
To counteract this, DRBD makes use of disk flushes. A disk flush is a |
| 455 |
write operation that completes only when the associated data has been |
| 456 |
committed to stable (non-volatile) storage -- that is to say, it has |
| 457 |
effectively been written to disk, rather than to the cache. DRBD uses |
| 458 |
disk flushes for write operations both to its replicated data set and |
| 459 |
to its meta data. In effect, DRBD circumvents the write cache in |
| 460 |
situations it deems necessary, as in <<s-activity-log,activity log>> |
| 461 |
updates or enforcement of implicit write-after-write |
| 462 |
dependencies. This means additional reliability even in the face of |
| 463 |
power failure. |
| 464 |
|
| 465 |
It is important to understand that DRBD can use disk flushes only when |
| 466 |
layered on top of backing devices that support them. Most reasonably |
| 467 |
recent kernels support disk flushes for most SCSI and SATA |
| 468 |
devices. Linux software RAID (md) supports disk flushes for RAID-1 |
| 469 |
provided that all component devices support them too. The same is true for |
| 470 |
device-mapper devices (LVM2, dm-raid, multipath). |
| 471 |
|
| 472 |
Controllers with battery-backed write cache (BBWC) use a battery to |
| 473 |
back up their volatile storage. On such devices, when power is |
| 474 |
restored after an outage, the controller flushes all pending writes out |
| 475 |
to disk from the battery-backed cache, ensuring that all |
| 476 |
writes committed to the volatile cache are actually transferred to |
| 477 |
stable storage. When running DRBD on top of such devices, it may be |
| 478 |
acceptable to disable disk flushes, thereby improving DRBD's write |
| 479 |
performance. See <<s-disable-flushes>> for details. |
| 480 |
|
| 481 |
[[s-trim-discard]] |
| 482 |
=== Trim/Discard support |
| 483 |
|
| 484 |
_Trim_/_Discard_ are two names for the same feature: a request to a storage |
| 485 |
system, telling it that some data range is not being used anymorefootnote:[For |
| 486 |
example, a deleted file's data.] and can get recycled. + |
| 487 |
This call originates in Flash-based storages (SSDs, FusionIO cards, etc.), |
| 488 |
which cannot easily _rewrite_ a sector but instead have to _erase_ and write |
| 489 |
the (new) data again (incurring some latency cost). For more details, see e.g. |
| 490 |
the https://en.wikipedia.org/wiki/Trim_%28computing%29[wikipedia page]. |
| 491 |
|
| 492 |
|
| 493 |
Since 8.4.3 DRBD includes support for _Trim_/_Discard_. You don't need to configure |
| 494 |
or enable anything; if DRBD detects that the local (underlying) storage system |
| 495 |
allows using these commands, it will transparently enable them and pass such |
| 496 |
requests through. |
| 497 |
|
| 498 |
The effect is that e.g. a recent-enough `mkfs.ext4` on a multi-TB volume can |
| 499 |
shorten the initial sync time to a few seconds to minutes - just by telling |
| 500 |
DRBD (which will relay that information to all connected nodes) that most/all |
| 501 |
of the storage is now to be seen as invalidated. |
| 502 |
|
| 503 |
Nodes that connect to that resource later on will not have seen the |
| 504 |
_Trim_/_Discard_ requests, and will therefore start a full resync; depending on |
| 505 |
kernel version and file system a call to `fstrim` might give the wanted result, |
| 506 |
though. |
| 507 |
|
| 508 |
NOTE: even if you don't have storage with _Trim_/_Discard_ support, some |
| 509 |
virtual block devices will provide you with the same feature, for example |
| 510 |
Thin LVM. |
| 511 |
|
| 512 |
|
| 513 |
[[s-handling-disk-errors]] |
| 514 |
=== Disk error handling strategies |
| 515 |
|
| 516 |
If a hard drive fails which is used as a backing block device for DRBD on one |
| 517 |
of the nodes, DRBD may either pass on the I/O error to the upper |
| 518 |
layer (usually the file system) or it can mask I/O errors from upper |
| 519 |
layers. |
| 520 |
|
| 521 |
[[fp-io-error-pass-on]] |
| 522 |
.Passing on I/O errors |
| 523 |
If DRBD is configured to pass on I/O errors, any such errors occurring |
| 524 |
on the lower-level device are transparently passed to upper I/O |
| 525 |
layers. Thus, it is left to upper layers to deal with such errors |
| 526 |
(this may result in a file system being remounted read-only, for |
| 527 |
example). This strategy does not ensure service continuity, and is |
| 528 |
hence not recommended for most users. |
| 529 |
|
| 530 |
[[fp-io-error-detach]] |
| 531 |
.Masking I/O errors |
| 532 |
If DRBD is configured to _detach_ on lower-level I/O error, DRBD will |
| 533 |
do so, automatically, upon occurrence of the first lower-level I/O |
| 534 |
error. The I/O error is masked from upper layers while DRBD |
| 535 |
transparently fetches the affected block from a peer node, over the |
| 536 |
network. From then onwards, DRBD is said to operate in diskless mode, |
| 537 |
and carries out all subsequent I/O operations, read and write, on the |
| 538 |
peer node(s) only. Performance in this mode will be reduced, |
| 539 |
but the service continues without interruption, and can be moved to |
| 540 |
the peer node in a deliberate fashion at a convenient time. |
| 541 |
|
| 542 |
See <<s-configure-io-error-behavior>> for information on configuring |
| 543 |
I/O error handling strategies for DRBD. |
| 544 |
|
| 545 |
[[s-outdate]] |
| 546 |
=== Strategies for dealing with outdated data |
| 547 |
|
| 548 |
DRBD distinguishes between _inconsistent_ and _outdated_ |
| 549 |
data. Inconsistent data is data that cannot be expected to be |
| 550 |
accessible and useful in any manner. The prime example for this is |
| 551 |
data on a node that is currently the target of an on-going |
| 552 |
synchronization. Data on such a node is part obsolete, part up to |
| 553 |
date, and impossible to identify as either. Thus, for example, if the |
| 554 |
device holds a filesystem (as is commonly the case), that filesystem |
| 555 |
would be unexpected to mount or even pass an automatic filesystem |
| 556 |
check. |
| 557 |
|
| 558 |
Outdated data, by contrast, is data on a secondary node that is |
| 559 |
consistent, but no longer in sync with the primary node. This would |
| 560 |
occur in any interruption of the replication link, whether temporary |
| 561 |
or permanent. Data on an outdated, disconnected secondary node is |
| 562 |
expected to be clean, but it reflects a state of the peer node some |
| 563 |
time past. In order to avoid services using outdated data, DRBD |
| 564 |
disallows <<s-resource-roles,promoting a resource>> that |
| 565 |
is in the outdated state. |
| 566 |
|
| 567 |
DRBD has interfaces that allow an external application to outdate a |
| 568 |
secondary node as soon as a network interruption occurs. DRBD will |
| 569 |
then refuse to switch the node to the primary role, preventing |
| 570 |
applications from using the outdated data. |
| 571 |
ifndef::drbd-only[] |
| 572 |
A complete implementation |
| 573 |
of this functionality exists for the <<ch-pacemaker,Pacemaker cluster |
| 574 |
management framework>> (where it uses a communication channel separate |
| 575 |
from the DRBD replication link). However, the interfaces are generic |
| 576 |
and may be easily used by any other cluster management application. |
| 577 |
endif::drbd-only[] |
| 578 |
|
| 579 |
Whenever an outdated resource has its replication link re-established, |
| 580 |
its outdated flag is automatically cleared. A <<s-resync,background |
| 581 |
synchronization>> then follows. |
| 582 |
|
| 583 |
[[s-three-way-repl]] |
| 584 |
=== Three-way replication via stacking |
| 585 |
|
| 586 |
NOTE: Available in DRBD version 8.3.0 and above; deprecated in DRBD version |
| 587 |
9.x, as more nodes can be implemented on a single level. See |
| 588 |
<<s-drbdconf-conns>> for details. |
| 589 |
|
| 590 |
When using three-way replication, DRBD adds a third node to an |
| 591 |
existing 2-node cluster and replicates data to that node, where it can |
| 592 |
be used for backup and disaster recovery purposes. |
| 593 |
ifndef::drbd-only,de-brand[] |
| 594 |
This type of configuration generally involves <<s-drbd-proxy>>. |
| 595 |
endif::[] |
| 596 |
|
| 597 |
Three-way replication works by adding another, _stacked_ DRBD resource |
| 598 |
on top of the existing resource holding your production data, as seen |
| 599 |
in this illustration: |
| 600 |
|
| 601 |
.DRBD resource stacking |
| 602 |
image::images/drbd-resource-stacking.svg[] |
| 603 |
|
| 604 |
The stacked resource is replicated using asynchronous replication |
| 605 |
(DRBD protocol A), whereas the production data would usually make use |
| 606 |
of synchronous replication (DRBD protocol C). |
| 607 |
|
| 608 |
Three-way replication can be used permanently, where the third node is |
| 609 |
continuously updated with data from the production |
| 610 |
cluster. Alternatively, it may also be employed on demand, where the |
| 611 |
production cluster is normally disconnected from the backup site, and |
| 612 |
site-to-site synchronization is performed on a regular basis, for |
| 613 |
example by running a nightly cron job. |
| 614 |
|
| 615 |
ifndef::drbd-only,de-brand[] |
| 616 |
[[s-drbd-proxy]] |
| 617 |
=== Long-distance replication via DRBD Proxy |
| 618 |
|
| 619 |
DRBD's <<s-replication-protocols,protocol A>> is asynchronous, but the |
| 620 |
writing application will block as soon as the socket output buffer is |
| 621 |
full (see the `sndbuf-size` option in the man page of `drbd.conf`). In that |
| 622 |
event, the writing application has to wait until some of the data written |
| 623 |
runs off through a possibly small bandwidth network link. |
| 624 |
|
| 625 |
The average write bandwidth is limited by available bandwidth of the |
| 626 |
network link. Write bursts can only be handled gracefully if they fit |
| 627 |
into the limited socket output buffer. |
| 628 |
|
| 629 |
You can mitigate this by DRBD Proxy's buffering mechanism. DRBD Proxy |
| 630 |
will place changed data from the DRBD device on the primary node into |
| 631 |
its buffers. DRBD Proxy's buffer size is freely configurable, only |
| 632 |
limited by the address room size and available physical RAM. |
| 633 |
|
| 634 |
Optionally DRBD Proxy can be configured to compress and decompress the |
| 635 |
data it forwards. Compression and decompression of DRBD's data packets |
| 636 |
might slightly increase latency. However, when the bandwidth of the network |
| 637 |
link is the limiting factor, the gain in shortening transmit time |
| 638 |
outweighs the compression and decompression overhead. |
| 639 |
|
| 640 |
Compression and decompression were implemented with multi core SMP |
| 641 |
systems in mind, and can utilize multiple CPU cores. |
| 642 |
|
| 643 |
The fact that most block I/O data compresses very well and therefore |
| 644 |
the effective bandwidth increases justifies the use of the DRBD |
| 645 |
Proxy even with DRBD protocols B and C. |
| 646 |
|
| 647 |
See <<s-using-drbd-proxy>> for information on configuring DRBD Proxy. |
| 648 |
|
| 649 |
NOTE: DRBD Proxy is one of the few parts of the DRBD product family that is |
| 650 |
not published under an open source license. Please contact |
| 651 |
[email protected] or [email protected] for an evaluation license. |
| 652 |
endif::[] |
| 653 |
|
| 654 |
[[s-truck-based-replication]] |
| 655 |
=== Truck based replication |
| 656 |
|
| 657 |
Truck based replication, also known as disk shipping, is a means of |
| 658 |
preseeding a remote site with data to be replicated, by physically |
| 659 |
shipping storage media to the remote site. This is particularly suited |
| 660 |
for situations where |
| 661 |
|
| 662 |
* the total amount of data to be replicated is fairly |
| 663 |
large (more than a few hundreds of gigabytes); |
| 664 |
|
| 665 |
* the expected rate of change of the data to be replicated is less |
| 666 |
than enormous; |
| 667 |
|
| 668 |
* the available network bandwidth between sites is |
| 669 |
limited. |
| 670 |
|
| 671 |
In such situations, without truck based replication, DRBD would |
| 672 |
require a very long initial device synchronization (on the order of |
| 673 |
weeks, months, or years). Truck based replication allows to ship a data seed |
| 674 |
to the remote site, and so drastically reduces the initial synchronization |
| 675 |
time. See <<s-using-truck-based-replication>> for details on this use |
| 676 |
case. |
| 677 |
|
| 678 |
[[s-floating-peers]] |
| 679 |
=== Floating peers |
| 680 |
|
| 681 |
NOTE: This feature is available in DRBD versions 8.3.2 and above. |
| 682 |
|
| 683 |
A somewhat special use case for DRBD is the _floating peers_ |
| 684 |
configuration. In floating peer setups, DRBD peers are not tied to |
| 685 |
specific named hosts (as in conventional configurations), but instead |
| 686 |
have the ability to float between several hosts. In such a |
| 687 |
configuration, DRBD identifies peers by IP address, rather than by |
| 688 |
host name. |
| 689 |
|
| 690 |
ifndef::drbd-only[] |
| 691 |
For more information about managing floating peer configurations, see |
| 692 |
<<s-pacemaker-floating-peers>>. |
| 693 |
endif::drbd-only[] |
| 694 |
|
| 695 |
[[s-rebalance]] |
| 696 |
=== Data rebalancing (horizontal storage scaling) |
| 697 |
|
| 698 |
If your company's policy says that 3-way redundancy is needed, you need |
| 699 |
at least 3 servers for your setup. |
| 700 |
|
| 701 |
Now, as your storage demands grow, you will encounter the need for |
| 702 |
additional servers. Rather than having to buy 3 more servers at the same |
| 703 |
time, you can _rebalance_ your data across a single additional node. |
| 704 |
|
| 705 |
.DRBD data rebalancing |
| 706 |
image::images/rebalance.svg[] |
| 707 |
|
| 708 |
//[scaledwidth="75"] |
| 709 |
|
| 710 |
In the figure above you can see the _before_ and _after_ states: from |
| 711 |
3 nodes with three 25TiB volumes each (for a net 75TiB), to 4 nodes, |
| 712 |
with net 100TiB. |
| 713 |
|
| 714 |
DRBD 9 makes it possible to do an online, live migration of the data; please see |
| 715 |
<<s-rebalance-workflow>> for the exact steps needed. |
| 716 |
|
| 717 |
[[s-drbd-client]] |
| 718 |
=== DRBD client |
| 719 |
|
| 720 |
With the multiple-peer feature of DRBD a number of interesting use-cases have |
| 721 |
been added, for example the _DRBD client_. |
| 722 |
|
| 723 |
The basic idea is that the DRBD _backend_ can consist of 3, 4, or more nodes |
| 724 |
(depending on the policy of required redundancy); but, as DRBD 9 can connect more |
| 725 |
nodes than that. DRBD works then as a storage access protocol in addition to |
| 726 |
storage replication. |
| 727 |
|
| 728 |
All write requests executed on a primary _DRBD client_ gets shipped to all |
| 729 |
nodes equipped with storage. Read requests are only shipped to one of the server |
| 730 |
nodes. The _DRBD client_ will evenly distribute the read requests among all |
| 731 |
available server nodes. |
| 732 |
|
| 733 |
See <<s-permanently-diskless-nodes>> for more information. |
| 734 |
|
| 735 |
[[s-feature-quorum]] |
| 736 |
=== Quorum |
| 737 |
|
| 738 |
In order to avoid split brain or diverging data of replicas one has to configure |
| 739 |
fencing. It turns out that in real world deployments node fencing is not popular |
| 740 |
because often mistakes happen in planning or deploying it. |
| 741 |
|
| 742 |
In the moment a data-set has 3 replicas one can rely on the quorum implementation |
| 743 |
within DRBD instead of cluster manager level fencing. |
| 744 |
ifndef::drbd-only[] |
| 745 |
Pacemaker gets informed about quorum or loss-of-quorum via the master score of |
| 746 |
the resource. |
| 747 |
endif::drbd-only[] |
| 748 |
|
| 749 |
DRBD's quorum can be used with any kind of Linux based service. In case the |
| 750 |
service terminates in the moment it is exposed to an IO-error the on quorum |
| 751 |
loss behavior is very elegant. |
| 752 |
In case the service does not terminate upon IO-error the systems needs to |
| 753 |
be configured to reboot a primary node that looses quorum. |
| 754 |
|
| 755 |
See <<s-configuring-quorum>> for more information. |
| 756 |
|
| 757 |
[[s-feature-quorum-tiebreaker]] |
| 758 |
==== Tiebreaker |
| 759 |
|
| 760 |
[NOTE] |
| 761 |
==================== |
| 762 |
The quorum tiebreaker feature is available in DRBD versions 9.0.18 and above. |
| 763 |
==================== |
| 764 |
|
| 765 |
The fundamental problem with two node clusters is that in the moment they lose |
| 766 |
connectivity we have two partitions and none of them has quorum, which results |
| 767 |
in the cluster halting the service. This problem can be mitigated by adding a |
| 768 |
third, diskless node to the cluster which will then act as a quorum tiebreaker. |
| 769 |
|
| 770 |
See <<s-configuring-quorum-tiebreaker>> for more information. |
| 771 |
|
| 772 |
ifndef::drbd-only[] |
| 773 |
[[s-feature-VCS]] |
| 774 |
=== DRBD integration for VCS |
| 775 |
|
| 776 |
Veritas Cluster Server (or Veritas Infoscale Availability) is a commercial alternative |
| 777 |
to the Pacemaker open source software. In case you need to integrate DRBD resources |
| 778 |
into a VCS setup please see the README in |
| 779 |
https://github.com/LINBIT/drbd-utils/tree/master/scripts/VCS[drbd-utils/scripts/VCS] |
| 780 |
on github. |
| 781 |
endif::drbd-only[] |