Debian Codegraph @

UG9/en/administration-manual.adoc

raw
1 ifdef::env-github[]
2 :tip-caption: :bulb:
3 :note-caption: :information_source:
4 :important-caption: :heavy_exclamation_mark:
5 :caution-caption: :fire:
6 :warning-caption: :warning:
7 endif::[]
8
9
10 [[ch-admin-manual]]
11 == Common administrative tasks
12
13 This chapter outlines typical administrative tasks encountered during
14 day-to-day operations. It does not cover troubleshooting tasks, these
15 are covered in detail in <<ch-troubleshooting>>.
16
17 include::configure.adoc[]
18
19 [[s-check-status]]
20 === Checking DRBD status
21
22 [[s-drbd-drbdmon]]
23 ==== Retrieving status with drbdmon
24
25 One convenient way to look at DRBD's status is the
26 `drbdmon` utility. It updates the state of DRBD resources in real time.
27
28 [[s-proc-drbd]]
29 ==== Status information in `/proc/drbd`
30
31 NOTE: `/proc/drbd` is deprecated. While it won't be removed in the 8.4
32 series, we recommend to switch to other means, like <<s-drbdadm-status>>; or,
33 for monitoring even more convenient, <<s-drbdsetup-events2>>.
34
35 indexterm:[/proc/drbd]`/proc/drbd` is a virtual file displaying
36 basic information about the DRBD module.
37 It was used extensively up to DRBD 8.4, but couldn't keep up
38 with the amount of information provided by DRBD 9.
39
40 ----------------------------
41 $ cat /proc/drbd
42 version: 9.0.0 (api:1/proto:86-110)
43 GIT-hash: XXX build by [email protected], 2011-10-12 09:07:35
44 ----------------------------
45
46 The first line, prefixed with `version:`, shows the DRBD version used
47 on your system. The second line contains information about this
48 specific build.
49
50
51 [[s-drbdadm-status]]
52 ==== Status information via `drbdadm`
53
54 indexterm:[drbdadm, status]In its simplest invocation, we just ask for the
55 status of a single resource.
56
57 ----------------------------
58 # drbdadm status home
59 home role:Secondary
60 disk:UpToDate
61 nina role:Secondary
62 disk:UpToDate
63 nino role:Secondary
64 disk:UpToDate
65 nono connection:Connecting
66 ----------------------------
67
68 This here just says that the resource _home_ is locally, on 'nina', and
69 on 'nino' _UpToDate_ and _Secondary_; so the three nodes have the same
70 data on their storage devices, and nobody is using the device currently.
71
72 The node 'nono' is not connected, its state is reported as
73 _Connecting_; please see <<s-connection-states>> below for more details.
74
75
76 You can get more information by passing the `--verbose` and/or
77 `--statistics` arguments to `drbdsetup` (lines broken for readability):
78
79 ----------------------------
80 # drbdsetup status home --verbose --statistics
81 home node-id:1 role:Secondary suspended:no
82 write-ordering:none
83 volume:0 minor:0 disk:UpToDate
84 size:1048412 read:0 written:1048412 al-writes:0 bm-writes:48 upper-pending:0
85 lower-pending:0 al-suspended:no blocked:no
86 nina local:ipv4:10.9.9.111:7001 peer:ipv4:10.9.9.103:7010 node-id:0
87 connection:Connected role:Secondary
88 congested:no
89 volume:0 replication:Connected disk:UpToDate resync-suspended:no
90 received:1048412 sent:0 out-of-sync:0 pending:0 unacked:0
91 nino local:ipv4:10.9.9.111:7021 peer:ipv4:10.9.9.129:7012 node-id:2
92 connection:Connected role:Secondary
93 congested:no
94 volume:0 replication:Connected disk:UpToDate resync-suspended:no
95 received:0 sent:0 out-of-sync:0 pending:0 unacked:0
96 nono local:ipv4:10.9.9.111:7013 peer:ipv4:10.9.9.138:7031 node-id:3
97 connection:Connecting
98 ----------------------------
99
100
101 Every few lines in this example form a block that is repeated
102 for every node used in this resource, with small format exceptions
103 for the local node - see below for more details.
104
105 The first line in each block shows the `node-id` (for the current
106 resource; a host can have different ``node-id``s in different resources).
107 Furthermore the `role` (see <<s-roles>>) is shown.
108
109 The next important line begins with the `volume` specification; normally
110 these are numbered starting by zero, but the configuration may specify
111 other IDs as well. This line shows the indexterm:[connection state]
112 connection state in the
113 `replication` item (see <<s-connection-states>> for details) and the
114 remote indexterm:[disk state] disk state in `disk` (see <<s-disk-states>>).
115 Then there's a line for this volume giving a bit of statistics -
116 data `received`, `sent`, `out-of-sync`, etc; please see
117 <<s-performance-indicators>> and <<s-conn-info>> for more information.
118
119 For the local node the first line shows the resource name, _home_, in our
120 example. As the first block always describes the local node, there is no _Connection_ or
121 address information.
122
123 please see the `drbd.conf` manual page for more information.
124
125 The other four lines in this example form a block that is repeated for
126 every DRBD device configured, prefixed by the device minor number. In
127 this case, this is `0`, corresponding to the device `/dev/drbd0`.
128
129 The resource-specific output contains various pieces
130 of information about the resource:
131
132 ///////
133 //FIXME
134 .Replication protocol
135 The Replication protocol used by the resource. Either `A`, `B`, or `C`. See
136 <<s-replication-protocols>> for details.
137 ///////
138
139 [[s-drbdsetup-events2]]
140 ==== One-shot or real time monitoring via `drbdsetup events2`
141
142 NOTE: This is available only with userspace versions 8.9.3 and up.
143 // and 8.4.6
144
145 This is a low-level mechanism to get information out of DRBD, suitable for use
146 in automated tools, like monitoring.
147
148 In its simplest invocation, showing only the current status, the output looks
149 like this (but, when running on a terminal, will include colors):
150
151 ---------------
152 # drbdsetup events2 --now r0
153 exists resource name:r0 role:Secondary suspended:no
154 exists connection name:r0 peer-node-id:1 conn-name:remote-host connection:Connected role:Secondary
155 exists device name:r0 volume:0 minor:7 disk:UpToDate
156 exists device name:r0 volume:1 minor:8 disk:UpToDate
157 exists peer-device name:r0 peer-node-id:1 conn-name:remote-host volume:0
158 replication:Established peer-disk:UpToDate resync-suspended:no
159 exists peer-device name:r0 peer-node-id:1 conn-name:remote-host volume:1
160 replication:Established peer-disk:UpToDate resync-suspended:no
161 exists -
162 ---------------
163
164 Without the ''--now'', the process will keep running, and send continuous updates like this:
165
166 -----------------
167 # drbdsetup events2 r0
168 ...
169 change connection name:r0 peer-node-id:1 conn-name:remote-host connection:StandAlone
170 change connection name:r0 peer-node-id:1 conn-name:remote-host connection:Unconnected
171 change connection name:r0 peer-node-id:1 conn-name:remote-host connection:Connecting
172 -----------------
173
174 Then, for monitoring purposes, there's another argument ''--statistics'', that
175 will produce some performance counters and other facts:
176
177 'drbdsetup' **verbose output** (lines broken for readability):
178
179 -----------------
180 # drbdsetup events2 --statistics --now r0
181 exists resource name:r0 role:Secondary suspended:no write-ordering:drain
182 exists connection name:r0 peer-node-id:1 conn-name:remote-host connection:Connected
183 role:Secondary congested:no
184 exists device name:r0 volume:0 minor:7 disk:UpToDate size:6291228 read:6397188
185 written:131844 al-writes:34 bm-writes:0 upper-pending:0 lower-pending:0
186 al-suspended:no blocked:no
187 exists device name:r0 volume:1 minor:8 disk:UpToDate size:104854364 read:5910680
188 written:6634548 al-writes:417 bm-writes:0 upper-pending:0 lower-pending:0
189 al-suspended:no blocked:no
190 exists peer-device name:r0 peer-node-id:1 conn-name:remote-host volume:0
191 replication:Established peer-disk:UpToDate resync-suspended:no received:0
192 sent:131844 out-of-sync:0 pending:0 unacked:0
193 exists peer-device name:r0 peer-node-id:1 conn-name:remote-host volume:1
194 replication:Established peer-disk:UpToDate resync-suspended:no received:0
195 sent:6634548 out-of-sync:0 pending:0 unacked:0
196 exists -
197 -----------------
198
199 You might also like the ''--timestamp'' parameter.
200
201
202 [[s-connection-states]]
203 ==== Connection states
204
205 indexterm:[drbdadm, cstate]indexterm:[connection state, via drbdadm]A resource's connection state can be
206 observed either by issuing the `drbdadm cstate` command:
207
208 ----------------------------
209 # drbdadm cstate <resource>
210 Connected
211 Connected
212 StandAlone
213 ----------------------------
214
215 If you are interested in only a single connection of a resource, specify the connection name, too:
216
217 The default is the peer's hostname as given in the configuration file.
218
219 ----------------------------
220 # drbdadm cstate <peer>:<resource>
221 Connected
222 ----------------------------
223
224 A resource may have one of the following connection states:
225
226 ._StandAlone_
227 indexterm:[connection state, StandAlone]No network configuration available. The
228 resource has not yet been connected, or has been administratively
229 disconnected (using `drbdadm disconnect`), or has dropped its
230 connection due to failed authentication or split brain.
231
232 ._Disconnecting_
233 indexterm:[connection state, Disconnecting]Temporary state during disconnection. The
234 next state is _StandAlone_.
235
236 ._Unconnected_
237 indexterm:[connection state, Unconnected]Temporary state, prior to a connection
238 attempt. Possible next states: _Connecting_.
239
240 ._Timeout_
241 indexterm:[connection state, Timeout]Temporary state following a timeout in the
242 communication with the peer. Next state: _Unconnected_.
243
244 ._BrokenPipe_
245 indexterm:[connection state, BrokenPipe]Temporary state after the connection to
246 the peer was lost. Next state: _Unconnected_.
247
248 ._NetworkFailure_
249 indexterm:[connection state, NetworkFailure]Temporary state after the connection to
250 the partner was lost. Next state: _Unconnected_.
251
252 ._ProtocolError_
253 indexterm:[connection state, ProtocolError]Temporary state after the connection to
254 the partner was lost. Next state: _Unconnected_.
255
256 ._TearDown_
257 indexterm:[connection state, TearDown]Temporary state. The peer is closing the
258 connection. Next state: _Unconnected_.
259
260 ._Connecting_
261 indexterm:[connection state, Connecting]This node is waiting until the peer node
262 becomes visible on the network.
263
264 ._Connected_
265 indexterm:[connection state, Connected]A DRBD connection has been established,
266 data mirroring is now active. This is the normal state.
267
268 [[s-replication-states]]
269 ==== Replication states
270
271 Each volume has over each connection a replication state. The possible replication
272 states are:
273
274 ._Off_
275 indexterm:[replication state, Off]The volume is not replicated over this
276 connection, since the connection is not _Connected_.
277
278 ._Established_
279 indexterm:[replication state, Established]All writes to that volume are replicated
280 online. This is the normal state.
281
282 ._StartingSyncS_
283 indexterm:[replication state, StartingSyncS]Full synchronization, initiated by the
284 administrator, is just starting. The next possible states are:
285 _SyncSource_ or _PausedSyncS_.
286
287 ._StartingSyncT_
288 indexterm:[replication state, StartingSyncT]Full synchronization, initiated by the
289 administrator, is just starting. Next state: _WFSyncUUID_.
290
291 ._WFBitMapS_
292 indexterm:[replication state, WFBitMapS]Partial synchronization is just
293 starting. Next possible states: _SyncSource_ or _PausedSyncS_.
294
295 ._WFBitMapT_
296 indexterm:[replication state, WFBitMapT]Partial synchronization is just
297 starting. Next possible state: _WFSyncUUID_.
298
299 ._WFSyncUUID_
300 indexterm:[replication state, WFSyncUUID]Synchronization is about to begin. Next
301 possible states: _SyncTarget_ or _PausedSyncT_.
302
303 ._SyncSource_
304 indexterm:[replication state, SyncSource]Synchronization is currently running, with
305 the local node being the source of synchronization.
306
307 ._SyncTarget_
308 indexterm:[replication state, SyncTarget]Synchronization is currently running, with
309 the local node being the target of synchronization.
310
311 ._PausedSyncS_
312 indexterm:[replication state, PausedSyncS]The local node is the source of an ongoing
313 synchronization, but synchronization is currently paused. This may be
314 due to a dependency on the completion of another synchronization
315 process, or due to synchronization having been manually interrupted by
316 `drbdadm pause-sync`.
317
318 ._PausedSyncT_
319 indexterm:[replication state, PausedSyncT]The local node is the target of an ongoing
320 synchronization, but synchronization is currently paused. This may be
321 due to a dependency on the completion of another synchronization
322 process, or due to synchronization having been manually interrupted by
323 `drbdadm pause-sync`.
324
325 ._VerifyS_
326 indexterm:[replication state, VerifyS]On-line device verification is currently
327 running, with the local node being the source of verification.
328
329 ._VerifyT_
330 indexterm:[replication state, VerifyT]On-line device verification is currently
331 running, with the local node being the target of verification.
332
333 ._Ahead_
334 indexterm:[replication state, Ahead]Data replication was suspended, since the
335 link can not cope with the load. This state is enabled by the configuration
336 `on-congestion` option (see <<s-configure-congestion-policy>>).
337
338 ._Behind_
339 indexterm:[replication state, Behind]Data replication was suspended by the peer,
340 since the link can not cope with the load. This state is enabled by the configuration
341 `on-congestion` option on the peer node (see <<s-configure-congestion-policy>>).
342
343 [[s-roles]]
344 ==== Resource roles
345
346 indexterm:[resource]A resource's role can be observed by issuing the
347 indexterm:[drbdadm, role]`drbdadm role` command:
348
349 ----------------------------
350 # drbdadm role <resource>
351 Primary
352 ----------------------------
353
354 You may see one of the following resource roles:
355
356 ._Primary_
357 indexterm:[role, Primary]The resource is currently in the primary role, and may be read from
358 and written to. This role only occurs on one of the two nodes, unless
359 <<s-dual-primary-mode,dual-primary mode>> is enabled.
360
361 ._Secondary_
362 indexterm:[role, Secondary]The resource is currently in the secondary role. It normally receives
363 updates from its peer (unless running in disconnected mode), but may
364 neither be read from nor written to. This role may occur on one
365 or both nodes.
366
367 ._Unknown_
368 indexterm:[role, Unknown]The resource's role is currently unknown. The local resource role
369 never has this status. It is only displayed for the peer's resource
370 role, and only in disconnected mode.
371
372
373 [[s-disk-states]]
374 ==== Disk states
375
376 A resource's disk state can be observed either by
377 issuing the `drbdadm dstate` command:
378
379 ----------------------------
380 # drbdadm dstate <resource>
381 UpToDate
382 ----------------------------
383
384 The disk state may be one of the following:
385
386 ._Diskless_
387 indexterm:[disk state, Diskless]No local block device has been assigned to the
388 DRBD driver. This may mean that the resource has never attached to its
389 backing device, that it has been manually detached using `drbdadm
390 detach`, or that it automatically detached after a lower-level I/O
391 error.
392
393 ._Attaching_
394 indexterm:[disk state, Attaching]Transient state while reading meta data.
395
396 ._Detaching_
397 indexterm:[disk state, Detaching]Transient state while detaching and waiting
398 for ongoing IOs to complete.
399
400 ._Failed_
401 indexterm:[disk state, Failed]Transient state following an I/O failure report
402 by the local block device. Next state: _Diskless_.
403
404 ._Negotiating_
405 indexterm:[disk state, Negotiating]Transient state when an _Attach_ is carried out on
406 an already-__Connected__ DRBD device.
407
408 ._Inconsistent_
409 indexterm:[disk state, Inconsistent]The data is inconsistent. This status occurs
410 immediately upon creation of a new resource, on both nodes (before the
411 initial full sync). Also, this status is found in one node (the
412 synchronization target) during synchronization.
413
414 ._Outdated_
415 indexterm:[disk state, Outdated]Resource data is consistent, but
416 <<s-outdate,outdated>>.
417
418 ._DUnknown_
419 indexterm:[disk state, DUnknown]This state is used for the peer disk if no
420 network connection is available.
421
422 ._Consistent_
423 indexterm:[disk state, Consistent]Consistent data of a node without
424 connection. When the connection is established, it is decided whether
425 the data is _UpToDate_ or _Outdated_.
426
427 ._UpToDate_
428 indexterm:[disk state, UpToDate]Consistent, up-to-date state of the data. This
429 is the normal state.
430
431
432 [[s-conn-info]]
433 ==== Connection information data
434
435 ._local_
436 Shows the network family, the local address and port that is used to
437 accept connections from the peer.
438
439 ._peer_
440 Shows the network family, the peer address and port that is used
441 to connect.
442
443 ._congested_
444 This flag tells whether the TCP send buffer of the data connection is more than 80% filled.
445
446
447 [[s-performance-indicators]]
448 ==== Performance indicators
449
450 indexterm:[performance indicators]
451 The command ``drbdsetup status --verbose --statistics`` can be used to show performance statistics.
452 These are also available in ``drbdsetup events2 --statistics``,
453 although there will not be a _changed_ event for every change.
454 The statistics include the following counters and gauges:
455
456 Per volume/device:
457
458 ._read_ (disk read)
459 Net data read from local disk; in KiB.
460
461 ._written_ (disk written)
462 Net data written to local disk; in KiB.
463
464 ._al-writes_ (activity log)
465 Number of updates of the activity log area of the meta data.
466
467 ._bm-writes_ (bitmap)
468 Number of updates of the bitmap area of the meta data.
469
470 ._upper-pending_ (application pending)
471 Number of block I/O requests forwarded to DRBD, but not yet answered (completed)
472 by DRBD.
473
474 ._lower-pending_ (local count)
475 Number of open requests to the local I/O sub-system issued by DRBD.
476
477 ._blocked_
478 indexterm:[blocked]Shows local I/O congestion.
479 --
480
481 * _no_: No congestion.
482 * _upper_: I/O _above_ the DRBD device is blocked, i.e. to the filesystem. Typical causes are
483 ** I/O suspension by the administrator, see the `suspend-io` command in `drbdadm`.
484 ** transient blocks, e.g. during attach/detach
485 ** buffers depleted, see <<p-performance>>
486 ** Waiting for bitmap IO
487 * _lower_: Backing device is congested.
488 * _upper,lower_: Both _upper_ and _lower_ are blocked.
489 --
490
491 Per connection:
492
493 ._ap-in-flight_ (application in-flight)
494 Application data that is being written by the peer.
495 That is, DRBD has sent it to the peer and is waiting for the acknowledgement that it has been written.
496 In sectors (512 bytes).
497
498 ._rs-in-flight_ (resync in-flight)
499 Resync data that is being written by the peer.
500 That is, DRBD is _SyncSource_, has sent data to the peer as part of a resync
501 and is waiting for the acknowledgement that it has been written.
502 In sectors (512 bytes).
503
504 Per connection and volume ("peer device"):
505
506 ._received_ (network receive)
507 Net data received from the peer; in KiB.
508
509 ._sent_ (network send)
510 Net data sent to the peer; in KiB.
511
512 ._out-of-sync_
513 indexterm:[out-of-sync]indexterm:[oos]Amount of data currently out of sync
514 with this peer, according to the bitmap that DRBD has for it; in KiB.
515
516 ._pending_
517 Number of requests sent to the peer, but that have not yet been
518 acknowledged by the peer.
519
520 ._unacked_ (unacknowledged)
521 Number of requests received from the peer,
522 but that have not yet been acknowledged by DRBD on this node.
523
524 ._resync-suspended_
525 Whether the resynchronization is currently suspended or not. Possible values are _no_, _user_, _peer_, _dependency_.
526 Comma separated.
527
528
529 [[s-enable-disable]]
530 === Enabling and disabling resources
531
532 [[s-enable-resource]]
533 ==== Enabling resources
534
535 indexterm:[resource]Normally, all configured DRBD resources are
536 automatically enabled
537
538 * by a cluster resource management application at its discretion,
539 based on your cluster configuration, or
540
541 * by the systemd units (e.g., `[email protected]`)
542
543 If, however, you need to enable resources manually for any reason, you
544 may do so by issuing the command
545
546 ----------------------------
547 # drbdadm up <resource>
548 ----------------------------
549
550 As always, you may use the keyword `all` instead of a specific
551 resource name if you want to enable all resources configured in
552 `/etc/drbd.conf` at once.
553
554 [[s-disable-resource]]
555 ==== Disabling resources
556
557 indexterm:[resource]You may temporarily disable specific resources by
558 issuing the command
559
560 ----------------------------
561 # drbdadm down <resource>
562 ----------------------------
563
564 Here, too, you may use the keyword `all` in place of a resource name if
565 you wish to temporarily disable all resources listed in
566 `/etc/drbd.conf` at once.
567
568 [[s-reconfigure]]
569 === Reconfiguring resources
570
571 indexterm:[resource]DRBD allows you to reconfigure resources while
572 they are operational. To that end,
573
574 * make any necessary changes to the resource configuration in
575 `/etc/drbd.conf`,
576
577 * synchronize your `/etc/drbd.conf` file between both nodes,
578
579 * issue the indexterm:[drbdadm, adjust]`drbdadm adjust <resource>` command on
580 both nodes.
581
582 `drbdadm adjust` then hands off to `drbdsetup` to make the necessary
583 adjustments to the configuration. As always, you are able to review
584 the pending `drbdsetup` invocations by running `drbdadm` with the
585 `-d` (dry-run) option.
586
587 NOTE: When making changes to the `common` section in `/etc/drbd.conf`,
588 you can adjust the configuration for all resources in one run, by
589 issuing `drbdadm adjust all`.
590
591 [[s-switch-resource-roles]]
592 === Promoting and demoting resources
593
594 indexterm:[resource]Manually switching a <<s-resource-roles,resource's
595 role>> from secondary to primary (promotion) or vice versa (demotion)
596 is done using one of the following commands:
597
598 ----------------------------
599 # drbdadm primary <resource>
600 # drbdadm secondary <resource>
601 ----------------------------
602
603 In <<s-single-primary-mode,single-primary mode>> (DRBD's default), any
604 resource can be in the primary role on only one node at any given time
605 while the <<s-connection-states,connection state>> is
606 _Connected_. Thus, issuing `drbdadm primary <resource>` on one node
607 while the specified resource is still in the primary role on another node will
608 result in an error.
609
610 A resource configured to allow <<s-dual-primary-mode,dual-primary
611 mode>> can be switched to the primary role on two nodes; this is e.g. needed for
612 online migration of virtual machines.
613
614
615 [[s-manual-fail-over]]
616 === Basic manual fail-over
617
618 If not using a cluster manager and looking to handle fail-overs manually in a
619 passive/active configuration, the process is as follows.
620
621 On the current primary node stop any applications or services using the DRBD device,
622 unmount the DRBD device, and demote the resource to secondary.
623
624 ----------------------------
625 # umount /dev/drbd/by-res/<resource>/<vol-nr>
626 # drbdadm secondary <resource>
627 ----------------------------
628
629 Now on the node we wish to make primary promote the resource and mount the device.
630
631 ----------------------------
632 # drbdadm primary <resource>
633 # mount /dev/drbd/by-res/<resource>/<vol-nr> <mountpoint>
634 ----------------------------
635
636 If you're using the `auto-promote` feature, you don't need to change the roles
637 (_Primary_/_Secondary_) manually; only stopping of the services and unmounting,
638 respectively mounting, is necessary.
639
640 [[s-upgrading-drbd]]
641 include::upgrading.adoc[]
642
643 [[s-enable-dual-primary]]
644 include::dualprimary.adoc[]
645
646 [[s-use-online-verify]]
647 === Using on-line device verification
648
649 [[s-online-verify-enable]]
650 ==== Enabling on-line verification
651
652 indexterm:[on-line device verification]<<s-online-verify,On-line
653 device verification>> for resources is not enabled by default. To
654 enable it, add the following lines to your resource configuration in
655 `/etc/drbd.conf`:
656
657 [source,drbd]
658 ----------------------------
659 resource <resource>
660 net {
661 verify-alg <algorithm>;
662 }
663 ...
664 }
665 ----------------------------
666
667 _<algorithm>_ may be any message digest algorithm supported by the
668 kernel crypto API in your system's kernel configuration. Normally, you
669 should be able to choose at least from `sha1`, `md5`, and `crc32c`.
670
671 If you make this change to an existing resource, as always,
672 synchronize your `drbd.conf` to the peer, and run `drbdadm adjust
673 _<resource>_` on both nodes.
674
675 [[s-online-verify-invoke]]
676 ==== Invoking on-line verification
677
678 indexterm:[on-line device verification]After you have enabled on-line
679 verification, you will be able to initiate a verification run using
680 the following command:
681
682 ----------------------------
683 # drbdadm verify <resource>:<peer>/<volume>
684 ----------------------------
685
686 When you do so, DRBD starts an online verification run for
687 _<volume>_ to _<peer_> in _<resource>_, and if it detects any blocks
688 that are not in sync, will mark those blocks as such and write a
689 message to the kernel log. Any applications using the device at that
690 time can continue to do so unimpeded, and you may also
691 <<s-switch-resource-roles,switch resource roles>> at will.
692
693 _<Volume>_ is optional, if ommited, it will verify all volumes
694 in that resource.
695
696 If out-of-sync blocks were detected during the verification run, you
697 may resynchronize them using the following commands after verification
698 has completed.
699
700 Since drbd-9.0.29 the preferred way is one of these commands:
701
702 ----------------------------
703 # drbdadm invalidate <resource>:<peer>/volume --reset-bitmap=no
704 # drbdadm invalidate-remote <resource>:<peer>/volume --reset-bitmap=no
705 ----------------------------
706
707 The first command will cause the local differences to be overwritten
708 by the remote version. The second command does it in the opposite direction.
709
710 Before drbd-9.0.29 one needs to initiate a resync. A way to do that
711 is disconnecting from a primary and ensuring that the primary changes
712 at least one block while the peer is away.
713
714 ----------------------------
715 # drbdadm disconnect <resource>:<peer>
716 ## write one block on the primary
717 # drbdadm connect <resource>:<peer>
718 ----------------------------
719
720
721 [[s-online-verify-automate]]
722 ==== Automating on-line verification
723
724 indexterm:[on-line device verification]Most users will want to
725 automate on-line device verification. This can be easily
726 accomplished. Create a file with the following contents, named
727 `/etc/cron.d/drbd-verify` on *one* of your nodes:
728
729 [source,drbd]
730 ----------------------------
731 42 0 * * 0 root /sbin/drbdadm verify <resource>
732 ----------------------------
733
734 This will have `cron` invoke a device verification every Sunday at 42
735 minutes past midnight; so, if you come into the office on Monday morning,
736 a quick look at the resource's status would show the result. If your devices
737 are very big, and the \~32 hours were not enough, then you'll notice _VerifyS_ or
738 _VerifyT_ as connection state, meaning that the `verify` is still in progress.
739
740 If you have enabled on-line verification for all your resources (for
741 example, by adding `verify-alg _<algorithm>_` to the `common` section
742 in `/etc/drbd.d/global_common.conf`), you may also use:
743
744 [source,drbd]
745 ----------------------------
746 42 0 * * 0 root /sbin/drbdadm verify all
747 ----------------------------
748
749
750 [[s-configure-sync-rate]]
751 === Configuring the rate of synchronization
752
753 indexterm:[synchronization]Normally, one tries to ensure that
754 background synchronization (which makes the data on the
755 synchronization target temporarily inconsistent) completes as quickly
756 as possible. However, it is also necessary to keep background
757 synchronization from hogging all bandwidth otherwise available for
758 foreground replication, which would be detrimental to application
759 performance. Thus, you must configure the synchronization bandwidth to
760 match your hardware -- which you may do in a permanent fashion or
761 on-the-fly.
762
763 IMPORTANT: It does not make sense to set a synchronization rate that
764 is higher than the maximum write throughput on your secondary
765 node. You must not expect your secondary node to miraculously be able
766 to write faster than its I/O subsystem allows, just because it happens
767 to be the target of an ongoing device synchronization.
768
769 Likewise, and for the same reasons, it does not make sense to set a
770 synchronization rate that is higher than the bandwidth available on
771 the replication network.
772
773
774 [[s-estimating_a_synchronization_speed]]
775 ==== Estimating a synchronization speed
776
777 TIP: A good rule of thumb for this value is to use about 30% of the
778 available replication bandwidth. Thus, if you had an I/O subsystem
779 capable of sustaining write throughput of 400MB/s, and a Gigabit
780 Ethernet network capable of sustaining 110 MB/s network throughput
781 (the network being the bottleneck), you would calculate:
782
783 [[eq-sync-rate-example1]]
784 .Syncer rate example, 110MB/s effective available bandwidth
785 image::images/sync-rate-example1.svg[]
786
787 Thus, the recommended value for the `rate` option would be `33M`.
788
789 By contrast, if you had an I/O subsystem with a maximum throughput of
790 80MB/s and a Gigabit Ethernet connection (the I/O subsystem being the
791 bottleneck), you would calculate:
792
793 [[eq-sync-rate-example2]]
794 .Syncer rate example, 80MB/s effective available bandwidth
795 image::images/sync-rate-example2.svg[]
796
797 In this case, the recommended value for the `rate` option would be
798 `24M`.
799
800 Similarly, for a storage speed of 800MB/s and a 10Gbe network connection, you
801 would shoot for \~240MB/s synchronization rate.
802
803 [[s-configure-sync-rate-variable]]
804 ==== Variable sync rate configuration
805
806 When multiple DRBD resources share a
807 single replication/synchronization network, synchronization with a fixed rate
808 may not be an optimal approach. So, in DRBD 8.4.0 the
809 variable-rate synchronization was enabled by default. In this mode, DRBD uses an automated
810 control loop algorithm to determine, and adjust, the
811 synchronization rate. This algorithm ensures that there is always
812 sufficient bandwidth available for foreground replication, greatly
813 mitigating the impact that background synchronization has on
814 foreground I/O.
815
816 The optimal configuration for variable-rate synchronization may vary
817 greatly depending on the available network bandwidth, application I/O
818 pattern and link congestion.
819 ifndef::drbd-only,de-brand[]
820 Ideal configuration settings also depend
821 on whether <<s-drbd-proxy,DRBD Proxy>> is in use or not.
822 It may be
823 wise to engage professional consultancy in order to optimally
824 configure this DRBD feature. An _example_ configuration (which assumes
825 a deployment in conjunction with DRBD Proxy) is provided below:
826
827 [source,drbd]
828 ----------------------------
829 resource <resource> {
830 disk {
831 c-plan-ahead 5;
832 c-max-rate 10M;
833 c-fill-target 2M;
834 }
835 }
836 ----------------------------
837
838 endif::[]
839
840 TIP: A good starting value for `c-fill-target` is _BDP * 2_, where
841 _BDP_ is your bandwidth-delay-product on the replication link.
842
843 For example, when using a 1GBit/s crossover connection, you'll end up with
844 about 200µs latencyfootnote:[The rule-of-thumb is using the time reported by
845 `ping`.]. +
846 1GBit/s means about 120MB/s; times 200*10^-6^ seconds gives 24000 Byte. Just round that value up to the next MB, and you're good to go.
847
848 Another example: a 100MBit WAN connection with 200ms latency means 12MB/s times
849 0.2s, or about 2.5MB "on the wire". Here a good starting value for
850 `c-fill-target` would be 3MB.
851
852 Please see the `drbd.conf` manual page for more details on the other
853 configuration items.
854
855
856 [[s-configure-sync-rate-permanent]]
857 ==== Permanent fixed sync rate configuration
858
859 In a few, very restricted situationsfootnote:[Like benchmarking.], it might
860 make sense to just use some fixed synchronization rate. In this case, first
861 of all you need to turn the dynamic sync rate controller off, by using
862 `c-plan-ahead 0;`.
863
864 Then, the maximum bandwidth a resource uses for background
865 re-synchronization is determined by the `resync-rate` option
866 for a resource. This must be included in the resource's
867 `disk` section in `/etc/drbd.conf`:
868
869 [source,drbd]
870 ----------------------------
871 resource <resource>
872 disk {
873 resync-rate 40M;
874 ...
875 }
876 ...
877 }
878 ----------------------------
879
880 Note that the rate setting is given in _bytes_, not _bits_ per second; the
881 default unit is _Kibibyte_, so a value of `4096` would be interpreted as `4MiB`.
882
883 NOTE: This just defines a rate that DRBD tries to achieve. If there is a
884 bottleneck with lower throughput (network, storage speed), the defined speed
885 (aka the "wished-for" performance ;) won't be reached.
886
887
888 [[s-some_more_hints_about_synchronization]]
889 ==== Some more hints about synchronization
890
891 When some amount of the to-be-synchronized data isn't really in use anymore
892 (e.g. because files got deleted while one node wasn't connected), you
893 might benefit from the <<s-trim-discard>>.
894
895 Furthermore, `c-min-rate` is easy to misunderstand - it doesn't define
896 a *minimum* synchronization speed, but rather a limit below which DRBD will not
897 slow down further *on purpose*. +
898 Whether you manage to *reach* that synchronization rate depends on your network
899 and storage speed, network latency (which might be highly variable for shared
900 links), and application IO (which not might be able to do anything about).
901
902
903 [[s-configure-checksum-sync]]
904 === Configuring checksum-based synchronization
905
906 indexterm:[checksum-based
907 synchronization]<<p-checksum-sync,Checksum-based synchronization>> is
908 not enabled for resources by default. To enable it, add the following
909 lines to your resource configuration in `/etc/drbd.conf`:
910
911 [source,drbd]
912 ----------------------------
913 resource <resource>
914 net {
915 csums-alg <algorithm>;
916 }
917 ...
918 }
919 ----------------------------
920
921 _<algorithm>_ may be any message digest algorithm supported by the
922 kernel crypto API in your system's kernel configuration. Normally, you
923 should be able to choose at least from `sha1`, `md5`, and `crc32c`.
924
925 If you make this change to an existing resource, as always,
926 synchronize your `drbd.conf` to the peer, and run `drbdadm adjust
927 _<resource>_` on both nodes.
928
929 [[s-configure-congestion-policy]]
930 === Configuring congestion policies and suspended replication
931
932 In an environment where the replication bandwidth is highly variable
933 (as would be typical in WAN replication setups), the replication link
934 may occasionally become congested. In a default configuration, this
935 would cause I/O on the primary node to block, which is sometimes
936 undesirable.
937
938 Instead, you may configure DRBD to _suspend_ the ongoing replication
939 in this case, causing the Primary's data set to _pull ahead_ of the
940 Secondary. In this mode, DRBD keeps the replication channel open -- it
941 never switches to disconnected mode -- but does not actually replicate
942 until sufficient bandwidth becomes available again.
943
944 The following example is for a DRBD Proxy configuration:
945
946 [source,drbd]
947 ----------------------------
948 resource <resource> {
949 net {
950 on-congestion pull-ahead;
951 congestion-fill 2G;
952 congestion-extents 2000;
953 ...
954 }
955 ...
956 }
957 ----------------------------
958
959 It is usually wise to set both `congestion-fill` and
960 `congestion-extents` together with the `pull-ahead` option.
961
962 A good value for `congestion-fill` is 90%
963
964 * of the allocated DRBD proxy buffer memory, when replicating over
965 DRBD Proxy, or
966 * of the TCP network send buffer, in non-DRBD Proxy setups.
967
968 A good value for `congestion-extents` is 90% of your configured
969 `al-extents` for the affected resources.
970
971
972 [[s-configure-io-error-behavior]]
973 === Configuring I/O error handling strategies
974
975 indexterm:[I/O errors]indexterm:[drbd.conf]DRBD's
976 <<s-handling-disk-errors,strategy for handling lower-level I/O
977 errors>> is determined by the `on-io-error` option, included in the
978 resource `disk` configuration in `/etc/drbd.conf`:
979
980 [source,drbd]
981 ----------------------------
982 resource <resource> {
983 disk {
984 on-io-error <strategy>;
985 ...
986 }
987 ...
988 }
989 ----------------------------
990
991 You may, of course, set this in the `common` section too, if you want
992 to define a global I/O error handling policy for all resources.
993
994 _<strategy>_ may be one of the following options:
995
996 .`detach`
997 This is the default and recommended option. On the occurrence of a
998 lower-level I/O error, the node drops its backing device, and
999 continues in diskless mode.
1000
1001 .`pass-on`
1002 This causes DRBD to report the I/O error to the upper layers. On the
1003 primary node, it is reported to the mounted file system. On the
1004 secondary node, it is ignored (because the secondary has no upper
1005 layer to report to).
1006
1007 .`call-local-io-error`
1008 Invokes the command defined as the local I/O error handler. This
1009 requires that a corresponding `local-io-error` command invocation is
1010 defined in the resource's `handlers` section. It is entirely left to
1011 the administrator's discretion to implement I/O error handling using
1012 the command (or script) invoked by `local-io-error`.
1013
1014 NOTE: Early DRBD versions (prior to 8.0) included another option,
1015 `panic`, which would forcibly remove the node from the cluster by way
1016 of a kernel panic, whenever a local I/O error occurred. While that
1017 option is no longer available, the same behavior may be mimicked via
1018 the `local-io-error`/`call-local-io-error` interface. You should do so
1019 only if you fully understand the implications of such behavior.
1020
1021
1022 You may reconfigure a running resource's I/O error handling strategy
1023 by following this process:
1024
1025 * Edit the resource configuration in `/etc/drbd.d/__<resource>__.res`.
1026
1027 * Copy the configuration to the peer node.
1028
1029 * Issue `drbdadm adjust _<resource>_` on both nodes.
1030
1031
1032 [[s-configure-integrity-check]]
1033 === Configuring replication traffic integrity checking
1034
1035 indexterm:[replication traffic integrity
1036 checking]<<s-integrity-check,Replication traffic integrity checking>>
1037 is not enabled for resources by default. To enable it, add the
1038 following lines to your resource configuration in `/etc/drbd.conf`:
1039
1040 [source,drbd]
1041 ----------------------------
1042 resource <resource>
1043 net {
1044 data-integrity-alg <algorithm>;
1045 }
1046 ...
1047 }
1048 ----------------------------
1049
1050 _<algorithm>_ may be any message digest algorithm supported by the
1051 kernel crypto API in your system's kernel configuration. Normally, you
1052 should be able to choose at least from `sha1`, `md5`, and `crc32c`.
1053
1054 If you make this change to an existing resource, as always,
1055 synchronize your `drbd.conf` to the peer, and run `drbdadm adjust
1056 _<resource>_` on both nodes.
1057
1058 WARNING: This feature is not intended for production use. Enable only if you need to
1059 diagnose data corruption problems, and want to see whether the transport path
1060 (network hardware, drivers, switches) might be at fault!
1061
1062 [[s-resizing]]
1063 === Resizing resources
1064
1065
1066 When growing (extending) DRBD volumes, you need to grow from bottom to top.
1067 You need to extend the backing block devices on all nodes first.
1068 Then you can tell DRBD to use the new space.
1069
1070 Once the DRBD volume is extended, you need to still propagate that
1071 into whatever is using DRBD: extend the file system, or make a VM
1072 running with this volume attached aware of the new "disk size".
1073
1074 That all typically boils down to
1075
1076 -----------------------
1077 # # on all nodes, resize the backing LV:
1078 # lvextend -L +${additional_gb}g VG/LV
1079 # # on one node:
1080 # drbdadm resize ${resource_name}/${volume_number}
1081 # # on the Primary only:
1082 # # resize the file system using the file system specific tool, see below
1083 -----------------------
1084
1085 See also the next sections <<s-growing-online>> and following.
1086
1087 Note that different file systems have different capabilities and different sets
1088 of management tools. For example XFS can only grow.
1089 You point its tool to the active mount point: `xfs_growfs /where/you/have/it/mounted`.
1090
1091 While the EXT family can both grow (even online), and also shrink (only
1092 offline; you have to unmount it first). To resize an ext3 or ext4,
1093 you would point the tool not to the mount point, but to the (mounted)
1094 block device: `resize2fs /dev/drbd#`
1095
1096 Obviously use the correct DRBD (as displayed by `mount` or `df -T`, while mounted),
1097 and *not* the backing block device. As long as DRBD is up, that's not supposed
1098 to work anyways (`resize2fs: Device or resource busy while trying to open
1099 /dev/mapper/VG-LV Couldn't find valid filesystem superblock.`).
1100 If you tried to do that offline (with DRBD stopped), you may corrupt DRBD
1101 meta data if you ran the file system tools directly against the backing LV or
1102 partition. So don't.
1103
1104 You do the file system resize only *once* on the Primary, against the active
1105 DRBD device. DRBD replicates the changes to the file system structure.
1106 That is what you have it for.
1107
1108 Also, don't use resize2fs on XFS volumes, or XFS tools on EXT, or ...
1109 but the right tool for the file system in use.
1110
1111 `resize2fs: Bad magic number in super-block while trying to open /dev/drbd7`
1112 is probably just trying to tell you that this is *not* an EXT file system,
1113 and you should try an other tool instead. Maybe `xfs_growfs`? But as mentioned,
1114 that does not take the block device, but the mount point as argument.
1115
1116 When shrinking (reducing) DRBD volumes, you need to shrink from top to bottom.
1117 So first make sure no one is using the space you want to cut off.
1118 Next, shrink the file system (if your file system supports that).
1119 Then tell DRBD to stop using that space, which is not so easy with DRBD
1120 internal meta data, because they are by design "at the end"
1121 of the backing device.
1122
1123 Once you are sure that DRBD won't use the space anymore either,
1124 you can cut it off from the backing device, for example using `lvreduce`.
1125
1126 See also <<s-shrinking-online>>, <<s-shrinking-offline>>.
1127
1128
1129 [[s-growing-online]]
1130 ==== Growing on-line
1131
1132 indexterm:[resource]If the backing block devices can be grown while in
1133 operation (online), it is also possible to increase the size of a DRBD
1134 device based on these devices during operation. To do so, two criteria
1135 must be fulfilled:
1136
1137 . The affected resource's backing device must be one managed by a
1138 logical volume management subsystem, such as LVM.
1139
1140 . The resource must currently be in the _Connected_ connection state.
1141
1142 Having grown the backing block devices on all nodes, ensure that only
1143 one node is in primary state. Then enter on one node:
1144
1145 ----------------------------
1146 # drbdadm resize <resource>
1147 ----------------------------
1148
1149 This triggers a synchronization of the new section. The
1150 synchronization is done from the primary node to the secondary node.
1151
1152 If the space you're adding is clean, you can skip syncing the additional
1153 space by using the --assume-clean option.
1154
1155 ----------------------------
1156 # drbdadm -- --assume-clean resize <resource>
1157 ----------------------------
1158
1159 [[s-growing-offline]]
1160 ==== Growing off-line
1161
1162 indexterm:[resource]When the backing block devices on both nodes are
1163 grown while DRBD is inactive, and the DRBD resource is using
1164 <<s-external-meta-data,external meta data>>, then the new size is
1165 recognized automatically. No administrative intervention is
1166 necessary. The DRBD device will have the new size after the next
1167 activation of DRBD on both nodes and a successful establishment of a
1168 network connection.
1169
1170 If however the DRBD resource is configured to use
1171 <<s-internal-meta-data,internal meta data>>, then this meta data must
1172 be moved to the end of the grown device before the new size becomes
1173 available. To do so, complete the following steps:
1174
1175 WARNING: This is an advanced procedure. Use at your own discretion.
1176
1177 * Unconfigure your DRBD resource:
1178
1179 [source,drbd]
1180
1181 ----------------------------
1182 # drbdadm down <resource>
1183 ----------------------------
1184
1185 * Save the meta data in a text file prior to resizing:
1186
1187 ----------------------------
1188 # drbdadm dump-md <resource> > /tmp/metadata
1189 ----------------------------
1190
1191 You must do this on both nodes, using a separate dump file for every
1192 node. *Do not* dump the meta data on one node, and simply copy the
1193 dump file to the peer. *This. will. not. work.*
1194
1195 * Grow the backing block device on both nodes.
1196
1197 * Adjust the size information (`la-size-sect`) in the file
1198 `/tmp/metadata` accordingly, on both nodes. Remember that
1199 `la-size-sect` must be specified in sectors.
1200
1201 * Re-initialize the metadata area:
1202
1203 ----------------------------
1204 # drbdadm create-md <resource>
1205 ----------------------------
1206
1207 * Re-import the corrected meta data, on both nodes:
1208
1209 ----------------------------
1210 # drbdmeta_cmd=$(drbdadm -d dump-md <resource>)
1211 # ${drbdmeta_cmd/dump-md/restore-md} /tmp/metadata
1212 Valid meta-data in place, overwrite? [need to type 'yes' to confirm]
1213 yes
1214 Successfully restored meta data
1215 ----------------------------
1216
1217 NOTE: This example uses `bash` parameter substitution. It may or may
1218 not work in other shells. Check your `SHELL` environment variable if
1219 you are unsure which shell you are currently using.
1220
1221 * Re-enable your DRBD resource:
1222
1223 ----------------------------
1224 # drbdadm up <resource>
1225 ----------------------------
1226
1227 * On one node, promote the DRBD resource:
1228
1229 ----------------------------
1230 # drbdadm primary <resource>
1231 ----------------------------
1232
1233 * Finally, grow the file system so it fills the extended size of the
1234 DRBD device.
1235
1236
1237 [[s-shrinking-online]]
1238 ==== Shrinking on-line
1239
1240
1241 WARNING: Online shrinking is only supported with external metadata.
1242
1243 indexterm:[resource]Before shrinking a DRBD device, you _must_ shrink
1244 the layers above DRBD, i.e. usually the file system. Since DRBD cannot
1245 ask the file system how much space it actually uses, you have to be
1246 careful in order not to cause data loss.
1247
1248 NOTE: Whether or not the _filesystem_ can be shrunk on-line depends on
1249 the filesystem being used. Most filesystems do not support on-line
1250 shrinking. XFS does not support shrinking at all.
1251
1252 To shrink DRBD on-line, issue the following command _after_ you have
1253 shrunk the file system residing on top of it:
1254
1255 [source,drbd]
1256 ----------------------------
1257 # drbdadm resize --size=<new-size> <resource>
1258 ----------------------------
1259
1260 You may use the usual multiplier suffixes for _<new-size>_ (K, M, G
1261 etc.). After you have shrunk DRBD, you may also shrink the containing
1262 block device (if it supports shrinking).
1263
1264 NOTE: It might be a good idea to issue `drbdadm resize _<resource>_` after
1265 resizing the lower level device, so that the DRBD metadata *really* gets
1266 written into the expected space at the end of the volume.
1267
1268 [[s-shrinking-offline]]
1269 ==== Shrinking off-line
1270
1271 indexterm:[resource]If you were to shrink a backing block device while
1272 DRBD is inactive, DRBD would refuse to attach to this block device
1273 during the next attach attempt, since it is now too small (in case
1274 external meta data is used), or it would be unable to find its meta
1275 data (in case internal meta data is used). To work around these
1276 issues, use this procedure (if you cannot use
1277 <<s-shrinking-online,on-line shrinking>>):
1278
1279
1280 WARNING: This is an advanced procedure. Use at your own discretion.
1281
1282 * Shrink the file system from one node, while DRBD is still
1283 configured.
1284
1285 * Unconfigure your DRBD resource:
1286
1287 ----------------------------
1288 # drbdadm down <resource>
1289 ----------------------------
1290
1291 * Save the meta data in a text file prior to shrinking:
1292
1293 ----------------------------
1294 # drbdadm dump-md <resource> > /tmp/metadata
1295 ----------------------------
1296
1297 You must do this on both nodes, using a separate dump file for every
1298 node. *Do not* dump the meta data on one node, and simply copy the dump
1299 file to the peer. *This. will. not. work.*
1300
1301 * Shrink the backing block device on both nodes.
1302
1303 * Adjust the size information (`la-size-sect`) in the file
1304 `/tmp/metadata` accordingly, on both nodes. Remember that
1305 `la-size-sect` must be specified in sectors.
1306
1307 * *Only if you are using internal metadata* (which at this time have
1308 probably been lost due to the shrinking process), re-initialize the
1309 metadata area:
1310 +
1311 ----------------------------
1312 # drbdadm create-md <resource>
1313 ----------------------------
1314
1315 * Re-import the corrected meta data, on both nodes:
1316 +
1317 ----------------------------
1318 # drbdmeta_cmd=$(drbdadm -d dump-md <resource>)
1319 # ${drbdmeta_cmd/dump-md/restore-md} /tmp/metadata
1320 Valid meta-data in place, overwrite? [need to type 'yes' to confirm]
1321 yes
1322 Successfully restored meta data
1323 ----------------------------
1324
1325 NOTE: This example uses `bash` parameter substitution. It may or may not
1326 work in other shells. Check your `SHELL` environment variable if you
1327 are unsure which shell you are currently using.
1328
1329 * Re-enable your DRBD resource:
1330 +
1331 ----------------------------
1332 # drbdadm up <resource>
1333 ----------------------------
1334
1335
1336 [[s-disable-flushes]]
1337 === Disabling backing device flushes
1338
1339 CAUTION: You should only disable device flushes when running DRBD on
1340 devices with a battery-backed write cache (BBWC). Most storage
1341 controllers allow to automatically disable the write cache when the
1342 battery is depleted, switching to write-through mode when the battery
1343 dies. It is strongly recommended to enable such a feature.
1344
1345 Disabling DRBD's flushes when running without BBWC, or on BBWC with a
1346 depleted battery, is _likely to cause data loss_ and should not be
1347 attempted.
1348
1349 DRBD allows you to enable and disable <<s-disk-flush-support,backing
1350 device flushes>> separately for the replicated data set and DRBD's own
1351 meta data. Both of these options are enabled by default. If you wish
1352 to disable either (or both), you would set this in the `disk` section
1353 for the DRBD configuration file, `/etc/drbd.conf`.
1354
1355 To disable disk flushes for the replicated data set, include the
1356 following line in your configuration:
1357
1358 [source,drbd]
1359 ----------------------------
1360 resource <resource>
1361 disk {
1362 disk-flushes no;
1363 ...
1364 }
1365 ...
1366 }
1367 ----------------------------
1368
1369
1370 To disable disk flushes on DRBD's meta data, include the following
1371 line:
1372
1373 [source,drbd]
1374 ----------------------------
1375 resource <resource>
1376 disk {
1377 md-flushes no;
1378 ...
1379 }
1380 ...
1381 }
1382 ----------------------------
1383
1384 After you have modified your resource configuration (and synchronized
1385 your `/etc/drbd.conf` between nodes, of course), you may enable these
1386 settings by issuing this command on both nodes:
1387
1388 ----------------------------
1389 # drbdadm adjust <resource>
1390 ----------------------------
1391
1392
1393 In case only one of the serves has a BBWCfootnote:[For example, in the DR site
1394 you might be using different hardware, right?], you should move the setting into a host section, like this:
1395
1396 [source,drbd]
1397 ----------------------------
1398 resource <resource> {
1399 disk {
1400 ... common settings ...
1401 }
1402
1403 on host-1 {
1404 disk {
1405 md-flushes no;
1406 }
1407 ...
1408 }
1409 ...
1410 }
1411 ----------------------------
1412
1413
1414 [[s-configure-split-brain-behavior]]
1415 === Configuring split brain behavior
1416
1417 [[s-split-brain-notification]]
1418 ==== Split brain notification
1419
1420 DRBD invokes the `split-brain` handler, if configured, at any time
1421 split brain is _detected_. To configure this handler, add the
1422 following item to your resource configuration:
1423
1424 ----------------------------
1425 resource <resource>
1426 handlers {
1427 split-brain <handler>;
1428 ...
1429 }
1430 ...
1431 }
1432 ----------------------------
1433
1434 _<handler>_ may be any executable present on the system.
1435
1436 The DRBD distribution contains a split brain handler script that
1437 installs as `/usr/lib/drbd/notify-split-brain.sh`. It simply sends a
1438 notification e-mail message to a specified address. To configure the
1439 handler to send a message to `root@localhost` (which is expected to be
1440 an email address that forwards the notification to a real system
1441 administrator), configure the `split-brain handler` as follows:
1442
1443 ----------------------------
1444 resource <resource>
1445 handlers {
1446 split-brain "/usr/lib/drbd/notify-split-brain.sh root";
1447 ...
1448 }
1449 ...
1450 }
1451 ----------------------------
1452
1453 After you have made this modification on a running resource (and
1454 synchronized the configuration file between nodes), no additional
1455 intervention is needed to enable the handler. DRBD will simply invoke
1456 the newly-configured handler on the next occurrence of split brain.
1457
1458 [[s-automatic-split-brain-recovery-configuration]]
1459 ==== Automatic split brain recovery policies
1460
1461 CAUTION: Configuring DRBD to automatically resolve data divergence
1462 situations resulting from split-brain (or other) scenarios
1463 is configuring for potential *automatic data loss*.
1464 Understand the implications, and don't do it if you don't mean to.
1465
1466 TIP: You rather want to look into fencing policies, quorum settings,
1467 cluster manager integration, and redundant cluster manager communication links
1468 to *avoid* data divergence in the first place.
1469
1470 In order to be able to enable and configure DRBD's automatic split
1471 brain recovery policies, you must understand that DRBD offers several
1472 configuration options for this purpose. DRBD applies its split brain
1473 recovery procedures based on the number of nodes in the Primary role
1474 at the time the split brain is detected. To that end, DRBD examines
1475 the following keywords, all found in the resource's `net` configuration
1476 section:
1477
1478 .`after-sb-0pri`
1479 Split brain has just been detected, but at this time the resource is
1480 not in the Primary role on any host. For this option, DRBD understands
1481 the following keywords:
1482
1483 * `disconnect`: Do not recover automatically, simply invoke the
1484 `split-brain` handler script (if configured), drop the connection and
1485 continue in disconnected mode.
1486
1487 * `discard-younger-primary`: Discard and roll back the modifications
1488 made on the host which assumed the Primary role last.
1489
1490 * `discard-least-changes`: Discard and roll back the modifications on
1491 the host where fewer changes occurred.
1492
1493 * `discard-zero-changes`: If there is any host on which no changes
1494 occurred at all, simply apply all modifications made on the other
1495 and continue.
1496
1497 .`after-sb-1pri`
1498 Split brain has just been detected, and at this time the resource is
1499 in the Primary role on one host. For this option, DRBD understands the
1500 following keywords:
1501
1502 * `disconnect`: As with `after-sb-0pri`, simply invoke the
1503 `split-brain` handler script (if configured), drop the connection
1504 and continue in disconnected mode.
1505
1506 * `consensus`: Apply the same recovery policies as specified in
1507 `after-sb-0pri`. If a split brain victim can be selected after
1508 applying these policies, automatically resolve. Otherwise, behave
1509 exactly as if `disconnect` were specified.
1510
1511 * `call-pri-lost-after-sb`: Apply the recovery policies as specified
1512 in `after-sb-0pri`. If a split brain victim can be selected after
1513 applying these policies, invoke the `pri-lost-after-sb` handler on
1514 the victim node. This handler must be configured in the
1515 `handlers` section and is expected to forcibly remove the node from
1516 the cluster.
1517
1518 * `discard-secondary`: Whichever host is currently in the Secondary
1519 role, make that host the split brain victim.
1520
1521 .`after-sb-2pri`
1522 Split brain has just been detected, and at this time the resource is
1523 in the Primary role on both hosts. This option accepts the same
1524 keywords as `after-sb-1pri` except `discard-secondary` and `consensus`.
1525
1526 NOTE: DRBD understands additional keywords for these three options,
1527 which have been omitted here because they are very rarely used. Refer
1528 to the man page of `drbd.conf` for details on split brain recovery keywords not
1529 discussed here.
1530
1531 For example, a resource which serves as the block device for a GFS or
1532 OCFS2 file system in dual-Primary mode may have its recovery policy
1533 defined as follows:
1534
1535 ----------------------------
1536 resource <resource> {
1537 handlers {
1538 split-brain "/usr/lib/drbd/notify-split-brain.sh root"
1539 ...
1540 }
1541 net {
1542 after-sb-0pri discard-zero-changes;
1543 after-sb-1pri discard-secondary;
1544 after-sb-2pri disconnect;
1545 ...
1546 }
1547 ...
1548 }
1549 ----------------------------
1550
1551
1552 [[s-three-nodes]]
1553 === Creating a stacked three-node setup
1554
1555 A three-node setup involves one DRBD device _stacked_ atop another.
1556
1557 NOTE: Stacking is deprecated in DRBD version
1558 9.x, as more nodes can be implemented on a single level. See
1559 <<s-drbdconf-conns>> for details.
1560
1561 [[s-stacking-considerations]]
1562 ==== Device stacking considerations
1563
1564 The following considerations apply to this type of setup:
1565
1566 * The stacked device is the active one. Assume you have configured one
1567 DRBD device `/dev/drbd0`, and the stacked device atop it is
1568 `/dev/drbd10`, then `/dev/drbd10` will be the device that you mount
1569 and use.
1570
1571 * Device meta data will be stored twice, on the underlying DRBD device
1572 _and_ the stacked DRBD device. On the stacked device, you must always
1573 use <<s-internal-meta-data,internal meta data>>. This means that the
1574 effectively available storage area on a stacked device is slightly
1575 smaller, compared to an unstacked device.
1576
1577 * To get the stacked upper level device running, the underlying device
1578 must be in the primary role.
1579
1580 * To be able to synchronize the backup node, the stacked device on the
1581 active node must be up and in the primary role.
1582
1583
1584 [[s-three-node-config]]
1585 ==== Configuring a stacked resource
1586
1587 In the following example, nodes are named 'alice', 'bob', and
1588 'charlie', with 'alice' and 'bob' forming a two-node cluster, and
1589 'charlie' being the backup node.
1590
1591 [source,drbd]
1592 ----------------------------
1593 resource r0 {
1594 protocol C;
1595 device /dev/drbd0;
1596 disk /dev/sda6;
1597 meta-disk internal;
1598
1599 on alice {
1600 address 10.0.0.1:7788;
1601 }
1602
1603 on bob {
1604 address 10.0.0.2:7788;
1605 }
1606 }
1607
1608 resource r0-U {
1609 protocol A;
1610
1611 stacked-on-top-of r0 {
1612 device /dev/drbd10;
1613 address 192.168.42.1:7789;
1614 }
1615
1616 on charlie {
1617 device /dev/drbd10;
1618 disk /dev/hda6;
1619 address 192.168.42.2:7789; # Public IP of the backup node
1620 meta-disk internal;
1621 }
1622 }
1623 ----------------------------
1624
1625 As with any `drbd.conf` configuration file, this must be distributed
1626 across all nodes in the cluster -- in this case, three nodes. Notice
1627 the following extra keyword not found in an unstacked resource
1628 configuration:
1629
1630 .`stacked-on-top-of`
1631 This option informs DRBD that the resource which contains it is a
1632 stacked resource. It replaces one of the `on` sections normally found
1633 in any resource configuration. Do not use `stacked-on-top-of` in an
1634 lower-level resource.
1635
1636
1637 NOTE: It is not a requirement to use <<fp-protocol-a,Protocol A>> for
1638 stacked resources. You may select any of DRBD's replication protocols
1639 depending on your application.
1640
1641 [[f-single-stacked]]
1642 .Single stacked setup
1643 image::images/single-stacked.svg[]
1644
1645
1646 [[s-three-node-enable]]
1647 ==== Enabling stacked resources
1648
1649 To enable a stacked resource, you first enable its lower-level
1650 resource and promote it:
1651 ----------------------------
1652 drbdadm up r0
1653 drbdadm primary r0
1654 ----------------------------
1655
1656 As with unstacked resources, you must create DRBD meta data on the
1657 stacked resources. This is done using the following command:
1658
1659 ----------------------------
1660 # drbdadm create-md --stacked r0-U
1661 ----------------------------
1662
1663 Then, you may enable the stacked resource:
1664
1665 ----------------------------
1666 # drbdadm up --stacked r0-U
1667 # drbdadm primary --stacked r0-U
1668 ----------------------------
1669
1670 After this, you may bring up the resource on the backup node, enabling
1671 three-node replication:
1672
1673 ----------------------------
1674 # drbdadm create-md r0-U
1675 # drbdadm up r0-U
1676 ----------------------------
1677
1678 In order to automate stacked resource management, you may integrate
1679 stacked resources in your cluster manager configuration.
1680 ifdef::drbd-only[]
1681 See <<s-pacemaker-stacked-resources>> for information on doing this in a
1682 cluster managed by the Pacemaker cluster management framework.
1683 endif::drbd-only[]
1684
1685 [[s-permanently-diskless-nodes]]
1686 === Permanently diskless nodes
1687
1688 A node might be permanently diskless in DRBD. Here is a configuration example
1689 showing a resource with 3 diskful nodes (servers) and one permanently diskless
1690 node (client).
1691
1692 -------------------------------------
1693 resource kvm-mail {
1694 device /dev/drbd6;
1695 disk /dev/vg/kvm-mail;
1696 meta-disk internal;
1697
1698 on store1 {
1699 address 10.1.10.1:7006;
1700 node-id 0;
1701 }
1702 on store2 {
1703 address 10.1.10.2:7006;
1704 node-id 1;
1705 }
1706 on store3 {
1707 address 10.1.10.3:7006;
1708 node-id 2;
1709 }
1710
1711 on for-later-rebalancing {
1712 address 10.1.10.4:7006;
1713 node-id 3;
1714 }
1715
1716 # DRBD "client"
1717 floating 10.1.11.6:8006 {
1718 disk none;
1719 node-id 4;
1720 }
1721
1722 # rest omitted for brevity
1723 ...
1724 }
1725 -------------------------------------
1726
1727 For permanently diskless nodes no bitmap slot gets allocated. For such nodes the
1728 diskless status is displayed in green color since it is not an error or unexpected
1729 state. See <<s-drbd-client-internals>> for internal details.
1730
1731 [[s-rebalance-workflow]]
1732 === Data rebalancing
1733
1734 indexterm:[rebalance]Given the (example) policy that data needs to be available on 3 nodes,
1735 you need at least 3 servers for your setup.
1736
1737 Now, as your storage demands grow, you will encounter the need for
1738 additional servers. Rather than having to buy 3 more servers at the same
1739 time, you can _rebalance_ your data across a single additional node.
1740
1741
1742 .DRBD data rebalancing
1743 image::images/rebalance.svg[]
1744
1745 In the figure above you can see the _before_ and _after_ states: from
1746 3 nodes with three 25TiB volumes each (for a net 75TiB), to 4 nodes,
1747 with net 100TiB.
1748
1749 To redistribute the data across your cluster you have to choose a _new_ node,
1750 and one where you want to remove this DRBD resource. +
1751 Please note that removing the resource from a currently _active_ node (i.e.
1752 where DRBD is _Primary_) will involve either migrating the service or running
1753 this resource on this node as a <<s-drbd-client,DRBD client>>; it's easier to
1754 choose a node in _Secondary_ role. (Of course, that might not always be possible.)
1755
1756
1757 [[s-prepare_a_bitmap_slot]]
1758 ==== Prepare a bitmap slot
1759
1760 You will need to have a free <<s-quick-sync-bitmap,bitmap slot>> for
1761 temporary use, on each of the nodes that have the resource that is to be moved.
1762
1763 You can allocate one more at
1764 <<s-first-time-up,`drbdadm create-md` time>>, or simply put a placeholder in
1765 your configuration, so that `drbdadm` sees that it should reserve one more slot:
1766
1767 ---------
1768 resource r0 {
1769 ...
1770 on for-later-rebalancing {
1771 address 10.254.254.254:65533;
1772 node-id 3;
1773 }
1774 }
1775 ---------
1776
1777 [NOTE]
1778 =================
1779 If you need to make that slot available during live use,
1780 you will have to
1781
1782 . dump the metadata,
1783 . enlarge the metadata space,
1784 . edit the dumpfile,
1785 . load the changed metadata.
1786
1787 In a future version `drbdadm` will have a shortcut for you; most
1788 probably you'll be able to say `drbdadm resize --peers _N_` and have the
1789 kernel rewrite the metadata for you.
1790 =================
1791
1792
1793 [[s-preparing_and_activating_the_new_node]]
1794 ==== Preparing and activating the new node
1795
1796 First of all you have to
1797 create the underlying storage volume on the new node (using e.g. `lvcreate`).
1798 Then the placeholder in the configuration can be filled with the correct host
1799 name, address, and storage path. Now copy the resource configuration to all
1800 relevant nodes.
1801
1802
1803 On the new node initialize the meta-data (once) by doing
1804
1805 -------------
1806 # drbdadm create-md <resource>
1807 v09 Magic number not found
1808 Writing meta data...
1809 initialising activity log
1810 NOT initializing bitmap
1811 New drbd meta data block successfully created.
1812 -------------
1813
1814
1815 [[s-starting_the_initial_sync]]
1816 ==== Starting the initial sync
1817
1818 Now the new node needs to get the data.
1819
1820 This is done by defining the network connection on the existing nodes via
1821
1822 -------------------------------------
1823 # drbdadm adjust <resource>
1824 -------------------------------------
1825
1826 and starting the DRBD device on the new node via
1827
1828 -------------------------------------
1829 # drbdadm up <resource>
1830 -------------------------------------
1831
1832
1833 [[s-check_connectivity]]
1834 ==== Check connectivity
1835
1836 At this time please do a
1837 ------------
1838 # drbdadm status <resource>
1839 ------------
1840
1841 on the new node, and check that _all_ other nodes are connected.
1842
1843 [[s-after_the_initial_sync]]
1844 ==== After the initial sync
1845
1846 As soon as the new host is _UpToDate_, one of the
1847 other nodes in the configuration can be renamed to `for-later-rebalancing`, and
1848 kept for another migration.
1849
1850 NOTE: Perhaps you want to comment the section; although that has the risk that
1851 doing a `drbdadm create-md` for a new node has too few bitmap slots for the
1852 next rebalancing. +
1853 It might be easier to use a reserved (unused) IP address and host name.
1854
1855 Copy the changed configuration around again, and use it by running
1856
1857 -------------------------------------
1858 # drbdadm adjust <resource>
1859 -------------------------------------
1860
1861 on all nodes.
1862
1863
1864 [[s-cleaning_up]]
1865 ==== Cleaning up
1866
1867 On the one node that had the data up to now, but isn't used anymore for this
1868 resource, you can now take the DRBD device down by starting
1869
1870 -------------------
1871 # drbdadm down <resource>
1872 -------------------
1873
1874 Now the lower level storage device isn't used anymore, and can either be
1875 re-used for other purposes or, if it is a logical volume, its space can be
1876 returned to the volume group via `lvremove`.
1877
1878 [[s-conclusion_and_further_steps]]
1879 ==== Conclusion and further steps
1880
1881 One of the resources has been migrated to the new node. The same could be done
1882 for one or more other resources, to free space on two or three nodes in the
1883 existing cluster.
1884
1885 Then new resources can be configured, as there are enough nodes with free space
1886 to achieve 3-way redundancy again.
1887
1888 [[s-configuring-quorum]]
1889 === Configuring quorum
1890
1891 In order to avoid split brain or diverging data of replicas one has to configure
1892 fencing. All the options for fencing rely on redundant communication in the end.
1893 That might be in the form of a management network that connects the nodes to the IPMI
1894 network interfaces of the peer machines. In case of the crm-fence-peer script it
1895 is necessary that Pacemakers communication stays available when DRBD's network
1896 link breaks.
1897
1898 The quorum mechanism on the other hand takes a completely difference approach. The basic
1899 idea is that a cluster partition may only modify the replicated data set if the
1900 number of nodes that can communicate is greater than the half of the overall number
1901 of nodes. A node of such a partition _has quorum_.
1902 On the other hand a node does not have quorum needs to guarantee that the replicated
1903 data set it not touched, that it does not create a diverging data set.
1904
1905 The quorum implementation in DRBD gets enabled by setting the `quorum` resource
1906 option to `majority`, `all` or a numeric value. Where `majority` selects the
1907 behavior that was described in the previous paragraph.
1908
1909 [[s-guaranteed_minimal_redundancy]]
1910 ==== Guaranteed minimal redundancy
1911
1912 By default every node with a disk gets a vote in the quorum election. In other words
1913 only diskless nodes do not count. So a partition with two _Inconsistent_ disks gets
1914 quorum, while a partition with one _UpToDate_ node will have quorum in a 3 node
1915 cluster. By configuring `quorum-minimum-redundancy` this behavior can be changed so that
1916 only nodes that are `UpToDate` have a vote in the quorum election. The option takes the
1917 same arguments as the `quorum` option.
1918
1919 With this option you express that you rather want to wait until eventually necessary
1920 resync operations finish before any services start. So in a way you prefer
1921 that the minimal redundancy of your data is guaranteed over the availability of your
1922 service. Financial data and services is an example that comes to mind.
1923
1924 Consider following example for a 5 node cluster. It requires a partitions to have at
1925 least 3 nodes, and two of them must be `UpToDate`:
1926
1927 -------------------------------------
1928 resource quorum-demo {
1929 options {
1930 quorum majority;
1931 quorum-minimum-redundancy 2;
1932 ...
1933 }
1934 }
1935 -------------------------------------
1936
1937 [[s-actions_on_loss_of_quorum]]
1938 ==== Actions on loss of quorum
1939
1940 When a node that is running the service loses quorum it needs to cease write-operations
1941 on the data set immediately. That means that IO immediately starts to complete all IO
1942 requests with errors. Usually that means that a graceful shutdown is not possible, since
1943 that would require more modifications to the data set. The IO errors propagate from
1944 the block level to the file system and from the file system to the user space application(s).
1945
1946 Ideally the application simply terminates in case of IO errors. This allows then Pacemaker
1947 to unmount the filesystem and to demote the DRBD resource to secondary role.
1948 If that is true you should set the `on-no-quorum` resource option to `io-error`.
1949 Here is an example:
1950
1951 -------------------------------------
1952 resource quorum-demo {
1953 options {
1954 quorum majority;
1955 on-no-quorum io-error;
1956 ...
1957 }
1958 }
1959 -------------------------------------
1960
1961 If you application does not terminate on the first IO error, you can choose to freeze IO
1962 instead and to reboot the node. Here is a configuration example:
1963
1964 -------------------------------------
1965 resource quorum-demo {
1966 options {
1967 quorum majority;
1968 on-no-quorum suspend-io;
1969 ...
1970 }
1971
1972 handlers {
1973 quorum-lost "echo b > /proc/sysrq-trigger";
1974 }
1975 ...
1976 }
1977 -------------------------------------
1978
1979 [[s-configuring-quorum-tiebreaker]]
1980 ==== Using a diskless node as a tiebreaker
1981
1982 A diskless node with connections to all nodes in a cluster can be utilized to break ties
1983 in the quorum negotiation process.
1984
1985 Consider the following two-node cluster, where node A is the primary and node B is a secondary:
1986
1987 image::images/quorum-tiebreaker-without.svg[]
1988
1989 As soon as the connection between the two nodes is interrupted, they lose quorum
1990 and the application on top of the cluster cannot write data anymore.
1991
1992 image::images/quorum-tiebreaker-without-disconnect.svg[]
1993
1994 Now if we add a third node, C, to the cluster and configure it as diskless, we
1995 can take advantage of the tiebreaker mechanism.
1996
1997 image::images/quorum-tiebreaker.svg[]
1998
1999 In this case, when the primary and secondary nodes lose connection, they can
2000 still "see" the diskless tiebreaker. Because of this, the primary can continue
2001 working, while the secondary demotes its disk to Outdated and the service can
2002 therefore not be migrated there.
2003
2004 image::images/quorum-tiebreaker-disconnect.svg[]
2005
2006 There are a few special cases in case two connections fail. Consider the following
2007 scenario:
2008
2009 image::images/quorum-tiebreaker-disconnect-case2.svg[]
2010
2011 In this case, the tiebreaker node forms a partition with the primary node. The primary
2012 therefore keeps quorum, while the secondary becomes outdated. Note that the secondary's
2013 disk state will be "UpToDate", but regardless it cannot be promoted to primary because
2014 it lacks quorum.
2015
2016 Let's consider the possibility of the primary losing connection to the tiebreaker:
2017
2018 image::images/quorum-tiebreaker-disconnect-case3.svg[]
2019
2020 In this case, the primary becomes unusable and goes into a "quorum suspended" state. This
2021 effectively results in the application on top of DRBD receiving I/O errors. A cluster manager
2022 could then promote node B to primary and keep the service running there instead.
2023
2024 In case the diskless tiebreaker "switches sides", we need to avoid data divergence. Consider
2025 this scenario:
2026
2027 image::images/quorum-tiebreaker-disconnect-case1a.svg[]
2028
2029 The connection between the primary and secondary has failed, and the application is continuing
2030 to run on the primary, when the primary suddenly loses its connection to the diskless node.
2031
2032 In this case, no node can be promoted to primary and the cluster cannot continue to operate.
2033
2034 [NOTE]
2035 ====================
2036 Protecting from data divergence always takes priority over ensuring service availability.
2037 ====================
2038
2039 Let's look at another scenario:
2040
2041 image::images/quorum-tiebreaker-disconnect-case2a.svg[]
2042
2043 Here, the application is running on the primary, while the secondary is unavailable. Then,
2044 the tiebreaker first loses connection to the primary, and then reconnects to the secondary.
2045 It is important to note here that *a node that has lost quorum cannot regain quorum by connecting
2046 to a diskless node*. Thus, in this case, no node has quorum and the cluster halts.
2047
2048 [[s-quorum_last_man_standing]]
2049 ==== Last man standing
2050
2051 It needs to be mentioned that nodes that leave a cluster gracefully are counted
2052 different than failed nodes. In this context _leaving gracefully_ means that the leaving
2053 node marked it's data as Outdated, and that it was able to tell the remaining nodes
2054 that it's data is Outdated.
2055
2056 In a group of nodes where all disks are Outdated no one in that group can promote to Primary
2057 footnote:[The exception is a manual promotion with the --force flag. We assume that
2058 one that uses the --force knows what he is doing.].
2059
2060 An implication is that, if a cluster is left by all but one node, it can keep quorum as
2061 long as all the others left in a graceful way. If a single node leaves ungraceful,
2062 it has to assume that all away nodes form a partition and have access to up-to-date
2063 data. If the other partition might (potentially) be larger than my own partition it
2064 looses quorum.
2065
2066 [[s-removing-DRBD]]
2067 === Removing DRBD
2068
2069 For the unlikely case that you want to remove DRBD, here are the necessary steps.
2070
2071 . Stop the services and unmount the filesystems on top of the DRBD volumes. In case you are
2072 using a cluster manager, make sure it ceases to control the services fist.
2073 . Stop the DRBD resource(s) by using `drbdadm down <res>` or `drbdadm down all`
2074 .. In case the DRBD resource was using internal meta-data you might choose to
2075 resize the filesystem to cover all of the backing device's space. This step
2076 effectively removes DRBD's meta-data. This is an action that can not be reversed
2077 easily. You can do that with `resize2fs <backing_dev>` for ext[234] family of file systems.
2078 It supports resizing of unmounted filesystem and under certain conditions also online
2079 grow. XFS can be grown online only with the `xfs_growfs` command.
2080 . Mount the backing device(s) directly, start the services on top of them
2081 . Unload the drbd kernel driver modules with `rmmod drbd_transport_tcp` and `rmmod drbd`.
2082 . Uninstall the drbd software packages.
2083
2084 //- list divider. Keep it otherwise next included section is interpreted as list element