ifdef::env-github[] :tip-caption: :bulb: :note-caption: :information_source: :important-caption: :heavy_exclamation_mark: :caution-caption: :fire: :warning-caption: :warning: endif::[] [[ch-admin-manual]] == Common administrative tasks This chapter outlines typical administrative tasks encountered during day-to-day operations. It does not cover troubleshooting tasks, these are covered in detail in <>. include::configure.adoc[] [[s-check-status]] === Checking DRBD status [[s-drbd-drbdmon]] ==== Retrieving status with drbdmon One convenient way to look at DRBD's status is the `drbdmon` utility. It updates the state of DRBD resources in real time. [[s-proc-drbd]] ==== Status information in `/proc/drbd` NOTE: `/proc/drbd` is deprecated. While it won't be removed in the 8.4 series, we recommend to switch to other means, like <>; or, for monitoring even more convenient, <>. indexterm:[/proc/drbd]`/proc/drbd` is a virtual file displaying basic information about the DRBD module. It was used extensively up to DRBD 8.4, but couldn't keep up with the amount of information provided by DRBD 9. ---------------------------- $ cat /proc/drbd version: 9.0.0 (api:1/proto:86-110) GIT-hash: XXX build by linbit@buildsystem.linbit, 2011-10-12 09:07:35 ---------------------------- The first line, prefixed with `version:`, shows the DRBD version used on your system. The second line contains information about this specific build. [[s-drbdadm-status]] ==== Status information via `drbdadm` indexterm:[drbdadm, status]In its simplest invocation, we just ask for the status of a single resource. ---------------------------- # drbdadm status home home role:Secondary disk:UpToDate nina role:Secondary disk:UpToDate nino role:Secondary disk:UpToDate nono connection:Connecting ---------------------------- This here just says that the resource _home_ is locally, on 'nina', and on 'nino' _UpToDate_ and _Secondary_; so the three nodes have the same data on their storage devices, and nobody is using the device currently. The node 'nono' is not connected, its state is reported as _Connecting_; please see <> below for more details. You can get more information by passing the `--verbose` and/or `--statistics` arguments to `drbdsetup` (lines broken for readability): ---------------------------- # drbdsetup status home --verbose --statistics home node-id:1 role:Secondary suspended:no write-ordering:none volume:0 minor:0 disk:UpToDate size:1048412 read:0 written:1048412 al-writes:0 bm-writes:48 upper-pending:0 lower-pending:0 al-suspended:no blocked:no nina local:ipv4:10.9.9.111:7001 peer:ipv4:10.9.9.103:7010 node-id:0 connection:Connected role:Secondary congested:no volume:0 replication:Connected disk:UpToDate resync-suspended:no received:1048412 sent:0 out-of-sync:0 pending:0 unacked:0 nino local:ipv4:10.9.9.111:7021 peer:ipv4:10.9.9.129:7012 node-id:2 connection:Connected role:Secondary congested:no volume:0 replication:Connected disk:UpToDate resync-suspended:no received:0 sent:0 out-of-sync:0 pending:0 unacked:0 nono local:ipv4:10.9.9.111:7013 peer:ipv4:10.9.9.138:7031 node-id:3 connection:Connecting ---------------------------- Every few lines in this example form a block that is repeated for every node used in this resource, with small format exceptions for the local node - see below for more details. The first line in each block shows the `node-id` (for the current resource; a host can have different ``node-id``s in different resources). Furthermore the `role` (see <>) is shown. The next important line begins with the `volume` specification; normally these are numbered starting by zero, but the configuration may specify other IDs as well. This line shows the indexterm:[connection state] connection state in the `replication` item (see <> for details) and the remote indexterm:[disk state] disk state in `disk` (see <>). Then there's a line for this volume giving a bit of statistics - data `received`, `sent`, `out-of-sync`, etc; please see <> and <> for more information. For the local node the first line shows the resource name, _home_, in our example. As the first block always describes the local node, there is no _Connection_ or address information. please see the `drbd.conf` manual page for more information. The other four lines in this example form a block that is repeated for every DRBD device configured, prefixed by the device minor number. In this case, this is `0`, corresponding to the device `/dev/drbd0`. The resource-specific output contains various pieces of information about the resource: /////// //FIXME .Replication protocol The Replication protocol used by the resource. Either `A`, `B`, or `C`. See <> for details. /////// [[s-drbdsetup-events2]] ==== One-shot or real time monitoring via `drbdsetup events2` NOTE: This is available only with userspace versions 8.9.3 and up. // and 8.4.6 This is a low-level mechanism to get information out of DRBD, suitable for use in automated tools, like monitoring. In its simplest invocation, showing only the current status, the output looks like this (but, when running on a terminal, will include colors): --------------- # drbdsetup events2 --now r0 exists resource name:r0 role:Secondary suspended:no exists connection name:r0 peer-node-id:1 conn-name:remote-host connection:Connected role:Secondary exists device name:r0 volume:0 minor:7 disk:UpToDate exists device name:r0 volume:1 minor:8 disk:UpToDate exists peer-device name:r0 peer-node-id:1 conn-name:remote-host volume:0 replication:Established peer-disk:UpToDate resync-suspended:no exists peer-device name:r0 peer-node-id:1 conn-name:remote-host volume:1 replication:Established peer-disk:UpToDate resync-suspended:no exists - --------------- Without the ''--now'', the process will keep running, and send continuous updates like this: ----------------- # drbdsetup events2 r0 ... change connection name:r0 peer-node-id:1 conn-name:remote-host connection:StandAlone change connection name:r0 peer-node-id:1 conn-name:remote-host connection:Unconnected change connection name:r0 peer-node-id:1 conn-name:remote-host connection:Connecting ----------------- Then, for monitoring purposes, there's another argument ''--statistics'', that will produce some performance counters and other facts: 'drbdsetup' **verbose output** (lines broken for readability): ----------------- # drbdsetup events2 --statistics --now r0 exists resource name:r0 role:Secondary suspended:no write-ordering:drain exists connection name:r0 peer-node-id:1 conn-name:remote-host connection:Connected role:Secondary congested:no exists device name:r0 volume:0 minor:7 disk:UpToDate size:6291228 read:6397188 written:131844 al-writes:34 bm-writes:0 upper-pending:0 lower-pending:0 al-suspended:no blocked:no exists device name:r0 volume:1 minor:8 disk:UpToDate size:104854364 read:5910680 written:6634548 al-writes:417 bm-writes:0 upper-pending:0 lower-pending:0 al-suspended:no blocked:no exists peer-device name:r0 peer-node-id:1 conn-name:remote-host volume:0 replication:Established peer-disk:UpToDate resync-suspended:no received:0 sent:131844 out-of-sync:0 pending:0 unacked:0 exists peer-device name:r0 peer-node-id:1 conn-name:remote-host volume:1 replication:Established peer-disk:UpToDate resync-suspended:no received:0 sent:6634548 out-of-sync:0 pending:0 unacked:0 exists - ----------------- You might also like the ''--timestamp'' parameter. [[s-connection-states]] ==== Connection states indexterm:[drbdadm, cstate]indexterm:[connection state, via drbdadm]A resource's connection state can be observed either by issuing the `drbdadm cstate` command: ---------------------------- # drbdadm cstate Connected Connected StandAlone ---------------------------- If you are interested in only a single connection of a resource, specify the connection name, too: The default is the peer's hostname as given in the configuration file. ---------------------------- # drbdadm cstate : Connected ---------------------------- A resource may have one of the following connection states: ._StandAlone_ indexterm:[connection state, StandAlone]No network configuration available. The resource has not yet been connected, or has been administratively disconnected (using `drbdadm disconnect`), or has dropped its connection due to failed authentication or split brain. ._Disconnecting_ indexterm:[connection state, Disconnecting]Temporary state during disconnection. The next state is _StandAlone_. ._Unconnected_ indexterm:[connection state, Unconnected]Temporary state, prior to a connection attempt. Possible next states: _Connecting_. ._Timeout_ indexterm:[connection state, Timeout]Temporary state following a timeout in the communication with the peer. Next state: _Unconnected_. ._BrokenPipe_ indexterm:[connection state, BrokenPipe]Temporary state after the connection to the peer was lost. Next state: _Unconnected_. ._NetworkFailure_ indexterm:[connection state, NetworkFailure]Temporary state after the connection to the partner was lost. Next state: _Unconnected_. ._ProtocolError_ indexterm:[connection state, ProtocolError]Temporary state after the connection to the partner was lost. Next state: _Unconnected_. ._TearDown_ indexterm:[connection state, TearDown]Temporary state. The peer is closing the connection. Next state: _Unconnected_. ._Connecting_ indexterm:[connection state, Connecting]This node is waiting until the peer node becomes visible on the network. ._Connected_ indexterm:[connection state, Connected]A DRBD connection has been established, data mirroring is now active. This is the normal state. [[s-replication-states]] ==== Replication states Each volume has over each connection a replication state. The possible replication states are: ._Off_ indexterm:[replication state, Off]The volume is not replicated over this connection, since the connection is not _Connected_. ._Established_ indexterm:[replication state, Established]All writes to that volume are replicated online. This is the normal state. ._StartingSyncS_ indexterm:[replication state, StartingSyncS]Full synchronization, initiated by the administrator, is just starting. The next possible states are: _SyncSource_ or _PausedSyncS_. ._StartingSyncT_ indexterm:[replication state, StartingSyncT]Full synchronization, initiated by the administrator, is just starting. Next state: _WFSyncUUID_. ._WFBitMapS_ indexterm:[replication state, WFBitMapS]Partial synchronization is just starting. Next possible states: _SyncSource_ or _PausedSyncS_. ._WFBitMapT_ indexterm:[replication state, WFBitMapT]Partial synchronization is just starting. Next possible state: _WFSyncUUID_. ._WFSyncUUID_ indexterm:[replication state, WFSyncUUID]Synchronization is about to begin. Next possible states: _SyncTarget_ or _PausedSyncT_. ._SyncSource_ indexterm:[replication state, SyncSource]Synchronization is currently running, with the local node being the source of synchronization. ._SyncTarget_ indexterm:[replication state, SyncTarget]Synchronization is currently running, with the local node being the target of synchronization. ._PausedSyncS_ indexterm:[replication state, PausedSyncS]The local node is the source of an ongoing synchronization, but synchronization is currently paused. This may be due to a dependency on the completion of another synchronization process, or due to synchronization having been manually interrupted by `drbdadm pause-sync`. ._PausedSyncT_ indexterm:[replication state, PausedSyncT]The local node is the target of an ongoing synchronization, but synchronization is currently paused. This may be due to a dependency on the completion of another synchronization process, or due to synchronization having been manually interrupted by `drbdadm pause-sync`. ._VerifyS_ indexterm:[replication state, VerifyS]On-line device verification is currently running, with the local node being the source of verification. ._VerifyT_ indexterm:[replication state, VerifyT]On-line device verification is currently running, with the local node being the target of verification. ._Ahead_ indexterm:[replication state, Ahead]Data replication was suspended, since the link can not cope with the load. This state is enabled by the configuration `on-congestion` option (see <>). ._Behind_ indexterm:[replication state, Behind]Data replication was suspended by the peer, since the link can not cope with the load. This state is enabled by the configuration `on-congestion` option on the peer node (see <>). [[s-roles]] ==== Resource roles indexterm:[resource]A resource's role can be observed by issuing the indexterm:[drbdadm, role]`drbdadm role` command: ---------------------------- # drbdadm role Primary ---------------------------- You may see one of the following resource roles: ._Primary_ indexterm:[role, Primary]The resource is currently in the primary role, and may be read from and written to. This role only occurs on one of the two nodes, unless <> is enabled. ._Secondary_ indexterm:[role, Secondary]The resource is currently in the secondary role. It normally receives updates from its peer (unless running in disconnected mode), but may neither be read from nor written to. This role may occur on one or both nodes. ._Unknown_ indexterm:[role, Unknown]The resource's role is currently unknown. The local resource role never has this status. It is only displayed for the peer's resource role, and only in disconnected mode. [[s-disk-states]] ==== Disk states A resource's disk state can be observed either by issuing the `drbdadm dstate` command: ---------------------------- # drbdadm dstate UpToDate ---------------------------- The disk state may be one of the following: ._Diskless_ indexterm:[disk state, Diskless]No local block device has been assigned to the DRBD driver. This may mean that the resource has never attached to its backing device, that it has been manually detached using `drbdadm detach`, or that it automatically detached after a lower-level I/O error. ._Attaching_ indexterm:[disk state, Attaching]Transient state while reading meta data. ._Detaching_ indexterm:[disk state, Detaching]Transient state while detaching and waiting for ongoing IOs to complete. ._Failed_ indexterm:[disk state, Failed]Transient state following an I/O failure report by the local block device. Next state: _Diskless_. ._Negotiating_ indexterm:[disk state, Negotiating]Transient state when an _Attach_ is carried out on an already-__Connected__ DRBD device. ._Inconsistent_ indexterm:[disk state, Inconsistent]The data is inconsistent. This status occurs immediately upon creation of a new resource, on both nodes (before the initial full sync). Also, this status is found in one node (the synchronization target) during synchronization. ._Outdated_ indexterm:[disk state, Outdated]Resource data is consistent, but <>. ._DUnknown_ indexterm:[disk state, DUnknown]This state is used for the peer disk if no network connection is available. ._Consistent_ indexterm:[disk state, Consistent]Consistent data of a node without connection. When the connection is established, it is decided whether the data is _UpToDate_ or _Outdated_. ._UpToDate_ indexterm:[disk state, UpToDate]Consistent, up-to-date state of the data. This is the normal state. [[s-conn-info]] ==== Connection information data ._local_ Shows the network family, the local address and port that is used to accept connections from the peer. ._peer_ Shows the network family, the peer address and port that is used to connect. ._congested_ This flag tells whether the TCP send buffer of the data connection is more than 80% filled. [[s-performance-indicators]] ==== Performance indicators indexterm:[performance indicators] The command ``drbdsetup status --verbose --statistics`` can be used to show performance statistics. These are also available in ``drbdsetup events2 --statistics``, although there will not be a _changed_ event for every change. The statistics include the following counters and gauges: Per volume/device: ._read_ (disk read) Net data read from local disk; in KiB. ._written_ (disk written) Net data written to local disk; in KiB. ._al-writes_ (activity log) Number of updates of the activity log area of the meta data. ._bm-writes_ (bitmap) Number of updates of the bitmap area of the meta data. ._upper-pending_ (application pending) Number of block I/O requests forwarded to DRBD, but not yet answered (completed) by DRBD. ._lower-pending_ (local count) Number of open requests to the local I/O sub-system issued by DRBD. ._blocked_ indexterm:[blocked]Shows local I/O congestion. -- * _no_: No congestion. * _upper_: I/O _above_ the DRBD device is blocked, i.e. to the filesystem. Typical causes are ** I/O suspension by the administrator, see the `suspend-io` command in `drbdadm`. ** transient blocks, e.g. during attach/detach ** buffers depleted, see <> ** Waiting for bitmap IO * _lower_: Backing device is congested. * _upper,lower_: Both _upper_ and _lower_ are blocked. -- Per connection: ._ap-in-flight_ (application in-flight) Application data that is being written by the peer. That is, DRBD has sent it to the peer and is waiting for the acknowledgement that it has been written. In sectors (512 bytes). ._rs-in-flight_ (resync in-flight) Resync data that is being written by the peer. That is, DRBD is _SyncSource_, has sent data to the peer as part of a resync and is waiting for the acknowledgement that it has been written. In sectors (512 bytes). Per connection and volume ("peer device"): ._received_ (network receive) Net data received from the peer; in KiB. ._sent_ (network send) Net data sent to the peer; in KiB. ._out-of-sync_ indexterm:[out-of-sync]indexterm:[oos]Amount of data currently out of sync with this peer, according to the bitmap that DRBD has for it; in KiB. ._pending_ Number of requests sent to the peer, but that have not yet been acknowledged by the peer. ._unacked_ (unacknowledged) Number of requests received from the peer, but that have not yet been acknowledged by DRBD on this node. ._resync-suspended_ Whether the resynchronization is currently suspended or not. Possible values are _no_, _user_, _peer_, _dependency_. Comma separated. [[s-enable-disable]] === Enabling and disabling resources [[s-enable-resource]] ==== Enabling resources indexterm:[resource]Normally, all configured DRBD resources are automatically enabled * by a cluster resource management application at its discretion, based on your cluster configuration, or * by the systemd units (e.g., `drbd@.target`) If, however, you need to enable resources manually for any reason, you may do so by issuing the command ---------------------------- # drbdadm up ---------------------------- As always, you may use the keyword `all` instead of a specific resource name if you want to enable all resources configured in `/etc/drbd.conf` at once. [[s-disable-resource]] ==== Disabling resources indexterm:[resource]You may temporarily disable specific resources by issuing the command ---------------------------- # drbdadm down ---------------------------- Here, too, you may use the keyword `all` in place of a resource name if you wish to temporarily disable all resources listed in `/etc/drbd.conf` at once. [[s-reconfigure]] === Reconfiguring resources indexterm:[resource]DRBD allows you to reconfigure resources while they are operational. To that end, * make any necessary changes to the resource configuration in `/etc/drbd.conf`, * synchronize your `/etc/drbd.conf` file between both nodes, * issue the indexterm:[drbdadm, adjust]`drbdadm adjust ` command on both nodes. `drbdadm adjust` then hands off to `drbdsetup` to make the necessary adjustments to the configuration. As always, you are able to review the pending `drbdsetup` invocations by running `drbdadm` with the `-d` (dry-run) option. NOTE: When making changes to the `common` section in `/etc/drbd.conf`, you can adjust the configuration for all resources in one run, by issuing `drbdadm adjust all`. [[s-switch-resource-roles]] === Promoting and demoting resources indexterm:[resource]Manually switching a <> from secondary to primary (promotion) or vice versa (demotion) is done using one of the following commands: ---------------------------- # drbdadm primary # drbdadm secondary ---------------------------- In <> (DRBD's default), any resource can be in the primary role on only one node at any given time while the <> is _Connected_. Thus, issuing `drbdadm primary ` on one node while the specified resource is still in the primary role on another node will result in an error. A resource configured to allow <> can be switched to the primary role on two nodes; this is e.g. needed for online migration of virtual machines. [[s-manual-fail-over]] === Basic manual fail-over If not using a cluster manager and looking to handle fail-overs manually in a passive/active configuration, the process is as follows. On the current primary node stop any applications or services using the DRBD device, unmount the DRBD device, and demote the resource to secondary. ---------------------------- # umount /dev/drbd/by-res// # drbdadm secondary ---------------------------- Now on the node we wish to make primary promote the resource and mount the device. ---------------------------- # drbdadm primary # mount /dev/drbd/by-res// ---------------------------- If you're using the `auto-promote` feature, you don't need to change the roles (_Primary_/_Secondary_) manually; only stopping of the services and unmounting, respectively mounting, is necessary. [[s-upgrading-drbd]] include::upgrading.adoc[] [[s-enable-dual-primary]] include::dualprimary.adoc[] [[s-use-online-verify]] === Using on-line device verification [[s-online-verify-enable]] ==== Enabling on-line verification indexterm:[on-line device verification]<> for resources is not enabled by default. To enable it, add the following lines to your resource configuration in `/etc/drbd.conf`: [source,drbd] ---------------------------- resource net { verify-alg ; } ... } ---------------------------- __ may be any message digest algorithm supported by the kernel crypto API in your system's kernel configuration. Normally, you should be able to choose at least from `sha1`, `md5`, and `crc32c`. If you make this change to an existing resource, as always, synchronize your `drbd.conf` to the peer, and run `drbdadm adjust __` on both nodes. [[s-online-verify-invoke]] ==== Invoking on-line verification indexterm:[on-line device verification]After you have enabled on-line verification, you will be able to initiate a verification run using the following command: ---------------------------- # drbdadm verify :/ ---------------------------- When you do so, DRBD starts an online verification run for __ to _ in __, and if it detects any blocks that are not in sync, will mark those blocks as such and write a message to the kernel log. Any applications using the device at that time can continue to do so unimpeded, and you may also <> at will. __ is optional, if ommited, it will verify all volumes in that resource. If out-of-sync blocks were detected during the verification run, you may resynchronize them using the following commands after verification has completed. Since drbd-9.0.29 the preferred way is one of these commands: ---------------------------- # drbdadm invalidate :/volume --reset-bitmap=no # drbdadm invalidate-remote :/volume --reset-bitmap=no ---------------------------- The first command will cause the local differences to be overwritten by the remote version. The second command does it in the opposite direction. Before drbd-9.0.29 one needs to initiate a resync. A way to do that is disconnecting from a primary and ensuring that the primary changes at least one block while the peer is away. ---------------------------- # drbdadm disconnect : ## write one block on the primary # drbdadm connect : ---------------------------- [[s-online-verify-automate]] ==== Automating on-line verification indexterm:[on-line device verification]Most users will want to automate on-line device verification. This can be easily accomplished. Create a file with the following contents, named `/etc/cron.d/drbd-verify` on *one* of your nodes: [source,drbd] ---------------------------- 42 0 * * 0 root /sbin/drbdadm verify ---------------------------- This will have `cron` invoke a device verification every Sunday at 42 minutes past midnight; so, if you come into the office on Monday morning, a quick look at the resource's status would show the result. If your devices are very big, and the \~32 hours were not enough, then you'll notice _VerifyS_ or _VerifyT_ as connection state, meaning that the `verify` is still in progress. If you have enabled on-line verification for all your resources (for example, by adding `verify-alg __` to the `common` section in `/etc/drbd.d/global_common.conf`), you may also use: [source,drbd] ---------------------------- 42 0 * * 0 root /sbin/drbdadm verify all ---------------------------- [[s-configure-sync-rate]] === Configuring the rate of synchronization indexterm:[synchronization]Normally, one tries to ensure that background synchronization (which makes the data on the synchronization target temporarily inconsistent) completes as quickly as possible. However, it is also necessary to keep background synchronization from hogging all bandwidth otherwise available for foreground replication, which would be detrimental to application performance. Thus, you must configure the synchronization bandwidth to match your hardware -- which you may do in a permanent fashion or on-the-fly. IMPORTANT: It does not make sense to set a synchronization rate that is higher than the maximum write throughput on your secondary node. You must not expect your secondary node to miraculously be able to write faster than its I/O subsystem allows, just because it happens to be the target of an ongoing device synchronization. Likewise, and for the same reasons, it does not make sense to set a synchronization rate that is higher than the bandwidth available on the replication network. [[s-estimating_a_synchronization_speed]] ==== Estimating a synchronization speed TIP: A good rule of thumb for this value is to use about 30% of the available replication bandwidth. Thus, if you had an I/O subsystem capable of sustaining write throughput of 400MB/s, and a Gigabit Ethernet network capable of sustaining 110 MB/s network throughput (the network being the bottleneck), you would calculate: [[eq-sync-rate-example1]] .Syncer rate example, 110MB/s effective available bandwidth image::images/sync-rate-example1.svg[] Thus, the recommended value for the `rate` option would be `33M`. By contrast, if you had an I/O subsystem with a maximum throughput of 80MB/s and a Gigabit Ethernet connection (the I/O subsystem being the bottleneck), you would calculate: [[eq-sync-rate-example2]] .Syncer rate example, 80MB/s effective available bandwidth image::images/sync-rate-example2.svg[] In this case, the recommended value for the `rate` option would be `24M`. Similarly, for a storage speed of 800MB/s and a 10Gbe network connection, you would shoot for \~240MB/s synchronization rate. [[s-configure-sync-rate-variable]] ==== Variable sync rate configuration When multiple DRBD resources share a single replication/synchronization network, synchronization with a fixed rate may not be an optimal approach. So, in DRBD 8.4.0 the variable-rate synchronization was enabled by default. In this mode, DRBD uses an automated control loop algorithm to determine, and adjust, the synchronization rate. This algorithm ensures that there is always sufficient bandwidth available for foreground replication, greatly mitigating the impact that background synchronization has on foreground I/O. The optimal configuration for variable-rate synchronization may vary greatly depending on the available network bandwidth, application I/O pattern and link congestion. ifndef::drbd-only,de-brand[] Ideal configuration settings also depend on whether <> is in use or not. It may be wise to engage professional consultancy in order to optimally configure this DRBD feature. An _example_ configuration (which assumes a deployment in conjunction with DRBD Proxy) is provided below: [source,drbd] ---------------------------- resource { disk { c-plan-ahead 5; c-max-rate 10M; c-fill-target 2M; } } ---------------------------- endif::[] TIP: A good starting value for `c-fill-target` is _BDP * 2_, where _BDP_ is your bandwidth-delay-product on the replication link. For example, when using a 1GBit/s crossover connection, you'll end up with about 200µs latencyfootnote:[The rule-of-thumb is using the time reported by `ping`.]. + 1GBit/s means about 120MB/s; times 200*10^-6^ seconds gives 24000 Byte. Just round that value up to the next MB, and you're good to go. Another example: a 100MBit WAN connection with 200ms latency means 12MB/s times 0.2s, or about 2.5MB "on the wire". Here a good starting value for `c-fill-target` would be 3MB. Please see the `drbd.conf` manual page for more details on the other configuration items. [[s-configure-sync-rate-permanent]] ==== Permanent fixed sync rate configuration In a few, very restricted situationsfootnote:[Like benchmarking.], it might make sense to just use some fixed synchronization rate. In this case, first of all you need to turn the dynamic sync rate controller off, by using `c-plan-ahead 0;`. Then, the maximum bandwidth a resource uses for background re-synchronization is determined by the `resync-rate` option for a resource. This must be included in the resource's `disk` section in `/etc/drbd.conf`: [source,drbd] ---------------------------- resource disk { resync-rate 40M; ... } ... } ---------------------------- Note that the rate setting is given in _bytes_, not _bits_ per second; the default unit is _Kibibyte_, so a value of `4096` would be interpreted as `4MiB`. NOTE: This just defines a rate that DRBD tries to achieve. If there is a bottleneck with lower throughput (network, storage speed), the defined speed (aka the "wished-for" performance ;) won't be reached. [[s-some_more_hints_about_synchronization]] ==== Some more hints about synchronization When some amount of the to-be-synchronized data isn't really in use anymore (e.g. because files got deleted while one node wasn't connected), you might benefit from the <>. Furthermore, `c-min-rate` is easy to misunderstand - it doesn't define a *minimum* synchronization speed, but rather a limit below which DRBD will not slow down further *on purpose*. + Whether you manage to *reach* that synchronization rate depends on your network and storage speed, network latency (which might be highly variable for shared links), and application IO (which not might be able to do anything about). [[s-configure-checksum-sync]] === Configuring checksum-based synchronization indexterm:[checksum-based synchronization]<> is not enabled for resources by default. To enable it, add the following lines to your resource configuration in `/etc/drbd.conf`: [source,drbd] ---------------------------- resource net { csums-alg ; } ... } ---------------------------- __ may be any message digest algorithm supported by the kernel crypto API in your system's kernel configuration. Normally, you should be able to choose at least from `sha1`, `md5`, and `crc32c`. If you make this change to an existing resource, as always, synchronize your `drbd.conf` to the peer, and run `drbdadm adjust __` on both nodes. [[s-configure-congestion-policy]] === Configuring congestion policies and suspended replication In an environment where the replication bandwidth is highly variable (as would be typical in WAN replication setups), the replication link may occasionally become congested. In a default configuration, this would cause I/O on the primary node to block, which is sometimes undesirable. Instead, you may configure DRBD to _suspend_ the ongoing replication in this case, causing the Primary's data set to _pull ahead_ of the Secondary. In this mode, DRBD keeps the replication channel open -- it never switches to disconnected mode -- but does not actually replicate until sufficient bandwidth becomes available again. The following example is for a DRBD Proxy configuration: [source,drbd] ---------------------------- resource { net { on-congestion pull-ahead; congestion-fill 2G; congestion-extents 2000; ... } ... } ---------------------------- It is usually wise to set both `congestion-fill` and `congestion-extents` together with the `pull-ahead` option. A good value for `congestion-fill` is 90% * of the allocated DRBD proxy buffer memory, when replicating over DRBD Proxy, or * of the TCP network send buffer, in non-DRBD Proxy setups. A good value for `congestion-extents` is 90% of your configured `al-extents` for the affected resources. [[s-configure-io-error-behavior]] === Configuring I/O error handling strategies indexterm:[I/O errors]indexterm:[drbd.conf]DRBD's <> is determined by the `on-io-error` option, included in the resource `disk` configuration in `/etc/drbd.conf`: [source,drbd] ---------------------------- resource { disk { on-io-error ; ... } ... } ---------------------------- You may, of course, set this in the `common` section too, if you want to define a global I/O error handling policy for all resources. __ may be one of the following options: .`detach` This is the default and recommended option. On the occurrence of a lower-level I/O error, the node drops its backing device, and continues in diskless mode. .`pass-on` This causes DRBD to report the I/O error to the upper layers. On the primary node, it is reported to the mounted file system. On the secondary node, it is ignored (because the secondary has no upper layer to report to). .`call-local-io-error` Invokes the command defined as the local I/O error handler. This requires that a corresponding `local-io-error` command invocation is defined in the resource's `handlers` section. It is entirely left to the administrator's discretion to implement I/O error handling using the command (or script) invoked by `local-io-error`. NOTE: Early DRBD versions (prior to 8.0) included another option, `panic`, which would forcibly remove the node from the cluster by way of a kernel panic, whenever a local I/O error occurred. While that option is no longer available, the same behavior may be mimicked via the `local-io-error`/`call-local-io-error` interface. You should do so only if you fully understand the implications of such behavior. You may reconfigure a running resource's I/O error handling strategy by following this process: * Edit the resource configuration in `/etc/drbd.d/____.res`. * Copy the configuration to the peer node. * Issue `drbdadm adjust __` on both nodes. [[s-configure-integrity-check]] === Configuring replication traffic integrity checking indexterm:[replication traffic integrity checking]<> is not enabled for resources by default. To enable it, add the following lines to your resource configuration in `/etc/drbd.conf`: [source,drbd] ---------------------------- resource net { data-integrity-alg ; } ... } ---------------------------- __ may be any message digest algorithm supported by the kernel crypto API in your system's kernel configuration. Normally, you should be able to choose at least from `sha1`, `md5`, and `crc32c`. If you make this change to an existing resource, as always, synchronize your `drbd.conf` to the peer, and run `drbdadm adjust __` on both nodes. WARNING: This feature is not intended for production use. Enable only if you need to diagnose data corruption problems, and want to see whether the transport path (network hardware, drivers, switches) might be at fault! [[s-resizing]] === Resizing resources When growing (extending) DRBD volumes, you need to grow from bottom to top. You need to extend the backing block devices on all nodes first. Then you can tell DRBD to use the new space. Once the DRBD volume is extended, you need to still propagate that into whatever is using DRBD: extend the file system, or make a VM running with this volume attached aware of the new "disk size". That all typically boils down to ----------------------- # # on all nodes, resize the backing LV: # lvextend -L +${additional_gb}g VG/LV # # on one node: # drbdadm resize ${resource_name}/${volume_number} # # on the Primary only: # # resize the file system using the file system specific tool, see below ----------------------- See also the next sections <> and following. Note that different file systems have different capabilities and different sets of management tools. For example XFS can only grow. You point its tool to the active mount point: `xfs_growfs /where/you/have/it/mounted`. While the EXT family can both grow (even online), and also shrink (only offline; you have to unmount it first). To resize an ext3 or ext4, you would point the tool not to the mount point, but to the (mounted) block device: `resize2fs /dev/drbd#` Obviously use the correct DRBD (as displayed by `mount` or `df -T`, while mounted), and *not* the backing block device. As long as DRBD is up, that's not supposed to work anyways (`resize2fs: Device or resource busy while trying to open /dev/mapper/VG-LV Couldn't find valid filesystem superblock.`). If you tried to do that offline (with DRBD stopped), you may corrupt DRBD meta data if you ran the file system tools directly against the backing LV or partition. So don't. You do the file system resize only *once* on the Primary, against the active DRBD device. DRBD replicates the changes to the file system structure. That is what you have it for. Also, don't use resize2fs on XFS volumes, or XFS tools on EXT, or ... but the right tool for the file system in use. `resize2fs: Bad magic number in super-block while trying to open /dev/drbd7` is probably just trying to tell you that this is *not* an EXT file system, and you should try an other tool instead. Maybe `xfs_growfs`? But as mentioned, that does not take the block device, but the mount point as argument. When shrinking (reducing) DRBD volumes, you need to shrink from top to bottom. So first make sure no one is using the space you want to cut off. Next, shrink the file system (if your file system supports that). Then tell DRBD to stop using that space, which is not so easy with DRBD internal meta data, because they are by design "at the end" of the backing device. Once you are sure that DRBD won't use the space anymore either, you can cut it off from the backing device, for example using `lvreduce`. See also <>, <>. [[s-growing-online]] ==== Growing on-line indexterm:[resource]If the backing block devices can be grown while in operation (online), it is also possible to increase the size of a DRBD device based on these devices during operation. To do so, two criteria must be fulfilled: . The affected resource's backing device must be one managed by a logical volume management subsystem, such as LVM. . The resource must currently be in the _Connected_ connection state. Having grown the backing block devices on all nodes, ensure that only one node is in primary state. Then enter on one node: ---------------------------- # drbdadm resize ---------------------------- This triggers a synchronization of the new section. The synchronization is done from the primary node to the secondary node. If the space you're adding is clean, you can skip syncing the additional space by using the --assume-clean option. ---------------------------- # drbdadm -- --assume-clean resize ---------------------------- [[s-growing-offline]] ==== Growing off-line indexterm:[resource]When the backing block devices on both nodes are grown while DRBD is inactive, and the DRBD resource is using <>, then the new size is recognized automatically. No administrative intervention is necessary. The DRBD device will have the new size after the next activation of DRBD on both nodes and a successful establishment of a network connection. If however the DRBD resource is configured to use <>, then this meta data must be moved to the end of the grown device before the new size becomes available. To do so, complete the following steps: WARNING: This is an advanced procedure. Use at your own discretion. * Unconfigure your DRBD resource: [source,drbd] ---------------------------- # drbdadm down ---------------------------- * Save the meta data in a text file prior to resizing: ---------------------------- # drbdadm dump-md > /tmp/metadata ---------------------------- You must do this on both nodes, using a separate dump file for every node. *Do not* dump the meta data on one node, and simply copy the dump file to the peer. *This. will. not. work.* * Grow the backing block device on both nodes. * Adjust the size information (`la-size-sect`) in the file `/tmp/metadata` accordingly, on both nodes. Remember that `la-size-sect` must be specified in sectors. * Re-initialize the metadata area: ---------------------------- # drbdadm create-md ---------------------------- * Re-import the corrected meta data, on both nodes: ---------------------------- # drbdmeta_cmd=$(drbdadm -d dump-md ) # ${drbdmeta_cmd/dump-md/restore-md} /tmp/metadata Valid meta-data in place, overwrite? [need to type 'yes' to confirm] yes Successfully restored meta data ---------------------------- NOTE: This example uses `bash` parameter substitution. It may or may not work in other shells. Check your `SHELL` environment variable if you are unsure which shell you are currently using. * Re-enable your DRBD resource: ---------------------------- # drbdadm up ---------------------------- * On one node, promote the DRBD resource: ---------------------------- # drbdadm primary ---------------------------- * Finally, grow the file system so it fills the extended size of the DRBD device. [[s-shrinking-online]] ==== Shrinking on-line WARNING: Online shrinking is only supported with external metadata. indexterm:[resource]Before shrinking a DRBD device, you _must_ shrink the layers above DRBD, i.e. usually the file system. Since DRBD cannot ask the file system how much space it actually uses, you have to be careful in order not to cause data loss. NOTE: Whether or not the _filesystem_ can be shrunk on-line depends on the filesystem being used. Most filesystems do not support on-line shrinking. XFS does not support shrinking at all. To shrink DRBD on-line, issue the following command _after_ you have shrunk the file system residing on top of it: [source,drbd] ---------------------------- # drbdadm resize --size= ---------------------------- You may use the usual multiplier suffixes for __ (K, M, G etc.). After you have shrunk DRBD, you may also shrink the containing block device (if it supports shrinking). NOTE: It might be a good idea to issue `drbdadm resize __` after resizing the lower level device, so that the DRBD metadata *really* gets written into the expected space at the end of the volume. [[s-shrinking-offline]] ==== Shrinking off-line indexterm:[resource]If you were to shrink a backing block device while DRBD is inactive, DRBD would refuse to attach to this block device during the next attach attempt, since it is now too small (in case external meta data is used), or it would be unable to find its meta data (in case internal meta data is used). To work around these issues, use this procedure (if you cannot use <>): WARNING: This is an advanced procedure. Use at your own discretion. * Shrink the file system from one node, while DRBD is still configured. * Unconfigure your DRBD resource: ---------------------------- # drbdadm down ---------------------------- * Save the meta data in a text file prior to shrinking: ---------------------------- # drbdadm dump-md > /tmp/metadata ---------------------------- You must do this on both nodes, using a separate dump file for every node. *Do not* dump the meta data on one node, and simply copy the dump file to the peer. *This. will. not. work.* * Shrink the backing block device on both nodes. * Adjust the size information (`la-size-sect`) in the file `/tmp/metadata` accordingly, on both nodes. Remember that `la-size-sect` must be specified in sectors. * *Only if you are using internal metadata* (which at this time have probably been lost due to the shrinking process), re-initialize the metadata area: + ---------------------------- # drbdadm create-md ---------------------------- * Re-import the corrected meta data, on both nodes: + ---------------------------- # drbdmeta_cmd=$(drbdadm -d dump-md ) # ${drbdmeta_cmd/dump-md/restore-md} /tmp/metadata Valid meta-data in place, overwrite? [need to type 'yes' to confirm] yes Successfully restored meta data ---------------------------- NOTE: This example uses `bash` parameter substitution. It may or may not work in other shells. Check your `SHELL` environment variable if you are unsure which shell you are currently using. * Re-enable your DRBD resource: + ---------------------------- # drbdadm up ---------------------------- [[s-disable-flushes]] === Disabling backing device flushes CAUTION: You should only disable device flushes when running DRBD on devices with a battery-backed write cache (BBWC). Most storage controllers allow to automatically disable the write cache when the battery is depleted, switching to write-through mode when the battery dies. It is strongly recommended to enable such a feature. Disabling DRBD's flushes when running without BBWC, or on BBWC with a depleted battery, is _likely to cause data loss_ and should not be attempted. DRBD allows you to enable and disable <> separately for the replicated data set and DRBD's own meta data. Both of these options are enabled by default. If you wish to disable either (or both), you would set this in the `disk` section for the DRBD configuration file, `/etc/drbd.conf`. To disable disk flushes for the replicated data set, include the following line in your configuration: [source,drbd] ---------------------------- resource disk { disk-flushes no; ... } ... } ---------------------------- To disable disk flushes on DRBD's meta data, include the following line: [source,drbd] ---------------------------- resource disk { md-flushes no; ... } ... } ---------------------------- After you have modified your resource configuration (and synchronized your `/etc/drbd.conf` between nodes, of course), you may enable these settings by issuing this command on both nodes: ---------------------------- # drbdadm adjust ---------------------------- In case only one of the serves has a BBWCfootnote:[For example, in the DR site you might be using different hardware, right?], you should move the setting into a host section, like this: [source,drbd] ---------------------------- resource { disk { ... common settings ... } on host-1 { disk { md-flushes no; } ... } ... } ---------------------------- [[s-configure-split-brain-behavior]] === Configuring split brain behavior [[s-split-brain-notification]] ==== Split brain notification DRBD invokes the `split-brain` handler, if configured, at any time split brain is _detected_. To configure this handler, add the following item to your resource configuration: ---------------------------- resource handlers { split-brain ; ... } ... } ---------------------------- __ may be any executable present on the system. The DRBD distribution contains a split brain handler script that installs as `/usr/lib/drbd/notify-split-brain.sh`. It simply sends a notification e-mail message to a specified address. To configure the handler to send a message to `root@localhost` (which is expected to be an email address that forwards the notification to a real system administrator), configure the `split-brain handler` as follows: ---------------------------- resource handlers { split-brain "/usr/lib/drbd/notify-split-brain.sh root"; ... } ... } ---------------------------- After you have made this modification on a running resource (and synchronized the configuration file between nodes), no additional intervention is needed to enable the handler. DRBD will simply invoke the newly-configured handler on the next occurrence of split brain. [[s-automatic-split-brain-recovery-configuration]] ==== Automatic split brain recovery policies CAUTION: Configuring DRBD to automatically resolve data divergence situations resulting from split-brain (or other) scenarios is configuring for potential *automatic data loss*. Understand the implications, and don't do it if you don't mean to. TIP: You rather want to look into fencing policies, quorum settings, cluster manager integration, and redundant cluster manager communication links to *avoid* data divergence in the first place. In order to be able to enable and configure DRBD's automatic split brain recovery policies, you must understand that DRBD offers several configuration options for this purpose. DRBD applies its split brain recovery procedures based on the number of nodes in the Primary role at the time the split brain is detected. To that end, DRBD examines the following keywords, all found in the resource's `net` configuration section: .`after-sb-0pri` Split brain has just been detected, but at this time the resource is not in the Primary role on any host. For this option, DRBD understands the following keywords: * `disconnect`: Do not recover automatically, simply invoke the `split-brain` handler script (if configured), drop the connection and continue in disconnected mode. * `discard-younger-primary`: Discard and roll back the modifications made on the host which assumed the Primary role last. * `discard-least-changes`: Discard and roll back the modifications on the host where fewer changes occurred. * `discard-zero-changes`: If there is any host on which no changes occurred at all, simply apply all modifications made on the other and continue. .`after-sb-1pri` Split brain has just been detected, and at this time the resource is in the Primary role on one host. For this option, DRBD understands the following keywords: * `disconnect`: As with `after-sb-0pri`, simply invoke the `split-brain` handler script (if configured), drop the connection and continue in disconnected mode. * `consensus`: Apply the same recovery policies as specified in `after-sb-0pri`. If a split brain victim can be selected after applying these policies, automatically resolve. Otherwise, behave exactly as if `disconnect` were specified. * `call-pri-lost-after-sb`: Apply the recovery policies as specified in `after-sb-0pri`. If a split brain victim can be selected after applying these policies, invoke the `pri-lost-after-sb` handler on the victim node. This handler must be configured in the `handlers` section and is expected to forcibly remove the node from the cluster. * `discard-secondary`: Whichever host is currently in the Secondary role, make that host the split brain victim. .`after-sb-2pri` Split brain has just been detected, and at this time the resource is in the Primary role on both hosts. This option accepts the same keywords as `after-sb-1pri` except `discard-secondary` and `consensus`. NOTE: DRBD understands additional keywords for these three options, which have been omitted here because they are very rarely used. Refer to the man page of `drbd.conf` for details on split brain recovery keywords not discussed here. For example, a resource which serves as the block device for a GFS or OCFS2 file system in dual-Primary mode may have its recovery policy defined as follows: ---------------------------- resource { handlers { split-brain "/usr/lib/drbd/notify-split-brain.sh root" ... } net { after-sb-0pri discard-zero-changes; after-sb-1pri discard-secondary; after-sb-2pri disconnect; ... } ... } ---------------------------- [[s-three-nodes]] === Creating a stacked three-node setup A three-node setup involves one DRBD device _stacked_ atop another. NOTE: Stacking is deprecated in DRBD version 9.x, as more nodes can be implemented on a single level. See <> for details. [[s-stacking-considerations]] ==== Device stacking considerations The following considerations apply to this type of setup: * The stacked device is the active one. Assume you have configured one DRBD device `/dev/drbd0`, and the stacked device atop it is `/dev/drbd10`, then `/dev/drbd10` will be the device that you mount and use. * Device meta data will be stored twice, on the underlying DRBD device _and_ the stacked DRBD device. On the stacked device, you must always use <>. This means that the effectively available storage area on a stacked device is slightly smaller, compared to an unstacked device. * To get the stacked upper level device running, the underlying device must be in the primary role. * To be able to synchronize the backup node, the stacked device on the active node must be up and in the primary role. [[s-three-node-config]] ==== Configuring a stacked resource In the following example, nodes are named 'alice', 'bob', and 'charlie', with 'alice' and 'bob' forming a two-node cluster, and 'charlie' being the backup node. [source,drbd] ---------------------------- resource r0 { protocol C; device /dev/drbd0; disk /dev/sda6; meta-disk internal; on alice { address 10.0.0.1:7788; } on bob { address 10.0.0.2:7788; } } resource r0-U { protocol A; stacked-on-top-of r0 { device /dev/drbd10; address 192.168.42.1:7789; } on charlie { device /dev/drbd10; disk /dev/hda6; address 192.168.42.2:7789; # Public IP of the backup node meta-disk internal; } } ---------------------------- As with any `drbd.conf` configuration file, this must be distributed across all nodes in the cluster -- in this case, three nodes. Notice the following extra keyword not found in an unstacked resource configuration: .`stacked-on-top-of` This option informs DRBD that the resource which contains it is a stacked resource. It replaces one of the `on` sections normally found in any resource configuration. Do not use `stacked-on-top-of` in an lower-level resource. NOTE: It is not a requirement to use <> for stacked resources. You may select any of DRBD's replication protocols depending on your application. [[f-single-stacked]] .Single stacked setup image::images/single-stacked.svg[] [[s-three-node-enable]] ==== Enabling stacked resources To enable a stacked resource, you first enable its lower-level resource and promote it: ---------------------------- drbdadm up r0 drbdadm primary r0 ---------------------------- As with unstacked resources, you must create DRBD meta data on the stacked resources. This is done using the following command: ---------------------------- # drbdadm create-md --stacked r0-U ---------------------------- Then, you may enable the stacked resource: ---------------------------- # drbdadm up --stacked r0-U # drbdadm primary --stacked r0-U ---------------------------- After this, you may bring up the resource on the backup node, enabling three-node replication: ---------------------------- # drbdadm create-md r0-U # drbdadm up r0-U ---------------------------- In order to automate stacked resource management, you may integrate stacked resources in your cluster manager configuration. ifdef::drbd-only[] See <> for information on doing this in a cluster managed by the Pacemaker cluster management framework. endif::drbd-only[] [[s-permanently-diskless-nodes]] === Permanently diskless nodes A node might be permanently diskless in DRBD. Here is a configuration example showing a resource with 3 diskful nodes (servers) and one permanently diskless node (client). ------------------------------------- resource kvm-mail { device /dev/drbd6; disk /dev/vg/kvm-mail; meta-disk internal; on store1 { address 10.1.10.1:7006; node-id 0; } on store2 { address 10.1.10.2:7006; node-id 1; } on store3 { address 10.1.10.3:7006; node-id 2; } on for-later-rebalancing { address 10.1.10.4:7006; node-id 3; } # DRBD "client" floating 10.1.11.6:8006 { disk none; node-id 4; } # rest omitted for brevity ... } ------------------------------------- For permanently diskless nodes no bitmap slot gets allocated. For such nodes the diskless status is displayed in green color since it is not an error or unexpected state. See <> for internal details. [[s-rebalance-workflow]] === Data rebalancing indexterm:[rebalance]Given the (example) policy that data needs to be available on 3 nodes, you need at least 3 servers for your setup. Now, as your storage demands grow, you will encounter the need for additional servers. Rather than having to buy 3 more servers at the same time, you can _rebalance_ your data across a single additional node. .DRBD data rebalancing image::images/rebalance.svg[] In the figure above you can see the _before_ and _after_ states: from 3 nodes with three 25TiB volumes each (for a net 75TiB), to 4 nodes, with net 100TiB. To redistribute the data across your cluster you have to choose a _new_ node, and one where you want to remove this DRBD resource. + Please note that removing the resource from a currently _active_ node (i.e. where DRBD is _Primary_) will involve either migrating the service or running this resource on this node as a <>; it's easier to choose a node in _Secondary_ role. (Of course, that might not always be possible.) [[s-prepare_a_bitmap_slot]] ==== Prepare a bitmap slot You will need to have a free <> for temporary use, on each of the nodes that have the resource that is to be moved. You can allocate one more at <>, or simply put a placeholder in your configuration, so that `drbdadm` sees that it should reserve one more slot: --------- resource r0 { ... on for-later-rebalancing { address 10.254.254.254:65533; node-id 3; } } --------- [NOTE] ================= If you need to make that slot available during live use, you will have to . dump the metadata, . enlarge the metadata space, . edit the dumpfile, . load the changed metadata. In a future version `drbdadm` will have a shortcut for you; most probably you'll be able to say `drbdadm resize --peers _N_` and have the kernel rewrite the metadata for you. ================= [[s-preparing_and_activating_the_new_node]] ==== Preparing and activating the new node First of all you have to create the underlying storage volume on the new node (using e.g. `lvcreate`). Then the placeholder in the configuration can be filled with the correct host name, address, and storage path. Now copy the resource configuration to all relevant nodes. On the new node initialize the meta-data (once) by doing ------------- # drbdadm create-md v09 Magic number not found Writing meta data... initialising activity log NOT initializing bitmap New drbd meta data block successfully created. ------------- [[s-starting_the_initial_sync]] ==== Starting the initial sync Now the new node needs to get the data. This is done by defining the network connection on the existing nodes via ------------------------------------- # drbdadm adjust ------------------------------------- and starting the DRBD device on the new node via ------------------------------------- # drbdadm up ------------------------------------- [[s-check_connectivity]] ==== Check connectivity At this time please do a ------------ # drbdadm status ------------ on the new node, and check that _all_ other nodes are connected. [[s-after_the_initial_sync]] ==== After the initial sync As soon as the new host is _UpToDate_, one of the other nodes in the configuration can be renamed to `for-later-rebalancing`, and kept for another migration. NOTE: Perhaps you want to comment the section; although that has the risk that doing a `drbdadm create-md` for a new node has too few bitmap slots for the next rebalancing. + It might be easier to use a reserved (unused) IP address and host name. Copy the changed configuration around again, and use it by running ------------------------------------- # drbdadm adjust ------------------------------------- on all nodes. [[s-cleaning_up]] ==== Cleaning up On the one node that had the data up to now, but isn't used anymore for this resource, you can now take the DRBD device down by starting ------------------- # drbdadm down ------------------- Now the lower level storage device isn't used anymore, and can either be re-used for other purposes or, if it is a logical volume, its space can be returned to the volume group via `lvremove`. [[s-conclusion_and_further_steps]] ==== Conclusion and further steps One of the resources has been migrated to the new node. The same could be done for one or more other resources, to free space on two or three nodes in the existing cluster. Then new resources can be configured, as there are enough nodes with free space to achieve 3-way redundancy again. [[s-configuring-quorum]] === Configuring quorum In order to avoid split brain or diverging data of replicas one has to configure fencing. All the options for fencing rely on redundant communication in the end. That might be in the form of a management network that connects the nodes to the IPMI network interfaces of the peer machines. In case of the crm-fence-peer script it is necessary that Pacemakers communication stays available when DRBD's network link breaks. The quorum mechanism on the other hand takes a completely difference approach. The basic idea is that a cluster partition may only modify the replicated data set if the number of nodes that can communicate is greater than the half of the overall number of nodes. A node of such a partition _has quorum_. On the other hand a node does not have quorum needs to guarantee that the replicated data set it not touched, that it does not create a diverging data set. The quorum implementation in DRBD gets enabled by setting the `quorum` resource option to `majority`, `all` or a numeric value. Where `majority` selects the behavior that was described in the previous paragraph. [[s-guaranteed_minimal_redundancy]] ==== Guaranteed minimal redundancy By default every node with a disk gets a vote in the quorum election. In other words only diskless nodes do not count. So a partition with two _Inconsistent_ disks gets quorum, while a partition with one _UpToDate_ node will have quorum in a 3 node cluster. By configuring `quorum-minimum-redundancy` this behavior can be changed so that only nodes that are `UpToDate` have a vote in the quorum election. The option takes the same arguments as the `quorum` option. With this option you express that you rather want to wait until eventually necessary resync operations finish before any services start. So in a way you prefer that the minimal redundancy of your data is guaranteed over the availability of your service. Financial data and services is an example that comes to mind. Consider following example for a 5 node cluster. It requires a partitions to have at least 3 nodes, and two of them must be `UpToDate`: ------------------------------------- resource quorum-demo { options { quorum majority; quorum-minimum-redundancy 2; ... } } ------------------------------------- [[s-actions_on_loss_of_quorum]] ==== Actions on loss of quorum When a node that is running the service loses quorum it needs to cease write-operations on the data set immediately. That means that IO immediately starts to complete all IO requests with errors. Usually that means that a graceful shutdown is not possible, since that would require more modifications to the data set. The IO errors propagate from the block level to the file system and from the file system to the user space application(s). Ideally the application simply terminates in case of IO errors. This allows then Pacemaker to unmount the filesystem and to demote the DRBD resource to secondary role. If that is true you should set the `on-no-quorum` resource option to `io-error`. Here is an example: ------------------------------------- resource quorum-demo { options { quorum majority; on-no-quorum io-error; ... } } ------------------------------------- If you application does not terminate on the first IO error, you can choose to freeze IO instead and to reboot the node. Here is a configuration example: ------------------------------------- resource quorum-demo { options { quorum majority; on-no-quorum suspend-io; ... } handlers { quorum-lost "echo b > /proc/sysrq-trigger"; } ... } ------------------------------------- [[s-configuring-quorum-tiebreaker]] ==== Using a diskless node as a tiebreaker A diskless node with connections to all nodes in a cluster can be utilized to break ties in the quorum negotiation process. Consider the following two-node cluster, where node A is the primary and node B is a secondary: image::images/quorum-tiebreaker-without.svg[] As soon as the connection between the two nodes is interrupted, they lose quorum and the application on top of the cluster cannot write data anymore. image::images/quorum-tiebreaker-without-disconnect.svg[] Now if we add a third node, C, to the cluster and configure it as diskless, we can take advantage of the tiebreaker mechanism. image::images/quorum-tiebreaker.svg[] In this case, when the primary and secondary nodes lose connection, they can still "see" the diskless tiebreaker. Because of this, the primary can continue working, while the secondary demotes its disk to Outdated and the service can therefore not be migrated there. image::images/quorum-tiebreaker-disconnect.svg[] There are a few special cases in case two connections fail. Consider the following scenario: image::images/quorum-tiebreaker-disconnect-case2.svg[] In this case, the tiebreaker node forms a partition with the primary node. The primary therefore keeps quorum, while the secondary becomes outdated. Note that the secondary's disk state will be "UpToDate", but regardless it cannot be promoted to primary because it lacks quorum. Let's consider the possibility of the primary losing connection to the tiebreaker: image::images/quorum-tiebreaker-disconnect-case3.svg[] In this case, the primary becomes unusable and goes into a "quorum suspended" state. This effectively results in the application on top of DRBD receiving I/O errors. A cluster manager could then promote node B to primary and keep the service running there instead. In case the diskless tiebreaker "switches sides", we need to avoid data divergence. Consider this scenario: image::images/quorum-tiebreaker-disconnect-case1a.svg[] The connection between the primary and secondary has failed, and the application is continuing to run on the primary, when the primary suddenly loses its connection to the diskless node. In this case, no node can be promoted to primary and the cluster cannot continue to operate. [NOTE] ==================== Protecting from data divergence always takes priority over ensuring service availability. ==================== Let's look at another scenario: image::images/quorum-tiebreaker-disconnect-case2a.svg[] Here, the application is running on the primary, while the secondary is unavailable. Then, the tiebreaker first loses connection to the primary, and then reconnects to the secondary. It is important to note here that *a node that has lost quorum cannot regain quorum by connecting to a diskless node*. Thus, in this case, no node has quorum and the cluster halts. [[s-quorum_last_man_standing]] ==== Last man standing It needs to be mentioned that nodes that leave a cluster gracefully are counted different than failed nodes. In this context _leaving gracefully_ means that the leaving node marked it's data as Outdated, and that it was able to tell the remaining nodes that it's data is Outdated. In a group of nodes where all disks are Outdated no one in that group can promote to Primary footnote:[The exception is a manual promotion with the --force flag. We assume that one that uses the --force knows what he is doing.]. An implication is that, if a cluster is left by all but one node, it can keep quorum as long as all the others left in a graceful way. If a single node leaves ungraceful, it has to assume that all away nodes form a partition and have access to up-to-date data. If the other partition might (potentially) be larger than my own partition it looses quorum. [[s-removing-DRBD]] === Removing DRBD For the unlikely case that you want to remove DRBD, here are the necessary steps. . Stop the services and unmount the filesystems on top of the DRBD volumes. In case you are using a cluster manager, make sure it ceases to control the services fist. . Stop the DRBD resource(s) by using `drbdadm down ` or `drbdadm down all` .. In case the DRBD resource was using internal meta-data you might choose to resize the filesystem to cover all of the backing device's space. This step effectively removes DRBD's meta-data. This is an action that can not be reversed easily. You can do that with `resize2fs ` for ext[234] family of file systems. It supports resizing of unmounted filesystem and under certain conditions also online grow. XFS can be grown online only with the `xfs_growfs` command. . Mount the backing device(s) directly, start the services on top of them . Unload the drbd kernel driver modules with `rmmod drbd_transport_tcp` and `rmmod drbd`. . Uninstall the drbd software packages. //- list divider. Keep it otherwise next included section is interpreted as list element