| 1 |
ifdef::env-github[] |
| 2 |
:tip-caption: :bulb: |
| 3 |
:note-caption: :information_source: |
| 4 |
:important-caption: :heavy_exclamation_mark: |
| 5 |
:caution-caption: :fire: |
| 6 |
:warning-caption: :warning: |
| 7 |
endif::[] |
| 8 |
|
| 9 |
|
| 10 |
[[ch-admin-manual]] |
| 11 |
== Common administrative tasks |
| 12 |
|
| 13 |
This chapter outlines typical administrative tasks encountered during |
| 14 |
day-to-day operations. It does not cover troubleshooting tasks, these |
| 15 |
are covered in detail in <<ch-troubleshooting>>. |
| 16 |
|
| 17 |
include::configure.adoc[] |
| 18 |
|
| 19 |
[[s-check-status]] |
| 20 |
=== Checking DRBD status |
| 21 |
|
| 22 |
[[s-drbd-drbdmon]] |
| 23 |
==== Retrieving status with drbdmon |
| 24 |
|
| 25 |
One convenient way to look at DRBD's status is the |
| 26 |
`drbdmon` utility. It updates the state of DRBD resources in real time. |
| 27 |
|
| 28 |
[[s-proc-drbd]] |
| 29 |
==== Status information in `/proc/drbd` |
| 30 |
|
| 31 |
NOTE: `/proc/drbd` is deprecated. While it won't be removed in the 8.4 |
| 32 |
series, we recommend to switch to other means, like <<s-drbdadm-status>>; or, |
| 33 |
for monitoring even more convenient, <<s-drbdsetup-events2>>. |
| 34 |
|
| 35 |
indexterm:[/proc/drbd]`/proc/drbd` is a virtual file displaying |
| 36 |
basic information about the DRBD module. |
| 37 |
It was used extensively up to DRBD 8.4, but couldn't keep up |
| 38 |
with the amount of information provided by DRBD 9. |
| 39 |
|
| 40 |
---------------------------- |
| 41 |
$ cat /proc/drbd |
| 42 |
version: 9.0.0 (api:1/proto:86-110) |
| 43 |
GIT-hash: XXX build by [email protected], 2011-10-12 09:07:35 |
| 44 |
---------------------------- |
| 45 |
|
| 46 |
The first line, prefixed with `version:`, shows the DRBD version used |
| 47 |
on your system. The second line contains information about this |
| 48 |
specific build. |
| 49 |
|
| 50 |
|
| 51 |
[[s-drbdadm-status]] |
| 52 |
==== Status information via `drbdadm` |
| 53 |
|
| 54 |
indexterm:[drbdadm, status]In its simplest invocation, we just ask for the |
| 55 |
status of a single resource. |
| 56 |
|
| 57 |
---------------------------- |
| 58 |
# drbdadm status home |
| 59 |
home role:Secondary |
| 60 |
disk:UpToDate |
| 61 |
nina role:Secondary |
| 62 |
disk:UpToDate |
| 63 |
nino role:Secondary |
| 64 |
disk:UpToDate |
| 65 |
nono connection:Connecting |
| 66 |
---------------------------- |
| 67 |
|
| 68 |
This here just says that the resource _home_ is locally, on 'nina', and |
| 69 |
on 'nino' _UpToDate_ and _Secondary_; so the three nodes have the same |
| 70 |
data on their storage devices, and nobody is using the device currently. |
| 71 |
|
| 72 |
The node 'nono' is not connected, its state is reported as |
| 73 |
_Connecting_; please see <<s-connection-states>> below for more details. |
| 74 |
|
| 75 |
|
| 76 |
You can get more information by passing the `--verbose` and/or |
| 77 |
`--statistics` arguments to `drbdsetup` (lines broken for readability): |
| 78 |
|
| 79 |
---------------------------- |
| 80 |
# drbdsetup status home --verbose --statistics |
| 81 |
home node-id:1 role:Secondary suspended:no |
| 82 |
write-ordering:none |
| 83 |
volume:0 minor:0 disk:UpToDate |
| 84 |
size:1048412 read:0 written:1048412 al-writes:0 bm-writes:48 upper-pending:0 |
| 85 |
lower-pending:0 al-suspended:no blocked:no |
| 86 |
nina local:ipv4:10.9.9.111:7001 peer:ipv4:10.9.9.103:7010 node-id:0 |
| 87 |
connection:Connected role:Secondary |
| 88 |
congested:no |
| 89 |
volume:0 replication:Connected disk:UpToDate resync-suspended:no |
| 90 |
received:1048412 sent:0 out-of-sync:0 pending:0 unacked:0 |
| 91 |
nino local:ipv4:10.9.9.111:7021 peer:ipv4:10.9.9.129:7012 node-id:2 |
| 92 |
connection:Connected role:Secondary |
| 93 |
congested:no |
| 94 |
volume:0 replication:Connected disk:UpToDate resync-suspended:no |
| 95 |
received:0 sent:0 out-of-sync:0 pending:0 unacked:0 |
| 96 |
nono local:ipv4:10.9.9.111:7013 peer:ipv4:10.9.9.138:7031 node-id:3 |
| 97 |
connection:Connecting |
| 98 |
---------------------------- |
| 99 |
|
| 100 |
|
| 101 |
Every few lines in this example form a block that is repeated |
| 102 |
for every node used in this resource, with small format exceptions |
| 103 |
for the local node - see below for more details. |
| 104 |
|
| 105 |
The first line in each block shows the `node-id` (for the current |
| 106 |
resource; a host can have different ``node-id``s in different resources). |
| 107 |
Furthermore the `role` (see <<s-roles>>) is shown. |
| 108 |
|
| 109 |
The next important line begins with the `volume` specification; normally |
| 110 |
these are numbered starting by zero, but the configuration may specify |
| 111 |
other IDs as well. This line shows the indexterm:[connection state] |
| 112 |
connection state in the |
| 113 |
`replication` item (see <<s-connection-states>> for details) and the |
| 114 |
remote indexterm:[disk state] disk state in `disk` (see <<s-disk-states>>). |
| 115 |
Then there's a line for this volume giving a bit of statistics - |
| 116 |
data `received`, `sent`, `out-of-sync`, etc; please see |
| 117 |
<<s-performance-indicators>> and <<s-conn-info>> for more information. |
| 118 |
|
| 119 |
For the local node the first line shows the resource name, _home_, in our |
| 120 |
example. As the first block always describes the local node, there is no _Connection_ or |
| 121 |
address information. |
| 122 |
|
| 123 |
please see the `drbd.conf` manual page for more information. |
| 124 |
|
| 125 |
The other four lines in this example form a block that is repeated for |
| 126 |
every DRBD device configured, prefixed by the device minor number. In |
| 127 |
this case, this is `0`, corresponding to the device `/dev/drbd0`. |
| 128 |
|
| 129 |
The resource-specific output contains various pieces |
| 130 |
of information about the resource: |
| 131 |
|
| 132 |
/////// |
| 133 |
//FIXME |
| 134 |
.Replication protocol |
| 135 |
The Replication protocol used by the resource. Either `A`, `B`, or `C`. See |
| 136 |
<<s-replication-protocols>> for details. |
| 137 |
/////// |
| 138 |
|
| 139 |
[[s-drbdsetup-events2]] |
| 140 |
==== One-shot or real time monitoring via `drbdsetup events2` |
| 141 |
|
| 142 |
NOTE: This is available only with userspace versions 8.9.3 and up. |
| 143 |
// and 8.4.6 |
| 144 |
|
| 145 |
This is a low-level mechanism to get information out of DRBD, suitable for use |
| 146 |
in automated tools, like monitoring. |
| 147 |
|
| 148 |
In its simplest invocation, showing only the current status, the output looks |
| 149 |
like this (but, when running on a terminal, will include colors): |
| 150 |
|
| 151 |
--------------- |
| 152 |
# drbdsetup events2 --now r0 |
| 153 |
exists resource name:r0 role:Secondary suspended:no |
| 154 |
exists connection name:r0 peer-node-id:1 conn-name:remote-host connection:Connected role:Secondary |
| 155 |
exists device name:r0 volume:0 minor:7 disk:UpToDate |
| 156 |
exists device name:r0 volume:1 minor:8 disk:UpToDate |
| 157 |
exists peer-device name:r0 peer-node-id:1 conn-name:remote-host volume:0 |
| 158 |
replication:Established peer-disk:UpToDate resync-suspended:no |
| 159 |
exists peer-device name:r0 peer-node-id:1 conn-name:remote-host volume:1 |
| 160 |
replication:Established peer-disk:UpToDate resync-suspended:no |
| 161 |
exists - |
| 162 |
--------------- |
| 163 |
|
| 164 |
Without the ''--now'', the process will keep running, and send continuous updates like this: |
| 165 |
|
| 166 |
----------------- |
| 167 |
# drbdsetup events2 r0 |
| 168 |
... |
| 169 |
change connection name:r0 peer-node-id:1 conn-name:remote-host connection:StandAlone |
| 170 |
change connection name:r0 peer-node-id:1 conn-name:remote-host connection:Unconnected |
| 171 |
change connection name:r0 peer-node-id:1 conn-name:remote-host connection:Connecting |
| 172 |
----------------- |
| 173 |
|
| 174 |
Then, for monitoring purposes, there's another argument ''--statistics'', that |
| 175 |
will produce some performance counters and other facts: |
| 176 |
|
| 177 |
'drbdsetup' **verbose output** (lines broken for readability): |
| 178 |
|
| 179 |
----------------- |
| 180 |
# drbdsetup events2 --statistics --now r0 |
| 181 |
exists resource name:r0 role:Secondary suspended:no write-ordering:drain |
| 182 |
exists connection name:r0 peer-node-id:1 conn-name:remote-host connection:Connected |
| 183 |
role:Secondary congested:no |
| 184 |
exists device name:r0 volume:0 minor:7 disk:UpToDate size:6291228 read:6397188 |
| 185 |
written:131844 al-writes:34 bm-writes:0 upper-pending:0 lower-pending:0 |
| 186 |
al-suspended:no blocked:no |
| 187 |
exists device name:r0 volume:1 minor:8 disk:UpToDate size:104854364 read:5910680 |
| 188 |
written:6634548 al-writes:417 bm-writes:0 upper-pending:0 lower-pending:0 |
| 189 |
al-suspended:no blocked:no |
| 190 |
exists peer-device name:r0 peer-node-id:1 conn-name:remote-host volume:0 |
| 191 |
replication:Established peer-disk:UpToDate resync-suspended:no received:0 |
| 192 |
sent:131844 out-of-sync:0 pending:0 unacked:0 |
| 193 |
exists peer-device name:r0 peer-node-id:1 conn-name:remote-host volume:1 |
| 194 |
replication:Established peer-disk:UpToDate resync-suspended:no received:0 |
| 195 |
sent:6634548 out-of-sync:0 pending:0 unacked:0 |
| 196 |
exists - |
| 197 |
----------------- |
| 198 |
|
| 199 |
You might also like the ''--timestamp'' parameter. |
| 200 |
|
| 201 |
|
| 202 |
[[s-connection-states]] |
| 203 |
==== Connection states |
| 204 |
|
| 205 |
indexterm:[drbdadm, cstate]indexterm:[connection state, via drbdadm]A resource's connection state can be |
| 206 |
observed either by issuing the `drbdadm cstate` command: |
| 207 |
|
| 208 |
---------------------------- |
| 209 |
# drbdadm cstate <resource> |
| 210 |
Connected |
| 211 |
Connected |
| 212 |
StandAlone |
| 213 |
---------------------------- |
| 214 |
|
| 215 |
If you are interested in only a single connection of a resource, specify the connection name, too: |
| 216 |
|
| 217 |
The default is the peer's hostname as given in the configuration file. |
| 218 |
|
| 219 |
---------------------------- |
| 220 |
# drbdadm cstate <peer>:<resource> |
| 221 |
Connected |
| 222 |
---------------------------- |
| 223 |
|
| 224 |
A resource may have one of the following connection states: |
| 225 |
|
| 226 |
._StandAlone_ |
| 227 |
indexterm:[connection state, StandAlone]No network configuration available. The |
| 228 |
resource has not yet been connected, or has been administratively |
| 229 |
disconnected (using `drbdadm disconnect`), or has dropped its |
| 230 |
connection due to failed authentication or split brain. |
| 231 |
|
| 232 |
._Disconnecting_ |
| 233 |
indexterm:[connection state, Disconnecting]Temporary state during disconnection. The |
| 234 |
next state is _StandAlone_. |
| 235 |
|
| 236 |
._Unconnected_ |
| 237 |
indexterm:[connection state, Unconnected]Temporary state, prior to a connection |
| 238 |
attempt. Possible next states: _Connecting_. |
| 239 |
|
| 240 |
._Timeout_ |
| 241 |
indexterm:[connection state, Timeout]Temporary state following a timeout in the |
| 242 |
communication with the peer. Next state: _Unconnected_. |
| 243 |
|
| 244 |
._BrokenPipe_ |
| 245 |
indexterm:[connection state, BrokenPipe]Temporary state after the connection to |
| 246 |
the peer was lost. Next state: _Unconnected_. |
| 247 |
|
| 248 |
._NetworkFailure_ |
| 249 |
indexterm:[connection state, NetworkFailure]Temporary state after the connection to |
| 250 |
the partner was lost. Next state: _Unconnected_. |
| 251 |
|
| 252 |
._ProtocolError_ |
| 253 |
indexterm:[connection state, ProtocolError]Temporary state after the connection to |
| 254 |
the partner was lost. Next state: _Unconnected_. |
| 255 |
|
| 256 |
._TearDown_ |
| 257 |
indexterm:[connection state, TearDown]Temporary state. The peer is closing the |
| 258 |
connection. Next state: _Unconnected_. |
| 259 |
|
| 260 |
._Connecting_ |
| 261 |
indexterm:[connection state, Connecting]This node is waiting until the peer node |
| 262 |
becomes visible on the network. |
| 263 |
|
| 264 |
._Connected_ |
| 265 |
indexterm:[connection state, Connected]A DRBD connection has been established, |
| 266 |
data mirroring is now active. This is the normal state. |
| 267 |
|
| 268 |
[[s-replication-states]] |
| 269 |
==== Replication states |
| 270 |
|
| 271 |
Each volume has over each connection a replication state. The possible replication |
| 272 |
states are: |
| 273 |
|
| 274 |
._Off_ |
| 275 |
indexterm:[replication state, Off]The volume is not replicated over this |
| 276 |
connection, since the connection is not _Connected_. |
| 277 |
|
| 278 |
._Established_ |
| 279 |
indexterm:[replication state, Established]All writes to that volume are replicated |
| 280 |
online. This is the normal state. |
| 281 |
|
| 282 |
._StartingSyncS_ |
| 283 |
indexterm:[replication state, StartingSyncS]Full synchronization, initiated by the |
| 284 |
administrator, is just starting. The next possible states are: |
| 285 |
_SyncSource_ or _PausedSyncS_. |
| 286 |
|
| 287 |
._StartingSyncT_ |
| 288 |
indexterm:[replication state, StartingSyncT]Full synchronization, initiated by the |
| 289 |
administrator, is just starting. Next state: _WFSyncUUID_. |
| 290 |
|
| 291 |
._WFBitMapS_ |
| 292 |
indexterm:[replication state, WFBitMapS]Partial synchronization is just |
| 293 |
starting. Next possible states: _SyncSource_ or _PausedSyncS_. |
| 294 |
|
| 295 |
._WFBitMapT_ |
| 296 |
indexterm:[replication state, WFBitMapT]Partial synchronization is just |
| 297 |
starting. Next possible state: _WFSyncUUID_. |
| 298 |
|
| 299 |
._WFSyncUUID_ |
| 300 |
indexterm:[replication state, WFSyncUUID]Synchronization is about to begin. Next |
| 301 |
possible states: _SyncTarget_ or _PausedSyncT_. |
| 302 |
|
| 303 |
._SyncSource_ |
| 304 |
indexterm:[replication state, SyncSource]Synchronization is currently running, with |
| 305 |
the local node being the source of synchronization. |
| 306 |
|
| 307 |
._SyncTarget_ |
| 308 |
indexterm:[replication state, SyncTarget]Synchronization is currently running, with |
| 309 |
the local node being the target of synchronization. |
| 310 |
|
| 311 |
._PausedSyncS_ |
| 312 |
indexterm:[replication state, PausedSyncS]The local node is the source of an ongoing |
| 313 |
synchronization, but synchronization is currently paused. This may be |
| 314 |
due to a dependency on the completion of another synchronization |
| 315 |
process, or due to synchronization having been manually interrupted by |
| 316 |
`drbdadm pause-sync`. |
| 317 |
|
| 318 |
._PausedSyncT_ |
| 319 |
indexterm:[replication state, PausedSyncT]The local node is the target of an ongoing |
| 320 |
synchronization, but synchronization is currently paused. This may be |
| 321 |
due to a dependency on the completion of another synchronization |
| 322 |
process, or due to synchronization having been manually interrupted by |
| 323 |
`drbdadm pause-sync`. |
| 324 |
|
| 325 |
._VerifyS_ |
| 326 |
indexterm:[replication state, VerifyS]On-line device verification is currently |
| 327 |
running, with the local node being the source of verification. |
| 328 |
|
| 329 |
._VerifyT_ |
| 330 |
indexterm:[replication state, VerifyT]On-line device verification is currently |
| 331 |
running, with the local node being the target of verification. |
| 332 |
|
| 333 |
._Ahead_ |
| 334 |
indexterm:[replication state, Ahead]Data replication was suspended, since the |
| 335 |
link can not cope with the load. This state is enabled by the configuration |
| 336 |
`on-congestion` option (see <<s-configure-congestion-policy>>). |
| 337 |
|
| 338 |
._Behind_ |
| 339 |
indexterm:[replication state, Behind]Data replication was suspended by the peer, |
| 340 |
since the link can not cope with the load. This state is enabled by the configuration |
| 341 |
`on-congestion` option on the peer node (see <<s-configure-congestion-policy>>). |
| 342 |
|
| 343 |
[[s-roles]] |
| 344 |
==== Resource roles |
| 345 |
|
| 346 |
indexterm:[resource]A resource's role can be observed by issuing the |
| 347 |
indexterm:[drbdadm, role]`drbdadm role` command: |
| 348 |
|
| 349 |
---------------------------- |
| 350 |
# drbdadm role <resource> |
| 351 |
Primary |
| 352 |
---------------------------- |
| 353 |
|
| 354 |
You may see one of the following resource roles: |
| 355 |
|
| 356 |
._Primary_ |
| 357 |
indexterm:[role, Primary]The resource is currently in the primary role, and may be read from |
| 358 |
and written to. This role only occurs on one of the two nodes, unless |
| 359 |
<<s-dual-primary-mode,dual-primary mode>> is enabled. |
| 360 |
|
| 361 |
._Secondary_ |
| 362 |
indexterm:[role, Secondary]The resource is currently in the secondary role. It normally receives |
| 363 |
updates from its peer (unless running in disconnected mode), but may |
| 364 |
neither be read from nor written to. This role may occur on one |
| 365 |
or both nodes. |
| 366 |
|
| 367 |
._Unknown_ |
| 368 |
indexterm:[role, Unknown]The resource's role is currently unknown. The local resource role |
| 369 |
never has this status. It is only displayed for the peer's resource |
| 370 |
role, and only in disconnected mode. |
| 371 |
|
| 372 |
|
| 373 |
[[s-disk-states]] |
| 374 |
==== Disk states |
| 375 |
|
| 376 |
A resource's disk state can be observed either by |
| 377 |
issuing the `drbdadm dstate` command: |
| 378 |
|
| 379 |
---------------------------- |
| 380 |
# drbdadm dstate <resource> |
| 381 |
UpToDate |
| 382 |
---------------------------- |
| 383 |
|
| 384 |
The disk state may be one of the following: |
| 385 |
|
| 386 |
._Diskless_ |
| 387 |
indexterm:[disk state, Diskless]No local block device has been assigned to the |
| 388 |
DRBD driver. This may mean that the resource has never attached to its |
| 389 |
backing device, that it has been manually detached using `drbdadm |
| 390 |
detach`, or that it automatically detached after a lower-level I/O |
| 391 |
error. |
| 392 |
|
| 393 |
._Attaching_ |
| 394 |
indexterm:[disk state, Attaching]Transient state while reading meta data. |
| 395 |
|
| 396 |
._Detaching_ |
| 397 |
indexterm:[disk state, Detaching]Transient state while detaching and waiting |
| 398 |
for ongoing IOs to complete. |
| 399 |
|
| 400 |
._Failed_ |
| 401 |
indexterm:[disk state, Failed]Transient state following an I/O failure report |
| 402 |
by the local block device. Next state: _Diskless_. |
| 403 |
|
| 404 |
._Negotiating_ |
| 405 |
indexterm:[disk state, Negotiating]Transient state when an _Attach_ is carried out on |
| 406 |
an already-__Connected__ DRBD device. |
| 407 |
|
| 408 |
._Inconsistent_ |
| 409 |
indexterm:[disk state, Inconsistent]The data is inconsistent. This status occurs |
| 410 |
immediately upon creation of a new resource, on both nodes (before the |
| 411 |
initial full sync). Also, this status is found in one node (the |
| 412 |
synchronization target) during synchronization. |
| 413 |
|
| 414 |
._Outdated_ |
| 415 |
indexterm:[disk state, Outdated]Resource data is consistent, but |
| 416 |
<<s-outdate,outdated>>. |
| 417 |
|
| 418 |
._DUnknown_ |
| 419 |
indexterm:[disk state, DUnknown]This state is used for the peer disk if no |
| 420 |
network connection is available. |
| 421 |
|
| 422 |
._Consistent_ |
| 423 |
indexterm:[disk state, Consistent]Consistent data of a node without |
| 424 |
connection. When the connection is established, it is decided whether |
| 425 |
the data is _UpToDate_ or _Outdated_. |
| 426 |
|
| 427 |
._UpToDate_ |
| 428 |
indexterm:[disk state, UpToDate]Consistent, up-to-date state of the data. This |
| 429 |
is the normal state. |
| 430 |
|
| 431 |
|
| 432 |
[[s-conn-info]] |
| 433 |
==== Connection information data |
| 434 |
|
| 435 |
._local_ |
| 436 |
Shows the network family, the local address and port that is used to |
| 437 |
accept connections from the peer. |
| 438 |
|
| 439 |
._peer_ |
| 440 |
Shows the network family, the peer address and port that is used |
| 441 |
to connect. |
| 442 |
|
| 443 |
._congested_ |
| 444 |
This flag tells whether the TCP send buffer of the data connection is more than 80% filled. |
| 445 |
|
| 446 |
|
| 447 |
[[s-performance-indicators]] |
| 448 |
==== Performance indicators |
| 449 |
|
| 450 |
indexterm:[performance indicators] |
| 451 |
The command ``drbdsetup status --verbose --statistics`` can be used to show performance statistics. |
| 452 |
These are also available in ``drbdsetup events2 --statistics``, |
| 453 |
although there will not be a _changed_ event for every change. |
| 454 |
The statistics include the following counters and gauges: |
| 455 |
|
| 456 |
Per volume/device: |
| 457 |
|
| 458 |
._read_ (disk read) |
| 459 |
Net data read from local disk; in KiB. |
| 460 |
|
| 461 |
._written_ (disk written) |
| 462 |
Net data written to local disk; in KiB. |
| 463 |
|
| 464 |
._al-writes_ (activity log) |
| 465 |
Number of updates of the activity log area of the meta data. |
| 466 |
|
| 467 |
._bm-writes_ (bitmap) |
| 468 |
Number of updates of the bitmap area of the meta data. |
| 469 |
|
| 470 |
._upper-pending_ (application pending) |
| 471 |
Number of block I/O requests forwarded to DRBD, but not yet answered (completed) |
| 472 |
by DRBD. |
| 473 |
|
| 474 |
._lower-pending_ (local count) |
| 475 |
Number of open requests to the local I/O sub-system issued by DRBD. |
| 476 |
|
| 477 |
._blocked_ |
| 478 |
indexterm:[blocked]Shows local I/O congestion. |
| 479 |
-- |
| 480 |
|
| 481 |
* _no_: No congestion. |
| 482 |
* _upper_: I/O _above_ the DRBD device is blocked, i.e. to the filesystem. Typical causes are |
| 483 |
** I/O suspension by the administrator, see the `suspend-io` command in `drbdadm`. |
| 484 |
** transient blocks, e.g. during attach/detach |
| 485 |
** buffers depleted, see <<p-performance>> |
| 486 |
** Waiting for bitmap IO |
| 487 |
* _lower_: Backing device is congested. |
| 488 |
* _upper,lower_: Both _upper_ and _lower_ are blocked. |
| 489 |
-- |
| 490 |
|
| 491 |
Per connection: |
| 492 |
|
| 493 |
._ap-in-flight_ (application in-flight) |
| 494 |
Application data that is being written by the peer. |
| 495 |
That is, DRBD has sent it to the peer and is waiting for the acknowledgement that it has been written. |
| 496 |
In sectors (512 bytes). |
| 497 |
|
| 498 |
._rs-in-flight_ (resync in-flight) |
| 499 |
Resync data that is being written by the peer. |
| 500 |
That is, DRBD is _SyncSource_, has sent data to the peer as part of a resync |
| 501 |
and is waiting for the acknowledgement that it has been written. |
| 502 |
In sectors (512 bytes). |
| 503 |
|
| 504 |
Per connection and volume ("peer device"): |
| 505 |
|
| 506 |
._received_ (network receive) |
| 507 |
Net data received from the peer; in KiB. |
| 508 |
|
| 509 |
._sent_ (network send) |
| 510 |
Net data sent to the peer; in KiB. |
| 511 |
|
| 512 |
._out-of-sync_ |
| 513 |
indexterm:[out-of-sync]indexterm:[oos]Amount of data currently out of sync |
| 514 |
with this peer, according to the bitmap that DRBD has for it; in KiB. |
| 515 |
|
| 516 |
._pending_ |
| 517 |
Number of requests sent to the peer, but that have not yet been |
| 518 |
acknowledged by the peer. |
| 519 |
|
| 520 |
._unacked_ (unacknowledged) |
| 521 |
Number of requests received from the peer, |
| 522 |
but that have not yet been acknowledged by DRBD on this node. |
| 523 |
|
| 524 |
._resync-suspended_ |
| 525 |
Whether the resynchronization is currently suspended or not. Possible values are _no_, _user_, _peer_, _dependency_. |
| 526 |
Comma separated. |
| 527 |
|
| 528 |
|
| 529 |
[[s-enable-disable]] |
| 530 |
=== Enabling and disabling resources |
| 531 |
|
| 532 |
[[s-enable-resource]] |
| 533 |
==== Enabling resources |
| 534 |
|
| 535 |
indexterm:[resource]Normally, all configured DRBD resources are |
| 536 |
automatically enabled |
| 537 |
|
| 538 |
* by a cluster resource management application at its discretion, |
| 539 |
based on your cluster configuration, or |
| 540 |
|
| 541 |
* by the systemd units (e.g., `[email protected]`) |
| 542 |
|
| 543 |
If, however, you need to enable resources manually for any reason, you |
| 544 |
may do so by issuing the command |
| 545 |
|
| 546 |
---------------------------- |
| 547 |
# drbdadm up <resource> |
| 548 |
---------------------------- |
| 549 |
|
| 550 |
As always, you may use the keyword `all` instead of a specific |
| 551 |
resource name if you want to enable all resources configured in |
| 552 |
`/etc/drbd.conf` at once. |
| 553 |
|
| 554 |
[[s-disable-resource]] |
| 555 |
==== Disabling resources |
| 556 |
|
| 557 |
indexterm:[resource]You may temporarily disable specific resources by |
| 558 |
issuing the command |
| 559 |
|
| 560 |
---------------------------- |
| 561 |
# drbdadm down <resource> |
| 562 |
---------------------------- |
| 563 |
|
| 564 |
Here, too, you may use the keyword `all` in place of a resource name if |
| 565 |
you wish to temporarily disable all resources listed in |
| 566 |
`/etc/drbd.conf` at once. |
| 567 |
|
| 568 |
[[s-reconfigure]] |
| 569 |
=== Reconfiguring resources |
| 570 |
|
| 571 |
indexterm:[resource]DRBD allows you to reconfigure resources while |
| 572 |
they are operational. To that end, |
| 573 |
|
| 574 |
* make any necessary changes to the resource configuration in |
| 575 |
`/etc/drbd.conf`, |
| 576 |
|
| 577 |
* synchronize your `/etc/drbd.conf` file between both nodes, |
| 578 |
|
| 579 |
* issue the indexterm:[drbdadm, adjust]`drbdadm adjust <resource>` command on |
| 580 |
both nodes. |
| 581 |
|
| 582 |
`drbdadm adjust` then hands off to `drbdsetup` to make the necessary |
| 583 |
adjustments to the configuration. As always, you are able to review |
| 584 |
the pending `drbdsetup` invocations by running `drbdadm` with the |
| 585 |
`-d` (dry-run) option. |
| 586 |
|
| 587 |
NOTE: When making changes to the `common` section in `/etc/drbd.conf`, |
| 588 |
you can adjust the configuration for all resources in one run, by |
| 589 |
issuing `drbdadm adjust all`. |
| 590 |
|
| 591 |
[[s-switch-resource-roles]] |
| 592 |
=== Promoting and demoting resources |
| 593 |
|
| 594 |
indexterm:[resource]Manually switching a <<s-resource-roles,resource's |
| 595 |
role>> from secondary to primary (promotion) or vice versa (demotion) |
| 596 |
is done using one of the following commands: |
| 597 |
|
| 598 |
---------------------------- |
| 599 |
# drbdadm primary <resource> |
| 600 |
# drbdadm secondary <resource> |
| 601 |
---------------------------- |
| 602 |
|
| 603 |
In <<s-single-primary-mode,single-primary mode>> (DRBD's default), any |
| 604 |
resource can be in the primary role on only one node at any given time |
| 605 |
while the <<s-connection-states,connection state>> is |
| 606 |
_Connected_. Thus, issuing `drbdadm primary <resource>` on one node |
| 607 |
while the specified resource is still in the primary role on another node will |
| 608 |
result in an error. |
| 609 |
|
| 610 |
A resource configured to allow <<s-dual-primary-mode,dual-primary |
| 611 |
mode>> can be switched to the primary role on two nodes; this is e.g. needed for |
| 612 |
online migration of virtual machines. |
| 613 |
|
| 614 |
|
| 615 |
[[s-manual-fail-over]] |
| 616 |
=== Basic manual fail-over |
| 617 |
|
| 618 |
If not using a cluster manager and looking to handle fail-overs manually in a |
| 619 |
passive/active configuration, the process is as follows. |
| 620 |
|
| 621 |
On the current primary node stop any applications or services using the DRBD device, |
| 622 |
unmount the DRBD device, and demote the resource to secondary. |
| 623 |
|
| 624 |
---------------------------- |
| 625 |
# umount /dev/drbd/by-res/<resource>/<vol-nr> |
| 626 |
# drbdadm secondary <resource> |
| 627 |
---------------------------- |
| 628 |
|
| 629 |
Now on the node we wish to make primary promote the resource and mount the device. |
| 630 |
|
| 631 |
---------------------------- |
| 632 |
# drbdadm primary <resource> |
| 633 |
# mount /dev/drbd/by-res/<resource>/<vol-nr> <mountpoint> |
| 634 |
---------------------------- |
| 635 |
|
| 636 |
If you're using the `auto-promote` feature, you don't need to change the roles |
| 637 |
(_Primary_/_Secondary_) manually; only stopping of the services and unmounting, |
| 638 |
respectively mounting, is necessary. |
| 639 |
|
| 640 |
[[s-upgrading-drbd]] |
| 641 |
include::upgrading.adoc[] |
| 642 |
|
| 643 |
[[s-enable-dual-primary]] |
| 644 |
include::dualprimary.adoc[] |
| 645 |
|
| 646 |
[[s-use-online-verify]] |
| 647 |
=== Using on-line device verification |
| 648 |
|
| 649 |
[[s-online-verify-enable]] |
| 650 |
==== Enabling on-line verification |
| 651 |
|
| 652 |
indexterm:[on-line device verification]<<s-online-verify,On-line |
| 653 |
device verification>> for resources is not enabled by default. To |
| 654 |
enable it, add the following lines to your resource configuration in |
| 655 |
`/etc/drbd.conf`: |
| 656 |
|
| 657 |
[source,drbd] |
| 658 |
---------------------------- |
| 659 |
resource <resource> |
| 660 |
net { |
| 661 |
verify-alg <algorithm>; |
| 662 |
} |
| 663 |
... |
| 664 |
} |
| 665 |
---------------------------- |
| 666 |
|
| 667 |
_<algorithm>_ may be any message digest algorithm supported by the |
| 668 |
kernel crypto API in your system's kernel configuration. Normally, you |
| 669 |
should be able to choose at least from `sha1`, `md5`, and `crc32c`. |
| 670 |
|
| 671 |
If you make this change to an existing resource, as always, |
| 672 |
synchronize your `drbd.conf` to the peer, and run `drbdadm adjust |
| 673 |
_<resource>_` on both nodes. |
| 674 |
|
| 675 |
[[s-online-verify-invoke]] |
| 676 |
==== Invoking on-line verification |
| 677 |
|
| 678 |
indexterm:[on-line device verification]After you have enabled on-line |
| 679 |
verification, you will be able to initiate a verification run using |
| 680 |
the following command: |
| 681 |
|
| 682 |
---------------------------- |
| 683 |
# drbdadm verify <resource>:<peer>/<volume> |
| 684 |
---------------------------- |
| 685 |
|
| 686 |
When you do so, DRBD starts an online verification run for |
| 687 |
_<volume>_ to _<peer_> in _<resource>_, and if it detects any blocks |
| 688 |
that are not in sync, will mark those blocks as such and write a |
| 689 |
message to the kernel log. Any applications using the device at that |
| 690 |
time can continue to do so unimpeded, and you may also |
| 691 |
<<s-switch-resource-roles,switch resource roles>> at will. |
| 692 |
|
| 693 |
_<Volume>_ is optional, if ommited, it will verify all volumes |
| 694 |
in that resource. |
| 695 |
|
| 696 |
If out-of-sync blocks were detected during the verification run, you |
| 697 |
may resynchronize them using the following commands after verification |
| 698 |
has completed. |
| 699 |
|
| 700 |
Since drbd-9.0.29 the preferred way is one of these commands: |
| 701 |
|
| 702 |
---------------------------- |
| 703 |
# drbdadm invalidate <resource>:<peer>/volume --reset-bitmap=no |
| 704 |
# drbdadm invalidate-remote <resource>:<peer>/volume --reset-bitmap=no |
| 705 |
---------------------------- |
| 706 |
|
| 707 |
The first command will cause the local differences to be overwritten |
| 708 |
by the remote version. The second command does it in the opposite direction. |
| 709 |
|
| 710 |
Before drbd-9.0.29 one needs to initiate a resync. A way to do that |
| 711 |
is disconnecting from a primary and ensuring that the primary changes |
| 712 |
at least one block while the peer is away. |
| 713 |
|
| 714 |
---------------------------- |
| 715 |
# drbdadm disconnect <resource>:<peer> |
| 716 |
## write one block on the primary |
| 717 |
# drbdadm connect <resource>:<peer> |
| 718 |
---------------------------- |
| 719 |
|
| 720 |
|
| 721 |
[[s-online-verify-automate]] |
| 722 |
==== Automating on-line verification |
| 723 |
|
| 724 |
indexterm:[on-line device verification]Most users will want to |
| 725 |
automate on-line device verification. This can be easily |
| 726 |
accomplished. Create a file with the following contents, named |
| 727 |
`/etc/cron.d/drbd-verify` on *one* of your nodes: |
| 728 |
|
| 729 |
[source,drbd] |
| 730 |
---------------------------- |
| 731 |
42 0 * * 0 root /sbin/drbdadm verify <resource> |
| 732 |
---------------------------- |
| 733 |
|
| 734 |
This will have `cron` invoke a device verification every Sunday at 42 |
| 735 |
minutes past midnight; so, if you come into the office on Monday morning, |
| 736 |
a quick look at the resource's status would show the result. If your devices |
| 737 |
are very big, and the \~32 hours were not enough, then you'll notice _VerifyS_ or |
| 738 |
_VerifyT_ as connection state, meaning that the `verify` is still in progress. |
| 739 |
|
| 740 |
If you have enabled on-line verification for all your resources (for |
| 741 |
example, by adding `verify-alg _<algorithm>_` to the `common` section |
| 742 |
in `/etc/drbd.d/global_common.conf`), you may also use: |
| 743 |
|
| 744 |
[source,drbd] |
| 745 |
---------------------------- |
| 746 |
42 0 * * 0 root /sbin/drbdadm verify all |
| 747 |
---------------------------- |
| 748 |
|
| 749 |
|
| 750 |
[[s-configure-sync-rate]] |
| 751 |
=== Configuring the rate of synchronization |
| 752 |
|
| 753 |
indexterm:[synchronization]Normally, one tries to ensure that |
| 754 |
background synchronization (which makes the data on the |
| 755 |
synchronization target temporarily inconsistent) completes as quickly |
| 756 |
as possible. However, it is also necessary to keep background |
| 757 |
synchronization from hogging all bandwidth otherwise available for |
| 758 |
foreground replication, which would be detrimental to application |
| 759 |
performance. Thus, you must configure the synchronization bandwidth to |
| 760 |
match your hardware -- which you may do in a permanent fashion or |
| 761 |
on-the-fly. |
| 762 |
|
| 763 |
IMPORTANT: It does not make sense to set a synchronization rate that |
| 764 |
is higher than the maximum write throughput on your secondary |
| 765 |
node. You must not expect your secondary node to miraculously be able |
| 766 |
to write faster than its I/O subsystem allows, just because it happens |
| 767 |
to be the target of an ongoing device synchronization. |
| 768 |
|
| 769 |
Likewise, and for the same reasons, it does not make sense to set a |
| 770 |
synchronization rate that is higher than the bandwidth available on |
| 771 |
the replication network. |
| 772 |
|
| 773 |
|
| 774 |
[[s-estimating_a_synchronization_speed]] |
| 775 |
==== Estimating a synchronization speed |
| 776 |
|
| 777 |
TIP: A good rule of thumb for this value is to use about 30% of the |
| 778 |
available replication bandwidth. Thus, if you had an I/O subsystem |
| 779 |
capable of sustaining write throughput of 400MB/s, and a Gigabit |
| 780 |
Ethernet network capable of sustaining 110 MB/s network throughput |
| 781 |
(the network being the bottleneck), you would calculate: |
| 782 |
|
| 783 |
[[eq-sync-rate-example1]] |
| 784 |
.Syncer rate example, 110MB/s effective available bandwidth |
| 785 |
image::images/sync-rate-example1.svg[] |
| 786 |
|
| 787 |
Thus, the recommended value for the `rate` option would be `33M`. |
| 788 |
|
| 789 |
By contrast, if you had an I/O subsystem with a maximum throughput of |
| 790 |
80MB/s and a Gigabit Ethernet connection (the I/O subsystem being the |
| 791 |
bottleneck), you would calculate: |
| 792 |
|
| 793 |
[[eq-sync-rate-example2]] |
| 794 |
.Syncer rate example, 80MB/s effective available bandwidth |
| 795 |
image::images/sync-rate-example2.svg[] |
| 796 |
|
| 797 |
In this case, the recommended value for the `rate` option would be |
| 798 |
`24M`. |
| 799 |
|
| 800 |
Similarly, for a storage speed of 800MB/s and a 10Gbe network connection, you |
| 801 |
would shoot for \~240MB/s synchronization rate. |
| 802 |
|
| 803 |
[[s-configure-sync-rate-variable]] |
| 804 |
==== Variable sync rate configuration |
| 805 |
|
| 806 |
When multiple DRBD resources share a |
| 807 |
single replication/synchronization network, synchronization with a fixed rate |
| 808 |
may not be an optimal approach. So, in DRBD 8.4.0 the |
| 809 |
variable-rate synchronization was enabled by default. In this mode, DRBD uses an automated |
| 810 |
control loop algorithm to determine, and adjust, the |
| 811 |
synchronization rate. This algorithm ensures that there is always |
| 812 |
sufficient bandwidth available for foreground replication, greatly |
| 813 |
mitigating the impact that background synchronization has on |
| 814 |
foreground I/O. |
| 815 |
|
| 816 |
The optimal configuration for variable-rate synchronization may vary |
| 817 |
greatly depending on the available network bandwidth, application I/O |
| 818 |
pattern and link congestion. |
| 819 |
ifndef::drbd-only,de-brand[] |
| 820 |
Ideal configuration settings also depend |
| 821 |
on whether <<s-drbd-proxy,DRBD Proxy>> is in use or not. |
| 822 |
It may be |
| 823 |
wise to engage professional consultancy in order to optimally |
| 824 |
configure this DRBD feature. An _example_ configuration (which assumes |
| 825 |
a deployment in conjunction with DRBD Proxy) is provided below: |
| 826 |
|
| 827 |
[source,drbd] |
| 828 |
---------------------------- |
| 829 |
resource <resource> { |
| 830 |
disk { |
| 831 |
c-plan-ahead 5; |
| 832 |
c-max-rate 10M; |
| 833 |
c-fill-target 2M; |
| 834 |
} |
| 835 |
} |
| 836 |
---------------------------- |
| 837 |
|
| 838 |
endif::[] |
| 839 |
|
| 840 |
TIP: A good starting value for `c-fill-target` is _BDP * 2_, where |
| 841 |
_BDP_ is your bandwidth-delay-product on the replication link. |
| 842 |
|
| 843 |
For example, when using a 1GBit/s crossover connection, you'll end up with |
| 844 |
about 200µs latencyfootnote:[The rule-of-thumb is using the time reported by |
| 845 |
`ping`.]. + |
| 846 |
1GBit/s means about 120MB/s; times 200*10^-6^ seconds gives 24000 Byte. Just round that value up to the next MB, and you're good to go. |
| 847 |
|
| 848 |
Another example: a 100MBit WAN connection with 200ms latency means 12MB/s times |
| 849 |
0.2s, or about 2.5MB "on the wire". Here a good starting value for |
| 850 |
`c-fill-target` would be 3MB. |
| 851 |
|
| 852 |
Please see the `drbd.conf` manual page for more details on the other |
| 853 |
configuration items. |
| 854 |
|
| 855 |
|
| 856 |
[[s-configure-sync-rate-permanent]] |
| 857 |
==== Permanent fixed sync rate configuration |
| 858 |
|
| 859 |
In a few, very restricted situationsfootnote:[Like benchmarking.], it might |
| 860 |
make sense to just use some fixed synchronization rate. In this case, first |
| 861 |
of all you need to turn the dynamic sync rate controller off, by using |
| 862 |
`c-plan-ahead 0;`. |
| 863 |
|
| 864 |
Then, the maximum bandwidth a resource uses for background |
| 865 |
re-synchronization is determined by the `resync-rate` option |
| 866 |
for a resource. This must be included in the resource's |
| 867 |
`disk` section in `/etc/drbd.conf`: |
| 868 |
|
| 869 |
[source,drbd] |
| 870 |
---------------------------- |
| 871 |
resource <resource> |
| 872 |
disk { |
| 873 |
resync-rate 40M; |
| 874 |
... |
| 875 |
} |
| 876 |
... |
| 877 |
} |
| 878 |
---------------------------- |
| 879 |
|
| 880 |
Note that the rate setting is given in _bytes_, not _bits_ per second; the |
| 881 |
default unit is _Kibibyte_, so a value of `4096` would be interpreted as `4MiB`. |
| 882 |
|
| 883 |
NOTE: This just defines a rate that DRBD tries to achieve. If there is a |
| 884 |
bottleneck with lower throughput (network, storage speed), the defined speed |
| 885 |
(aka the "wished-for" performance ;) won't be reached. |
| 886 |
|
| 887 |
|
| 888 |
[[s-some_more_hints_about_synchronization]] |
| 889 |
==== Some more hints about synchronization |
| 890 |
|
| 891 |
When some amount of the to-be-synchronized data isn't really in use anymore |
| 892 |
(e.g. because files got deleted while one node wasn't connected), you |
| 893 |
might benefit from the <<s-trim-discard>>. |
| 894 |
|
| 895 |
Furthermore, `c-min-rate` is easy to misunderstand - it doesn't define |
| 896 |
a *minimum* synchronization speed, but rather a limit below which DRBD will not |
| 897 |
slow down further *on purpose*. + |
| 898 |
Whether you manage to *reach* that synchronization rate depends on your network |
| 899 |
and storage speed, network latency (which might be highly variable for shared |
| 900 |
links), and application IO (which not might be able to do anything about). |
| 901 |
|
| 902 |
|
| 903 |
[[s-configure-checksum-sync]] |
| 904 |
=== Configuring checksum-based synchronization |
| 905 |
|
| 906 |
indexterm:[checksum-based |
| 907 |
synchronization]<<p-checksum-sync,Checksum-based synchronization>> is |
| 908 |
not enabled for resources by default. To enable it, add the following |
| 909 |
lines to your resource configuration in `/etc/drbd.conf`: |
| 910 |
|
| 911 |
[source,drbd] |
| 912 |
---------------------------- |
| 913 |
resource <resource> |
| 914 |
net { |
| 915 |
csums-alg <algorithm>; |
| 916 |
} |
| 917 |
... |
| 918 |
} |
| 919 |
---------------------------- |
| 920 |
|
| 921 |
_<algorithm>_ may be any message digest algorithm supported by the |
| 922 |
kernel crypto API in your system's kernel configuration. Normally, you |
| 923 |
should be able to choose at least from `sha1`, `md5`, and `crc32c`. |
| 924 |
|
| 925 |
If you make this change to an existing resource, as always, |
| 926 |
synchronize your `drbd.conf` to the peer, and run `drbdadm adjust |
| 927 |
_<resource>_` on both nodes. |
| 928 |
|
| 929 |
[[s-configure-congestion-policy]] |
| 930 |
=== Configuring congestion policies and suspended replication |
| 931 |
|
| 932 |
In an environment where the replication bandwidth is highly variable |
| 933 |
(as would be typical in WAN replication setups), the replication link |
| 934 |
may occasionally become congested. In a default configuration, this |
| 935 |
would cause I/O on the primary node to block, which is sometimes |
| 936 |
undesirable. |
| 937 |
|
| 938 |
Instead, you may configure DRBD to _suspend_ the ongoing replication |
| 939 |
in this case, causing the Primary's data set to _pull ahead_ of the |
| 940 |
Secondary. In this mode, DRBD keeps the replication channel open -- it |
| 941 |
never switches to disconnected mode -- but does not actually replicate |
| 942 |
until sufficient bandwidth becomes available again. |
| 943 |
|
| 944 |
The following example is for a DRBD Proxy configuration: |
| 945 |
|
| 946 |
[source,drbd] |
| 947 |
---------------------------- |
| 948 |
resource <resource> { |
| 949 |
net { |
| 950 |
on-congestion pull-ahead; |
| 951 |
congestion-fill 2G; |
| 952 |
congestion-extents 2000; |
| 953 |
... |
| 954 |
} |
| 955 |
... |
| 956 |
} |
| 957 |
---------------------------- |
| 958 |
|
| 959 |
It is usually wise to set both `congestion-fill` and |
| 960 |
`congestion-extents` together with the `pull-ahead` option. |
| 961 |
|
| 962 |
A good value for `congestion-fill` is 90% |
| 963 |
|
| 964 |
* of the allocated DRBD proxy buffer memory, when replicating over |
| 965 |
DRBD Proxy, or |
| 966 |
* of the TCP network send buffer, in non-DRBD Proxy setups. |
| 967 |
|
| 968 |
A good value for `congestion-extents` is 90% of your configured |
| 969 |
`al-extents` for the affected resources. |
| 970 |
|
| 971 |
|
| 972 |
[[s-configure-io-error-behavior]] |
| 973 |
=== Configuring I/O error handling strategies |
| 974 |
|
| 975 |
indexterm:[I/O errors]indexterm:[drbd.conf]DRBD's |
| 976 |
<<s-handling-disk-errors,strategy for handling lower-level I/O |
| 977 |
errors>> is determined by the `on-io-error` option, included in the |
| 978 |
resource `disk` configuration in `/etc/drbd.conf`: |
| 979 |
|
| 980 |
[source,drbd] |
| 981 |
---------------------------- |
| 982 |
resource <resource> { |
| 983 |
disk { |
| 984 |
on-io-error <strategy>; |
| 985 |
... |
| 986 |
} |
| 987 |
... |
| 988 |
} |
| 989 |
---------------------------- |
| 990 |
|
| 991 |
You may, of course, set this in the `common` section too, if you want |
| 992 |
to define a global I/O error handling policy for all resources. |
| 993 |
|
| 994 |
_<strategy>_ may be one of the following options: |
| 995 |
|
| 996 |
.`detach` |
| 997 |
This is the default and recommended option. On the occurrence of a |
| 998 |
lower-level I/O error, the node drops its backing device, and |
| 999 |
continues in diskless mode. |
| 1000 |
|
| 1001 |
.`pass-on` |
| 1002 |
This causes DRBD to report the I/O error to the upper layers. On the |
| 1003 |
primary node, it is reported to the mounted file system. On the |
| 1004 |
secondary node, it is ignored (because the secondary has no upper |
| 1005 |
layer to report to). |
| 1006 |
|
| 1007 |
.`call-local-io-error` |
| 1008 |
Invokes the command defined as the local I/O error handler. This |
| 1009 |
requires that a corresponding `local-io-error` command invocation is |
| 1010 |
defined in the resource's `handlers` section. It is entirely left to |
| 1011 |
the administrator's discretion to implement I/O error handling using |
| 1012 |
the command (or script) invoked by `local-io-error`. |
| 1013 |
|
| 1014 |
NOTE: Early DRBD versions (prior to 8.0) included another option, |
| 1015 |
`panic`, which would forcibly remove the node from the cluster by way |
| 1016 |
of a kernel panic, whenever a local I/O error occurred. While that |
| 1017 |
option is no longer available, the same behavior may be mimicked via |
| 1018 |
the `local-io-error`/`call-local-io-error` interface. You should do so |
| 1019 |
only if you fully understand the implications of such behavior. |
| 1020 |
|
| 1021 |
|
| 1022 |
You may reconfigure a running resource's I/O error handling strategy |
| 1023 |
by following this process: |
| 1024 |
|
| 1025 |
* Edit the resource configuration in `/etc/drbd.d/__<resource>__.res`. |
| 1026 |
|
| 1027 |
* Copy the configuration to the peer node. |
| 1028 |
|
| 1029 |
* Issue `drbdadm adjust _<resource>_` on both nodes. |
| 1030 |
|
| 1031 |
|
| 1032 |
[[s-configure-integrity-check]] |
| 1033 |
=== Configuring replication traffic integrity checking |
| 1034 |
|
| 1035 |
indexterm:[replication traffic integrity |
| 1036 |
checking]<<s-integrity-check,Replication traffic integrity checking>> |
| 1037 |
is not enabled for resources by default. To enable it, add the |
| 1038 |
following lines to your resource configuration in `/etc/drbd.conf`: |
| 1039 |
|
| 1040 |
[source,drbd] |
| 1041 |
---------------------------- |
| 1042 |
resource <resource> |
| 1043 |
net { |
| 1044 |
data-integrity-alg <algorithm>; |
| 1045 |
} |
| 1046 |
... |
| 1047 |
} |
| 1048 |
---------------------------- |
| 1049 |
|
| 1050 |
_<algorithm>_ may be any message digest algorithm supported by the |
| 1051 |
kernel crypto API in your system's kernel configuration. Normally, you |
| 1052 |
should be able to choose at least from `sha1`, `md5`, and `crc32c`. |
| 1053 |
|
| 1054 |
If you make this change to an existing resource, as always, |
| 1055 |
synchronize your `drbd.conf` to the peer, and run `drbdadm adjust |
| 1056 |
_<resource>_` on both nodes. |
| 1057 |
|
| 1058 |
WARNING: This feature is not intended for production use. Enable only if you need to |
| 1059 |
diagnose data corruption problems, and want to see whether the transport path |
| 1060 |
(network hardware, drivers, switches) might be at fault! |
| 1061 |
|
| 1062 |
[[s-resizing]] |
| 1063 |
=== Resizing resources |
| 1064 |
|
| 1065 |
|
| 1066 |
When growing (extending) DRBD volumes, you need to grow from bottom to top. |
| 1067 |
You need to extend the backing block devices on all nodes first. |
| 1068 |
Then you can tell DRBD to use the new space. |
| 1069 |
|
| 1070 |
Once the DRBD volume is extended, you need to still propagate that |
| 1071 |
into whatever is using DRBD: extend the file system, or make a VM |
| 1072 |
running with this volume attached aware of the new "disk size". |
| 1073 |
|
| 1074 |
That all typically boils down to |
| 1075 |
|
| 1076 |
----------------------- |
| 1077 |
# # on all nodes, resize the backing LV: |
| 1078 |
# lvextend -L +${additional_gb}g VG/LV |
| 1079 |
# # on one node: |
| 1080 |
# drbdadm resize ${resource_name}/${volume_number} |
| 1081 |
# # on the Primary only: |
| 1082 |
# # resize the file system using the file system specific tool, see below |
| 1083 |
----------------------- |
| 1084 |
|
| 1085 |
See also the next sections <<s-growing-online>> and following. |
| 1086 |
|
| 1087 |
Note that different file systems have different capabilities and different sets |
| 1088 |
of management tools. For example XFS can only grow. |
| 1089 |
You point its tool to the active mount point: `xfs_growfs /where/you/have/it/mounted`. |
| 1090 |
|
| 1091 |
While the EXT family can both grow (even online), and also shrink (only |
| 1092 |
offline; you have to unmount it first). To resize an ext3 or ext4, |
| 1093 |
you would point the tool not to the mount point, but to the (mounted) |
| 1094 |
block device: `resize2fs /dev/drbd#` |
| 1095 |
|
| 1096 |
Obviously use the correct DRBD (as displayed by `mount` or `df -T`, while mounted), |
| 1097 |
and *not* the backing block device. As long as DRBD is up, that's not supposed |
| 1098 |
to work anyways (`resize2fs: Device or resource busy while trying to open |
| 1099 |
/dev/mapper/VG-LV Couldn't find valid filesystem superblock.`). |
| 1100 |
If you tried to do that offline (with DRBD stopped), you may corrupt DRBD |
| 1101 |
meta data if you ran the file system tools directly against the backing LV or |
| 1102 |
partition. So don't. |
| 1103 |
|
| 1104 |
You do the file system resize only *once* on the Primary, against the active |
| 1105 |
DRBD device. DRBD replicates the changes to the file system structure. |
| 1106 |
That is what you have it for. |
| 1107 |
|
| 1108 |
Also, don't use resize2fs on XFS volumes, or XFS tools on EXT, or ... |
| 1109 |
but the right tool for the file system in use. |
| 1110 |
|
| 1111 |
`resize2fs: Bad magic number in super-block while trying to open /dev/drbd7` |
| 1112 |
is probably just trying to tell you that this is *not* an EXT file system, |
| 1113 |
and you should try an other tool instead. Maybe `xfs_growfs`? But as mentioned, |
| 1114 |
that does not take the block device, but the mount point as argument. |
| 1115 |
|
| 1116 |
When shrinking (reducing) DRBD volumes, you need to shrink from top to bottom. |
| 1117 |
So first make sure no one is using the space you want to cut off. |
| 1118 |
Next, shrink the file system (if your file system supports that). |
| 1119 |
Then tell DRBD to stop using that space, which is not so easy with DRBD |
| 1120 |
internal meta data, because they are by design "at the end" |
| 1121 |
of the backing device. |
| 1122 |
|
| 1123 |
Once you are sure that DRBD won't use the space anymore either, |
| 1124 |
you can cut it off from the backing device, for example using `lvreduce`. |
| 1125 |
|
| 1126 |
See also <<s-shrinking-online>>, <<s-shrinking-offline>>. |
| 1127 |
|
| 1128 |
|
| 1129 |
[[s-growing-online]] |
| 1130 |
==== Growing on-line |
| 1131 |
|
| 1132 |
indexterm:[resource]If the backing block devices can be grown while in |
| 1133 |
operation (online), it is also possible to increase the size of a DRBD |
| 1134 |
device based on these devices during operation. To do so, two criteria |
| 1135 |
must be fulfilled: |
| 1136 |
|
| 1137 |
. The affected resource's backing device must be one managed by a |
| 1138 |
logical volume management subsystem, such as LVM. |
| 1139 |
|
| 1140 |
. The resource must currently be in the _Connected_ connection state. |
| 1141 |
|
| 1142 |
Having grown the backing block devices on all nodes, ensure that only |
| 1143 |
one node is in primary state. Then enter on one node: |
| 1144 |
|
| 1145 |
---------------------------- |
| 1146 |
# drbdadm resize <resource> |
| 1147 |
---------------------------- |
| 1148 |
|
| 1149 |
This triggers a synchronization of the new section. The |
| 1150 |
synchronization is done from the primary node to the secondary node. |
| 1151 |
|
| 1152 |
If the space you're adding is clean, you can skip syncing the additional |
| 1153 |
space by using the --assume-clean option. |
| 1154 |
|
| 1155 |
---------------------------- |
| 1156 |
# drbdadm -- --assume-clean resize <resource> |
| 1157 |
---------------------------- |
| 1158 |
|
| 1159 |
[[s-growing-offline]] |
| 1160 |
==== Growing off-line |
| 1161 |
|
| 1162 |
indexterm:[resource]When the backing block devices on both nodes are |
| 1163 |
grown while DRBD is inactive, and the DRBD resource is using |
| 1164 |
<<s-external-meta-data,external meta data>>, then the new size is |
| 1165 |
recognized automatically. No administrative intervention is |
| 1166 |
necessary. The DRBD device will have the new size after the next |
| 1167 |
activation of DRBD on both nodes and a successful establishment of a |
| 1168 |
network connection. |
| 1169 |
|
| 1170 |
If however the DRBD resource is configured to use |
| 1171 |
<<s-internal-meta-data,internal meta data>>, then this meta data must |
| 1172 |
be moved to the end of the grown device before the new size becomes |
| 1173 |
available. To do so, complete the following steps: |
| 1174 |
|
| 1175 |
WARNING: This is an advanced procedure. Use at your own discretion. |
| 1176 |
|
| 1177 |
* Unconfigure your DRBD resource: |
| 1178 |
|
| 1179 |
[source,drbd] |
| 1180 |
|
| 1181 |
---------------------------- |
| 1182 |
# drbdadm down <resource> |
| 1183 |
---------------------------- |
| 1184 |
|
| 1185 |
* Save the meta data in a text file prior to resizing: |
| 1186 |
|
| 1187 |
---------------------------- |
| 1188 |
# drbdadm dump-md <resource> > /tmp/metadata |
| 1189 |
---------------------------- |
| 1190 |
|
| 1191 |
You must do this on both nodes, using a separate dump file for every |
| 1192 |
node. *Do not* dump the meta data on one node, and simply copy the |
| 1193 |
dump file to the peer. *This. will. not. work.* |
| 1194 |
|
| 1195 |
* Grow the backing block device on both nodes. |
| 1196 |
|
| 1197 |
* Adjust the size information (`la-size-sect`) in the file |
| 1198 |
`/tmp/metadata` accordingly, on both nodes. Remember that |
| 1199 |
`la-size-sect` must be specified in sectors. |
| 1200 |
|
| 1201 |
* Re-initialize the metadata area: |
| 1202 |
|
| 1203 |
---------------------------- |
| 1204 |
# drbdadm create-md <resource> |
| 1205 |
---------------------------- |
| 1206 |
|
| 1207 |
* Re-import the corrected meta data, on both nodes: |
| 1208 |
|
| 1209 |
---------------------------- |
| 1210 |
# drbdmeta_cmd=$(drbdadm -d dump-md <resource>) |
| 1211 |
# ${drbdmeta_cmd/dump-md/restore-md} /tmp/metadata |
| 1212 |
Valid meta-data in place, overwrite? [need to type 'yes' to confirm] |
| 1213 |
yes |
| 1214 |
Successfully restored meta data |
| 1215 |
---------------------------- |
| 1216 |
|
| 1217 |
NOTE: This example uses `bash` parameter substitution. It may or may |
| 1218 |
not work in other shells. Check your `SHELL` environment variable if |
| 1219 |
you are unsure which shell you are currently using. |
| 1220 |
|
| 1221 |
* Re-enable your DRBD resource: |
| 1222 |
|
| 1223 |
---------------------------- |
| 1224 |
# drbdadm up <resource> |
| 1225 |
---------------------------- |
| 1226 |
|
| 1227 |
* On one node, promote the DRBD resource: |
| 1228 |
|
| 1229 |
---------------------------- |
| 1230 |
# drbdadm primary <resource> |
| 1231 |
---------------------------- |
| 1232 |
|
| 1233 |
* Finally, grow the file system so it fills the extended size of the |
| 1234 |
DRBD device. |
| 1235 |
|
| 1236 |
|
| 1237 |
[[s-shrinking-online]] |
| 1238 |
==== Shrinking on-line |
| 1239 |
|
| 1240 |
|
| 1241 |
WARNING: Online shrinking is only supported with external metadata. |
| 1242 |
|
| 1243 |
indexterm:[resource]Before shrinking a DRBD device, you _must_ shrink |
| 1244 |
the layers above DRBD, i.e. usually the file system. Since DRBD cannot |
| 1245 |
ask the file system how much space it actually uses, you have to be |
| 1246 |
careful in order not to cause data loss. |
| 1247 |
|
| 1248 |
NOTE: Whether or not the _filesystem_ can be shrunk on-line depends on |
| 1249 |
the filesystem being used. Most filesystems do not support on-line |
| 1250 |
shrinking. XFS does not support shrinking at all. |
| 1251 |
|
| 1252 |
To shrink DRBD on-line, issue the following command _after_ you have |
| 1253 |
shrunk the file system residing on top of it: |
| 1254 |
|
| 1255 |
[source,drbd] |
| 1256 |
---------------------------- |
| 1257 |
# drbdadm resize --size=<new-size> <resource> |
| 1258 |
---------------------------- |
| 1259 |
|
| 1260 |
You may use the usual multiplier suffixes for _<new-size>_ (K, M, G |
| 1261 |
etc.). After you have shrunk DRBD, you may also shrink the containing |
| 1262 |
block device (if it supports shrinking). |
| 1263 |
|
| 1264 |
NOTE: It might be a good idea to issue `drbdadm resize _<resource>_` after |
| 1265 |
resizing the lower level device, so that the DRBD metadata *really* gets |
| 1266 |
written into the expected space at the end of the volume. |
| 1267 |
|
| 1268 |
[[s-shrinking-offline]] |
| 1269 |
==== Shrinking off-line |
| 1270 |
|
| 1271 |
indexterm:[resource]If you were to shrink a backing block device while |
| 1272 |
DRBD is inactive, DRBD would refuse to attach to this block device |
| 1273 |
during the next attach attempt, since it is now too small (in case |
| 1274 |
external meta data is used), or it would be unable to find its meta |
| 1275 |
data (in case internal meta data is used). To work around these |
| 1276 |
issues, use this procedure (if you cannot use |
| 1277 |
<<s-shrinking-online,on-line shrinking>>): |
| 1278 |
|
| 1279 |
|
| 1280 |
WARNING: This is an advanced procedure. Use at your own discretion. |
| 1281 |
|
| 1282 |
* Shrink the file system from one node, while DRBD is still |
| 1283 |
configured. |
| 1284 |
|
| 1285 |
* Unconfigure your DRBD resource: |
| 1286 |
|
| 1287 |
---------------------------- |
| 1288 |
# drbdadm down <resource> |
| 1289 |
---------------------------- |
| 1290 |
|
| 1291 |
* Save the meta data in a text file prior to shrinking: |
| 1292 |
|
| 1293 |
---------------------------- |
| 1294 |
# drbdadm dump-md <resource> > /tmp/metadata |
| 1295 |
---------------------------- |
| 1296 |
|
| 1297 |
You must do this on both nodes, using a separate dump file for every |
| 1298 |
node. *Do not* dump the meta data on one node, and simply copy the dump |
| 1299 |
file to the peer. *This. will. not. work.* |
| 1300 |
|
| 1301 |
* Shrink the backing block device on both nodes. |
| 1302 |
|
| 1303 |
* Adjust the size information (`la-size-sect`) in the file |
| 1304 |
`/tmp/metadata` accordingly, on both nodes. Remember that |
| 1305 |
`la-size-sect` must be specified in sectors. |
| 1306 |
|
| 1307 |
* *Only if you are using internal metadata* (which at this time have |
| 1308 |
probably been lost due to the shrinking process), re-initialize the |
| 1309 |
metadata area: |
| 1310 |
+ |
| 1311 |
---------------------------- |
| 1312 |
# drbdadm create-md <resource> |
| 1313 |
---------------------------- |
| 1314 |
|
| 1315 |
* Re-import the corrected meta data, on both nodes: |
| 1316 |
+ |
| 1317 |
---------------------------- |
| 1318 |
# drbdmeta_cmd=$(drbdadm -d dump-md <resource>) |
| 1319 |
# ${drbdmeta_cmd/dump-md/restore-md} /tmp/metadata |
| 1320 |
Valid meta-data in place, overwrite? [need to type 'yes' to confirm] |
| 1321 |
yes |
| 1322 |
Successfully restored meta data |
| 1323 |
---------------------------- |
| 1324 |
|
| 1325 |
NOTE: This example uses `bash` parameter substitution. It may or may not |
| 1326 |
work in other shells. Check your `SHELL` environment variable if you |
| 1327 |
are unsure which shell you are currently using. |
| 1328 |
|
| 1329 |
* Re-enable your DRBD resource: |
| 1330 |
+ |
| 1331 |
---------------------------- |
| 1332 |
# drbdadm up <resource> |
| 1333 |
---------------------------- |
| 1334 |
|
| 1335 |
|
| 1336 |
[[s-disable-flushes]] |
| 1337 |
=== Disabling backing device flushes |
| 1338 |
|
| 1339 |
CAUTION: You should only disable device flushes when running DRBD on |
| 1340 |
devices with a battery-backed write cache (BBWC). Most storage |
| 1341 |
controllers allow to automatically disable the write cache when the |
| 1342 |
battery is depleted, switching to write-through mode when the battery |
| 1343 |
dies. It is strongly recommended to enable such a feature. |
| 1344 |
|
| 1345 |
Disabling DRBD's flushes when running without BBWC, or on BBWC with a |
| 1346 |
depleted battery, is _likely to cause data loss_ and should not be |
| 1347 |
attempted. |
| 1348 |
|
| 1349 |
DRBD allows you to enable and disable <<s-disk-flush-support,backing |
| 1350 |
device flushes>> separately for the replicated data set and DRBD's own |
| 1351 |
meta data. Both of these options are enabled by default. If you wish |
| 1352 |
to disable either (or both), you would set this in the `disk` section |
| 1353 |
for the DRBD configuration file, `/etc/drbd.conf`. |
| 1354 |
|
| 1355 |
To disable disk flushes for the replicated data set, include the |
| 1356 |
following line in your configuration: |
| 1357 |
|
| 1358 |
[source,drbd] |
| 1359 |
---------------------------- |
| 1360 |
resource <resource> |
| 1361 |
disk { |
| 1362 |
disk-flushes no; |
| 1363 |
... |
| 1364 |
} |
| 1365 |
... |
| 1366 |
} |
| 1367 |
---------------------------- |
| 1368 |
|
| 1369 |
|
| 1370 |
To disable disk flushes on DRBD's meta data, include the following |
| 1371 |
line: |
| 1372 |
|
| 1373 |
[source,drbd] |
| 1374 |
---------------------------- |
| 1375 |
resource <resource> |
| 1376 |
disk { |
| 1377 |
md-flushes no; |
| 1378 |
... |
| 1379 |
} |
| 1380 |
... |
| 1381 |
} |
| 1382 |
---------------------------- |
| 1383 |
|
| 1384 |
After you have modified your resource configuration (and synchronized |
| 1385 |
your `/etc/drbd.conf` between nodes, of course), you may enable these |
| 1386 |
settings by issuing this command on both nodes: |
| 1387 |
|
| 1388 |
---------------------------- |
| 1389 |
# drbdadm adjust <resource> |
| 1390 |
---------------------------- |
| 1391 |
|
| 1392 |
|
| 1393 |
In case only one of the serves has a BBWCfootnote:[For example, in the DR site |
| 1394 |
you might be using different hardware, right?], you should move the setting into a host section, like this: |
| 1395 |
|
| 1396 |
[source,drbd] |
| 1397 |
---------------------------- |
| 1398 |
resource <resource> { |
| 1399 |
disk { |
| 1400 |
... common settings ... |
| 1401 |
} |
| 1402 |
|
| 1403 |
on host-1 { |
| 1404 |
disk { |
| 1405 |
md-flushes no; |
| 1406 |
} |
| 1407 |
... |
| 1408 |
} |
| 1409 |
... |
| 1410 |
} |
| 1411 |
---------------------------- |
| 1412 |
|
| 1413 |
|
| 1414 |
[[s-configure-split-brain-behavior]] |
| 1415 |
=== Configuring split brain behavior |
| 1416 |
|
| 1417 |
[[s-split-brain-notification]] |
| 1418 |
==== Split brain notification |
| 1419 |
|
| 1420 |
DRBD invokes the `split-brain` handler, if configured, at any time |
| 1421 |
split brain is _detected_. To configure this handler, add the |
| 1422 |
following item to your resource configuration: |
| 1423 |
|
| 1424 |
---------------------------- |
| 1425 |
resource <resource> |
| 1426 |
handlers { |
| 1427 |
split-brain <handler>; |
| 1428 |
... |
| 1429 |
} |
| 1430 |
... |
| 1431 |
} |
| 1432 |
---------------------------- |
| 1433 |
|
| 1434 |
_<handler>_ may be any executable present on the system. |
| 1435 |
|
| 1436 |
The DRBD distribution contains a split brain handler script that |
| 1437 |
installs as `/usr/lib/drbd/notify-split-brain.sh`. It simply sends a |
| 1438 |
notification e-mail message to a specified address. To configure the |
| 1439 |
handler to send a message to `root@localhost` (which is expected to be |
| 1440 |
an email address that forwards the notification to a real system |
| 1441 |
administrator), configure the `split-brain handler` as follows: |
| 1442 |
|
| 1443 |
---------------------------- |
| 1444 |
resource <resource> |
| 1445 |
handlers { |
| 1446 |
split-brain "/usr/lib/drbd/notify-split-brain.sh root"; |
| 1447 |
... |
| 1448 |
} |
| 1449 |
... |
| 1450 |
} |
| 1451 |
---------------------------- |
| 1452 |
|
| 1453 |
After you have made this modification on a running resource (and |
| 1454 |
synchronized the configuration file between nodes), no additional |
| 1455 |
intervention is needed to enable the handler. DRBD will simply invoke |
| 1456 |
the newly-configured handler on the next occurrence of split brain. |
| 1457 |
|
| 1458 |
[[s-automatic-split-brain-recovery-configuration]] |
| 1459 |
==== Automatic split brain recovery policies |
| 1460 |
|
| 1461 |
CAUTION: Configuring DRBD to automatically resolve data divergence |
| 1462 |
situations resulting from split-brain (or other) scenarios |
| 1463 |
is configuring for potential *automatic data loss*. |
| 1464 |
Understand the implications, and don't do it if you don't mean to. |
| 1465 |
|
| 1466 |
TIP: You rather want to look into fencing policies, quorum settings, |
| 1467 |
cluster manager integration, and redundant cluster manager communication links |
| 1468 |
to *avoid* data divergence in the first place. |
| 1469 |
|
| 1470 |
In order to be able to enable and configure DRBD's automatic split |
| 1471 |
brain recovery policies, you must understand that DRBD offers several |
| 1472 |
configuration options for this purpose. DRBD applies its split brain |
| 1473 |
recovery procedures based on the number of nodes in the Primary role |
| 1474 |
at the time the split brain is detected. To that end, DRBD examines |
| 1475 |
the following keywords, all found in the resource's `net` configuration |
| 1476 |
section: |
| 1477 |
|
| 1478 |
.`after-sb-0pri` |
| 1479 |
Split brain has just been detected, but at this time the resource is |
| 1480 |
not in the Primary role on any host. For this option, DRBD understands |
| 1481 |
the following keywords: |
| 1482 |
|
| 1483 |
* `disconnect`: Do not recover automatically, simply invoke the |
| 1484 |
`split-brain` handler script (if configured), drop the connection and |
| 1485 |
continue in disconnected mode. |
| 1486 |
|
| 1487 |
* `discard-younger-primary`: Discard and roll back the modifications |
| 1488 |
made on the host which assumed the Primary role last. |
| 1489 |
|
| 1490 |
* `discard-least-changes`: Discard and roll back the modifications on |
| 1491 |
the host where fewer changes occurred. |
| 1492 |
|
| 1493 |
* `discard-zero-changes`: If there is any host on which no changes |
| 1494 |
occurred at all, simply apply all modifications made on the other |
| 1495 |
and continue. |
| 1496 |
|
| 1497 |
.`after-sb-1pri` |
| 1498 |
Split brain has just been detected, and at this time the resource is |
| 1499 |
in the Primary role on one host. For this option, DRBD understands the |
| 1500 |
following keywords: |
| 1501 |
|
| 1502 |
* `disconnect`: As with `after-sb-0pri`, simply invoke the |
| 1503 |
`split-brain` handler script (if configured), drop the connection |
| 1504 |
and continue in disconnected mode. |
| 1505 |
|
| 1506 |
* `consensus`: Apply the same recovery policies as specified in |
| 1507 |
`after-sb-0pri`. If a split brain victim can be selected after |
| 1508 |
applying these policies, automatically resolve. Otherwise, behave |
| 1509 |
exactly as if `disconnect` were specified. |
| 1510 |
|
| 1511 |
* `call-pri-lost-after-sb`: Apply the recovery policies as specified |
| 1512 |
in `after-sb-0pri`. If a split brain victim can be selected after |
| 1513 |
applying these policies, invoke the `pri-lost-after-sb` handler on |
| 1514 |
the victim node. This handler must be configured in the |
| 1515 |
`handlers` section and is expected to forcibly remove the node from |
| 1516 |
the cluster. |
| 1517 |
|
| 1518 |
* `discard-secondary`: Whichever host is currently in the Secondary |
| 1519 |
role, make that host the split brain victim. |
| 1520 |
|
| 1521 |
.`after-sb-2pri` |
| 1522 |
Split brain has just been detected, and at this time the resource is |
| 1523 |
in the Primary role on both hosts. This option accepts the same |
| 1524 |
keywords as `after-sb-1pri` except `discard-secondary` and `consensus`. |
| 1525 |
|
| 1526 |
NOTE: DRBD understands additional keywords for these three options, |
| 1527 |
which have been omitted here because they are very rarely used. Refer |
| 1528 |
to the man page of `drbd.conf` for details on split brain recovery keywords not |
| 1529 |
discussed here. |
| 1530 |
|
| 1531 |
For example, a resource which serves as the block device for a GFS or |
| 1532 |
OCFS2 file system in dual-Primary mode may have its recovery policy |
| 1533 |
defined as follows: |
| 1534 |
|
| 1535 |
---------------------------- |
| 1536 |
resource <resource> { |
| 1537 |
handlers { |
| 1538 |
split-brain "/usr/lib/drbd/notify-split-brain.sh root" |
| 1539 |
... |
| 1540 |
} |
| 1541 |
net { |
| 1542 |
after-sb-0pri discard-zero-changes; |
| 1543 |
after-sb-1pri discard-secondary; |
| 1544 |
after-sb-2pri disconnect; |
| 1545 |
... |
| 1546 |
} |
| 1547 |
... |
| 1548 |
} |
| 1549 |
---------------------------- |
| 1550 |
|
| 1551 |
|
| 1552 |
[[s-three-nodes]] |
| 1553 |
=== Creating a stacked three-node setup |
| 1554 |
|
| 1555 |
A three-node setup involves one DRBD device _stacked_ atop another. |
| 1556 |
|
| 1557 |
NOTE: Stacking is deprecated in DRBD version |
| 1558 |
9.x, as more nodes can be implemented on a single level. See |
| 1559 |
<<s-drbdconf-conns>> for details. |
| 1560 |
|
| 1561 |
[[s-stacking-considerations]] |
| 1562 |
==== Device stacking considerations |
| 1563 |
|
| 1564 |
The following considerations apply to this type of setup: |
| 1565 |
|
| 1566 |
* The stacked device is the active one. Assume you have configured one |
| 1567 |
DRBD device `/dev/drbd0`, and the stacked device atop it is |
| 1568 |
`/dev/drbd10`, then `/dev/drbd10` will be the device that you mount |
| 1569 |
and use. |
| 1570 |
|
| 1571 |
* Device meta data will be stored twice, on the underlying DRBD device |
| 1572 |
_and_ the stacked DRBD device. On the stacked device, you must always |
| 1573 |
use <<s-internal-meta-data,internal meta data>>. This means that the |
| 1574 |
effectively available storage area on a stacked device is slightly |
| 1575 |
smaller, compared to an unstacked device. |
| 1576 |
|
| 1577 |
* To get the stacked upper level device running, the underlying device |
| 1578 |
must be in the primary role. |
| 1579 |
|
| 1580 |
* To be able to synchronize the backup node, the stacked device on the |
| 1581 |
active node must be up and in the primary role. |
| 1582 |
|
| 1583 |
|
| 1584 |
[[s-three-node-config]] |
| 1585 |
==== Configuring a stacked resource |
| 1586 |
|
| 1587 |
In the following example, nodes are named 'alice', 'bob', and |
| 1588 |
'charlie', with 'alice' and 'bob' forming a two-node cluster, and |
| 1589 |
'charlie' being the backup node. |
| 1590 |
|
| 1591 |
[source,drbd] |
| 1592 |
---------------------------- |
| 1593 |
resource r0 { |
| 1594 |
protocol C; |
| 1595 |
device /dev/drbd0; |
| 1596 |
disk /dev/sda6; |
| 1597 |
meta-disk internal; |
| 1598 |
|
| 1599 |
on alice { |
| 1600 |
address 10.0.0.1:7788; |
| 1601 |
} |
| 1602 |
|
| 1603 |
on bob { |
| 1604 |
address 10.0.0.2:7788; |
| 1605 |
} |
| 1606 |
} |
| 1607 |
|
| 1608 |
resource r0-U { |
| 1609 |
protocol A; |
| 1610 |
|
| 1611 |
stacked-on-top-of r0 { |
| 1612 |
device /dev/drbd10; |
| 1613 |
address 192.168.42.1:7789; |
| 1614 |
} |
| 1615 |
|
| 1616 |
on charlie { |
| 1617 |
device /dev/drbd10; |
| 1618 |
disk /dev/hda6; |
| 1619 |
address 192.168.42.2:7789; # Public IP of the backup node |
| 1620 |
meta-disk internal; |
| 1621 |
} |
| 1622 |
} |
| 1623 |
---------------------------- |
| 1624 |
|
| 1625 |
As with any `drbd.conf` configuration file, this must be distributed |
| 1626 |
across all nodes in the cluster -- in this case, three nodes. Notice |
| 1627 |
the following extra keyword not found in an unstacked resource |
| 1628 |
configuration: |
| 1629 |
|
| 1630 |
.`stacked-on-top-of` |
| 1631 |
This option informs DRBD that the resource which contains it is a |
| 1632 |
stacked resource. It replaces one of the `on` sections normally found |
| 1633 |
in any resource configuration. Do not use `stacked-on-top-of` in an |
| 1634 |
lower-level resource. |
| 1635 |
|
| 1636 |
|
| 1637 |
NOTE: It is not a requirement to use <<fp-protocol-a,Protocol A>> for |
| 1638 |
stacked resources. You may select any of DRBD's replication protocols |
| 1639 |
depending on your application. |
| 1640 |
|
| 1641 |
[[f-single-stacked]] |
| 1642 |
.Single stacked setup |
| 1643 |
image::images/single-stacked.svg[] |
| 1644 |
|
| 1645 |
|
| 1646 |
[[s-three-node-enable]] |
| 1647 |
==== Enabling stacked resources |
| 1648 |
|
| 1649 |
To enable a stacked resource, you first enable its lower-level |
| 1650 |
resource and promote it: |
| 1651 |
---------------------------- |
| 1652 |
drbdadm up r0 |
| 1653 |
drbdadm primary r0 |
| 1654 |
---------------------------- |
| 1655 |
|
| 1656 |
As with unstacked resources, you must create DRBD meta data on the |
| 1657 |
stacked resources. This is done using the following command: |
| 1658 |
|
| 1659 |
---------------------------- |
| 1660 |
# drbdadm create-md --stacked r0-U |
| 1661 |
---------------------------- |
| 1662 |
|
| 1663 |
Then, you may enable the stacked resource: |
| 1664 |
|
| 1665 |
---------------------------- |
| 1666 |
# drbdadm up --stacked r0-U |
| 1667 |
# drbdadm primary --stacked r0-U |
| 1668 |
---------------------------- |
| 1669 |
|
| 1670 |
After this, you may bring up the resource on the backup node, enabling |
| 1671 |
three-node replication: |
| 1672 |
|
| 1673 |
---------------------------- |
| 1674 |
# drbdadm create-md r0-U |
| 1675 |
# drbdadm up r0-U |
| 1676 |
---------------------------- |
| 1677 |
|
| 1678 |
In order to automate stacked resource management, you may integrate |
| 1679 |
stacked resources in your cluster manager configuration. |
| 1680 |
ifdef::drbd-only[] |
| 1681 |
See <<s-pacemaker-stacked-resources>> for information on doing this in a |
| 1682 |
cluster managed by the Pacemaker cluster management framework. |
| 1683 |
endif::drbd-only[] |
| 1684 |
|
| 1685 |
[[s-permanently-diskless-nodes]] |
| 1686 |
=== Permanently diskless nodes |
| 1687 |
|
| 1688 |
A node might be permanently diskless in DRBD. Here is a configuration example |
| 1689 |
showing a resource with 3 diskful nodes (servers) and one permanently diskless |
| 1690 |
node (client). |
| 1691 |
|
| 1692 |
------------------------------------- |
| 1693 |
resource kvm-mail { |
| 1694 |
device /dev/drbd6; |
| 1695 |
disk /dev/vg/kvm-mail; |
| 1696 |
meta-disk internal; |
| 1697 |
|
| 1698 |
on store1 { |
| 1699 |
address 10.1.10.1:7006; |
| 1700 |
node-id 0; |
| 1701 |
} |
| 1702 |
on store2 { |
| 1703 |
address 10.1.10.2:7006; |
| 1704 |
node-id 1; |
| 1705 |
} |
| 1706 |
on store3 { |
| 1707 |
address 10.1.10.3:7006; |
| 1708 |
node-id 2; |
| 1709 |
} |
| 1710 |
|
| 1711 |
on for-later-rebalancing { |
| 1712 |
address 10.1.10.4:7006; |
| 1713 |
node-id 3; |
| 1714 |
} |
| 1715 |
|
| 1716 |
# DRBD "client" |
| 1717 |
floating 10.1.11.6:8006 { |
| 1718 |
disk none; |
| 1719 |
node-id 4; |
| 1720 |
} |
| 1721 |
|
| 1722 |
# rest omitted for brevity |
| 1723 |
... |
| 1724 |
} |
| 1725 |
------------------------------------- |
| 1726 |
|
| 1727 |
For permanently diskless nodes no bitmap slot gets allocated. For such nodes the |
| 1728 |
diskless status is displayed in green color since it is not an error or unexpected |
| 1729 |
state. See <<s-drbd-client-internals>> for internal details. |
| 1730 |
|
| 1731 |
[[s-rebalance-workflow]] |
| 1732 |
=== Data rebalancing |
| 1733 |
|
| 1734 |
indexterm:[rebalance]Given the (example) policy that data needs to be available on 3 nodes, |
| 1735 |
you need at least 3 servers for your setup. |
| 1736 |
|
| 1737 |
Now, as your storage demands grow, you will encounter the need for |
| 1738 |
additional servers. Rather than having to buy 3 more servers at the same |
| 1739 |
time, you can _rebalance_ your data across a single additional node. |
| 1740 |
|
| 1741 |
|
| 1742 |
.DRBD data rebalancing |
| 1743 |
image::images/rebalance.svg[] |
| 1744 |
|
| 1745 |
In the figure above you can see the _before_ and _after_ states: from |
| 1746 |
3 nodes with three 25TiB volumes each (for a net 75TiB), to 4 nodes, |
| 1747 |
with net 100TiB. |
| 1748 |
|
| 1749 |
To redistribute the data across your cluster you have to choose a _new_ node, |
| 1750 |
and one where you want to remove this DRBD resource. + |
| 1751 |
Please note that removing the resource from a currently _active_ node (i.e. |
| 1752 |
where DRBD is _Primary_) will involve either migrating the service or running |
| 1753 |
this resource on this node as a <<s-drbd-client,DRBD client>>; it's easier to |
| 1754 |
choose a node in _Secondary_ role. (Of course, that might not always be possible.) |
| 1755 |
|
| 1756 |
|
| 1757 |
[[s-prepare_a_bitmap_slot]] |
| 1758 |
==== Prepare a bitmap slot |
| 1759 |
|
| 1760 |
You will need to have a free <<s-quick-sync-bitmap,bitmap slot>> for |
| 1761 |
temporary use, on each of the nodes that have the resource that is to be moved. |
| 1762 |
|
| 1763 |
You can allocate one more at |
| 1764 |
<<s-first-time-up,`drbdadm create-md` time>>, or simply put a placeholder in |
| 1765 |
your configuration, so that `drbdadm` sees that it should reserve one more slot: |
| 1766 |
|
| 1767 |
--------- |
| 1768 |
resource r0 { |
| 1769 |
... |
| 1770 |
on for-later-rebalancing { |
| 1771 |
address 10.254.254.254:65533; |
| 1772 |
node-id 3; |
| 1773 |
} |
| 1774 |
} |
| 1775 |
--------- |
| 1776 |
|
| 1777 |
[NOTE] |
| 1778 |
================= |
| 1779 |
If you need to make that slot available during live use, |
| 1780 |
you will have to |
| 1781 |
|
| 1782 |
. dump the metadata, |
| 1783 |
. enlarge the metadata space, |
| 1784 |
. edit the dumpfile, |
| 1785 |
. load the changed metadata. |
| 1786 |
|
| 1787 |
In a future version `drbdadm` will have a shortcut for you; most |
| 1788 |
probably you'll be able to say `drbdadm resize --peers _N_` and have the |
| 1789 |
kernel rewrite the metadata for you. |
| 1790 |
================= |
| 1791 |
|
| 1792 |
|
| 1793 |
[[s-preparing_and_activating_the_new_node]] |
| 1794 |
==== Preparing and activating the new node |
| 1795 |
|
| 1796 |
First of all you have to |
| 1797 |
create the underlying storage volume on the new node (using e.g. `lvcreate`). |
| 1798 |
Then the placeholder in the configuration can be filled with the correct host |
| 1799 |
name, address, and storage path. Now copy the resource configuration to all |
| 1800 |
relevant nodes. |
| 1801 |
|
| 1802 |
|
| 1803 |
On the new node initialize the meta-data (once) by doing |
| 1804 |
|
| 1805 |
------------- |
| 1806 |
# drbdadm create-md <resource> |
| 1807 |
v09 Magic number not found |
| 1808 |
Writing meta data... |
| 1809 |
initialising activity log |
| 1810 |
NOT initializing bitmap |
| 1811 |
New drbd meta data block successfully created. |
| 1812 |
------------- |
| 1813 |
|
| 1814 |
|
| 1815 |
[[s-starting_the_initial_sync]] |
| 1816 |
==== Starting the initial sync |
| 1817 |
|
| 1818 |
Now the new node needs to get the data. |
| 1819 |
|
| 1820 |
This is done by defining the network connection on the existing nodes via |
| 1821 |
|
| 1822 |
------------------------------------- |
| 1823 |
# drbdadm adjust <resource> |
| 1824 |
------------------------------------- |
| 1825 |
|
| 1826 |
and starting the DRBD device on the new node via |
| 1827 |
|
| 1828 |
------------------------------------- |
| 1829 |
# drbdadm up <resource> |
| 1830 |
------------------------------------- |
| 1831 |
|
| 1832 |
|
| 1833 |
[[s-check_connectivity]] |
| 1834 |
==== Check connectivity |
| 1835 |
|
| 1836 |
At this time please do a |
| 1837 |
------------ |
| 1838 |
# drbdadm status <resource> |
| 1839 |
------------ |
| 1840 |
|
| 1841 |
on the new node, and check that _all_ other nodes are connected. |
| 1842 |
|
| 1843 |
[[s-after_the_initial_sync]] |
| 1844 |
==== After the initial sync |
| 1845 |
|
| 1846 |
As soon as the new host is _UpToDate_, one of the |
| 1847 |
other nodes in the configuration can be renamed to `for-later-rebalancing`, and |
| 1848 |
kept for another migration. |
| 1849 |
|
| 1850 |
NOTE: Perhaps you want to comment the section; although that has the risk that |
| 1851 |
doing a `drbdadm create-md` for a new node has too few bitmap slots for the |
| 1852 |
next rebalancing. + |
| 1853 |
It might be easier to use a reserved (unused) IP address and host name. |
| 1854 |
|
| 1855 |
Copy the changed configuration around again, and use it by running |
| 1856 |
|
| 1857 |
------------------------------------- |
| 1858 |
# drbdadm adjust <resource> |
| 1859 |
------------------------------------- |
| 1860 |
|
| 1861 |
on all nodes. |
| 1862 |
|
| 1863 |
|
| 1864 |
[[s-cleaning_up]] |
| 1865 |
==== Cleaning up |
| 1866 |
|
| 1867 |
On the one node that had the data up to now, but isn't used anymore for this |
| 1868 |
resource, you can now take the DRBD device down by starting |
| 1869 |
|
| 1870 |
------------------- |
| 1871 |
# drbdadm down <resource> |
| 1872 |
------------------- |
| 1873 |
|
| 1874 |
Now the lower level storage device isn't used anymore, and can either be |
| 1875 |
re-used for other purposes or, if it is a logical volume, its space can be |
| 1876 |
returned to the volume group via `lvremove`. |
| 1877 |
|
| 1878 |
[[s-conclusion_and_further_steps]] |
| 1879 |
==== Conclusion and further steps |
| 1880 |
|
| 1881 |
One of the resources has been migrated to the new node. The same could be done |
| 1882 |
for one or more other resources, to free space on two or three nodes in the |
| 1883 |
existing cluster. |
| 1884 |
|
| 1885 |
Then new resources can be configured, as there are enough nodes with free space |
| 1886 |
to achieve 3-way redundancy again. |
| 1887 |
|
| 1888 |
[[s-configuring-quorum]] |
| 1889 |
=== Configuring quorum |
| 1890 |
|
| 1891 |
In order to avoid split brain or diverging data of replicas one has to configure |
| 1892 |
fencing. All the options for fencing rely on redundant communication in the end. |
| 1893 |
That might be in the form of a management network that connects the nodes to the IPMI |
| 1894 |
network interfaces of the peer machines. In case of the crm-fence-peer script it |
| 1895 |
is necessary that Pacemakers communication stays available when DRBD's network |
| 1896 |
link breaks. |
| 1897 |
|
| 1898 |
The quorum mechanism on the other hand takes a completely difference approach. The basic |
| 1899 |
idea is that a cluster partition may only modify the replicated data set if the |
| 1900 |
number of nodes that can communicate is greater than the half of the overall number |
| 1901 |
of nodes. A node of such a partition _has quorum_. |
| 1902 |
On the other hand a node does not have quorum needs to guarantee that the replicated |
| 1903 |
data set it not touched, that it does not create a diverging data set. |
| 1904 |
|
| 1905 |
The quorum implementation in DRBD gets enabled by setting the `quorum` resource |
| 1906 |
option to `majority`, `all` or a numeric value. Where `majority` selects the |
| 1907 |
behavior that was described in the previous paragraph. |
| 1908 |
|
| 1909 |
[[s-guaranteed_minimal_redundancy]] |
| 1910 |
==== Guaranteed minimal redundancy |
| 1911 |
|
| 1912 |
By default every node with a disk gets a vote in the quorum election. In other words |
| 1913 |
only diskless nodes do not count. So a partition with two _Inconsistent_ disks gets |
| 1914 |
quorum, while a partition with one _UpToDate_ node will have quorum in a 3 node |
| 1915 |
cluster. By configuring `quorum-minimum-redundancy` this behavior can be changed so that |
| 1916 |
only nodes that are `UpToDate` have a vote in the quorum election. The option takes the |
| 1917 |
same arguments as the `quorum` option. |
| 1918 |
|
| 1919 |
With this option you express that you rather want to wait until eventually necessary |
| 1920 |
resync operations finish before any services start. So in a way you prefer |
| 1921 |
that the minimal redundancy of your data is guaranteed over the availability of your |
| 1922 |
service. Financial data and services is an example that comes to mind. |
| 1923 |
|
| 1924 |
Consider following example for a 5 node cluster. It requires a partitions to have at |
| 1925 |
least 3 nodes, and two of them must be `UpToDate`: |
| 1926 |
|
| 1927 |
------------------------------------- |
| 1928 |
resource quorum-demo { |
| 1929 |
options { |
| 1930 |
quorum majority; |
| 1931 |
quorum-minimum-redundancy 2; |
| 1932 |
... |
| 1933 |
} |
| 1934 |
} |
| 1935 |
------------------------------------- |
| 1936 |
|
| 1937 |
[[s-actions_on_loss_of_quorum]] |
| 1938 |
==== Actions on loss of quorum |
| 1939 |
|
| 1940 |
When a node that is running the service loses quorum it needs to cease write-operations |
| 1941 |
on the data set immediately. That means that IO immediately starts to complete all IO |
| 1942 |
requests with errors. Usually that means that a graceful shutdown is not possible, since |
| 1943 |
that would require more modifications to the data set. The IO errors propagate from |
| 1944 |
the block level to the file system and from the file system to the user space application(s). |
| 1945 |
|
| 1946 |
Ideally the application simply terminates in case of IO errors. This allows then Pacemaker |
| 1947 |
to unmount the filesystem and to demote the DRBD resource to secondary role. |
| 1948 |
If that is true you should set the `on-no-quorum` resource option to `io-error`. |
| 1949 |
Here is an example: |
| 1950 |
|
| 1951 |
------------------------------------- |
| 1952 |
resource quorum-demo { |
| 1953 |
options { |
| 1954 |
quorum majority; |
| 1955 |
on-no-quorum io-error; |
| 1956 |
... |
| 1957 |
} |
| 1958 |
} |
| 1959 |
------------------------------------- |
| 1960 |
|
| 1961 |
If you application does not terminate on the first IO error, you can choose to freeze IO |
| 1962 |
instead and to reboot the node. Here is a configuration example: |
| 1963 |
|
| 1964 |
------------------------------------- |
| 1965 |
resource quorum-demo { |
| 1966 |
options { |
| 1967 |
quorum majority; |
| 1968 |
on-no-quorum suspend-io; |
| 1969 |
... |
| 1970 |
} |
| 1971 |
|
| 1972 |
handlers { |
| 1973 |
quorum-lost "echo b > /proc/sysrq-trigger"; |
| 1974 |
} |
| 1975 |
... |
| 1976 |
} |
| 1977 |
------------------------------------- |
| 1978 |
|
| 1979 |
[[s-configuring-quorum-tiebreaker]] |
| 1980 |
==== Using a diskless node as a tiebreaker |
| 1981 |
|
| 1982 |
A diskless node with connections to all nodes in a cluster can be utilized to break ties |
| 1983 |
in the quorum negotiation process. |
| 1984 |
|
| 1985 |
Consider the following two-node cluster, where node A is the primary and node B is a secondary: |
| 1986 |
|
| 1987 |
image::images/quorum-tiebreaker-without.svg[] |
| 1988 |
|
| 1989 |
As soon as the connection between the two nodes is interrupted, they lose quorum |
| 1990 |
and the application on top of the cluster cannot write data anymore. |
| 1991 |
|
| 1992 |
image::images/quorum-tiebreaker-without-disconnect.svg[] |
| 1993 |
|
| 1994 |
Now if we add a third node, C, to the cluster and configure it as diskless, we |
| 1995 |
can take advantage of the tiebreaker mechanism. |
| 1996 |
|
| 1997 |
image::images/quorum-tiebreaker.svg[] |
| 1998 |
|
| 1999 |
In this case, when the primary and secondary nodes lose connection, they can |
| 2000 |
still "see" the diskless tiebreaker. Because of this, the primary can continue |
| 2001 |
working, while the secondary demotes its disk to Outdated and the service can |
| 2002 |
therefore not be migrated there. |
| 2003 |
|
| 2004 |
image::images/quorum-tiebreaker-disconnect.svg[] |
| 2005 |
|
| 2006 |
There are a few special cases in case two connections fail. Consider the following |
| 2007 |
scenario: |
| 2008 |
|
| 2009 |
image::images/quorum-tiebreaker-disconnect-case2.svg[] |
| 2010 |
|
| 2011 |
In this case, the tiebreaker node forms a partition with the primary node. The primary |
| 2012 |
therefore keeps quorum, while the secondary becomes outdated. Note that the secondary's |
| 2013 |
disk state will be "UpToDate", but regardless it cannot be promoted to primary because |
| 2014 |
it lacks quorum. |
| 2015 |
|
| 2016 |
Let's consider the possibility of the primary losing connection to the tiebreaker: |
| 2017 |
|
| 2018 |
image::images/quorum-tiebreaker-disconnect-case3.svg[] |
| 2019 |
|
| 2020 |
In this case, the primary becomes unusable and goes into a "quorum suspended" state. This |
| 2021 |
effectively results in the application on top of DRBD receiving I/O errors. A cluster manager |
| 2022 |
could then promote node B to primary and keep the service running there instead. |
| 2023 |
|
| 2024 |
In case the diskless tiebreaker "switches sides", we need to avoid data divergence. Consider |
| 2025 |
this scenario: |
| 2026 |
|
| 2027 |
image::images/quorum-tiebreaker-disconnect-case1a.svg[] |
| 2028 |
|
| 2029 |
The connection between the primary and secondary has failed, and the application is continuing |
| 2030 |
to run on the primary, when the primary suddenly loses its connection to the diskless node. |
| 2031 |
|
| 2032 |
In this case, no node can be promoted to primary and the cluster cannot continue to operate. |
| 2033 |
|
| 2034 |
[NOTE] |
| 2035 |
==================== |
| 2036 |
Protecting from data divergence always takes priority over ensuring service availability. |
| 2037 |
==================== |
| 2038 |
|
| 2039 |
Let's look at another scenario: |
| 2040 |
|
| 2041 |
image::images/quorum-tiebreaker-disconnect-case2a.svg[] |
| 2042 |
|
| 2043 |
Here, the application is running on the primary, while the secondary is unavailable. Then, |
| 2044 |
the tiebreaker first loses connection to the primary, and then reconnects to the secondary. |
| 2045 |
It is important to note here that *a node that has lost quorum cannot regain quorum by connecting |
| 2046 |
to a diskless node*. Thus, in this case, no node has quorum and the cluster halts. |
| 2047 |
|
| 2048 |
[[s-quorum_last_man_standing]] |
| 2049 |
==== Last man standing |
| 2050 |
|
| 2051 |
It needs to be mentioned that nodes that leave a cluster gracefully are counted |
| 2052 |
different than failed nodes. In this context _leaving gracefully_ means that the leaving |
| 2053 |
node marked it's data as Outdated, and that it was able to tell the remaining nodes |
| 2054 |
that it's data is Outdated. |
| 2055 |
|
| 2056 |
In a group of nodes where all disks are Outdated no one in that group can promote to Primary |
| 2057 |
footnote:[The exception is a manual promotion with the --force flag. We assume that |
| 2058 |
one that uses the --force knows what he is doing.]. |
| 2059 |
|
| 2060 |
An implication is that, if a cluster is left by all but one node, it can keep quorum as |
| 2061 |
long as all the others left in a graceful way. If a single node leaves ungraceful, |
| 2062 |
it has to assume that all away nodes form a partition and have access to up-to-date |
| 2063 |
data. If the other partition might (potentially) be larger than my own partition it |
| 2064 |
looses quorum. |
| 2065 |
|
| 2066 |
[[s-removing-DRBD]] |
| 2067 |
=== Removing DRBD |
| 2068 |
|
| 2069 |
For the unlikely case that you want to remove DRBD, here are the necessary steps. |
| 2070 |
|
| 2071 |
. Stop the services and unmount the filesystems on top of the DRBD volumes. In case you are |
| 2072 |
using a cluster manager, make sure it ceases to control the services fist. |
| 2073 |
. Stop the DRBD resource(s) by using `drbdadm down <res>` or `drbdadm down all` |
| 2074 |
.. In case the DRBD resource was using internal meta-data you might choose to |
| 2075 |
resize the filesystem to cover all of the backing device's space. This step |
| 2076 |
effectively removes DRBD's meta-data. This is an action that can not be reversed |
| 2077 |
easily. You can do that with `resize2fs <backing_dev>` for ext[234] family of file systems. |
| 2078 |
It supports resizing of unmounted filesystem and under certain conditions also online |
| 2079 |
grow. XFS can be grown online only with the `xfs_growfs` command. |
| 2080 |
. Mount the backing device(s) directly, start the services on top of them |
| 2081 |
. Unload the drbd kernel driver modules with `rmmod drbd_transport_tcp` and `rmmod drbd`. |
| 2082 |
. Uninstall the drbd software packages. |
| 2083 |
|
| 2084 |
//- list divider. Keep it otherwise next included section is interpreted as list element |