| 1 |
[[ch-internals]] |
| 2 |
== DRBD Internals |
| 3 |
|
| 4 |
This chapter gives _some_ background information about some of DRBD's |
| 5 |
internal algorithms and structures. It is intended for interested |
| 6 |
users wishing to gain a certain degree of background knowledge about |
| 7 |
DRBD. It does not dive into DRBD's inner workings deep enough to be a |
| 8 |
reference for DRBD developers. For that purpose, please refer to the |
| 9 |
papers listed in <<s-publications>>, and of course to the comments in |
| 10 |
the DRBD source code. |
| 11 |
|
| 12 |
[[s-metadata]] |
| 13 |
=== DRBD meta data |
| 14 |
|
| 15 |
indexterm:[meta data]DRBD stores various pieces of information about |
| 16 |
the data it keeps in a dedicated area. This metadata includes: |
| 17 |
|
| 18 |
* the size of the DRBD device, |
| 19 |
* the Generation Identifier (GI, described in detail in <<s-gi>>), |
| 20 |
* the Activity Log (AL, described in detail in <<s-activity-log>>). |
| 21 |
* the quick-sync bitmap (described in detail in <<s-quick-sync-bitmap>>), |
| 22 |
|
| 23 |
This metadata may be stored _internally_ or _externally_. Which method |
| 24 |
is used is configurable on a per-resource basis. |
| 25 |
|
| 26 |
[[s-internal-meta-data]] |
| 27 |
==== Internal meta data |
| 28 |
|
| 29 |
indexterm:[meta data]Configuring a resource to use internal meta data |
| 30 |
means that DRBD stores its meta data on the same physical lower-level |
| 31 |
device as the actual production data. It does so by setting aside an |
| 32 |
area at the _end_ of the device for the specific purpose of storing |
| 33 |
metadata. |
| 34 |
|
| 35 |
.Advantage |
| 36 |
Since the meta data are inextricably linked with the actual data, no |
| 37 |
special action is required from the administrator in case of a hard |
| 38 |
disk failure. The meta data are lost together with the actual data and |
| 39 |
are also restored together. |
| 40 |
|
| 41 |
.Disadvantage |
| 42 |
In case of the lower-level device being a single physical hard disk |
| 43 |
(as opposed to a RAID set), internal meta data may negatively affect |
| 44 |
write throughput. The performance of write requests by the application |
| 45 |
may trigger an update of the meta data in DRBD. If the meta data are |
| 46 |
stored on the same magnetic disk of a hard disk, the write operation |
| 47 |
may result in two additional movements of the write/read head of the |
| 48 |
hard disk. |
| 49 |
|
| 50 |
[CAUTION] |
| 51 |
============ |
| 52 |
If you are planning to use internal meta data in conjunction |
| 53 |
with an existing lower-level device that already has data which you |
| 54 |
wish to preserve, you _must_ account for the space required by DRBD's |
| 55 |
meta data. |
| 56 |
|
| 57 |
Otherwise, upon DRBD resource creation, the newly created metadata |
| 58 |
would overwrite data at the end of the lower-level device, potentially |
| 59 |
destroying existing files in the process. |
| 60 |
============ |
| 61 |
|
| 62 |
To avoid that, you must do one of the following things: |
| 63 |
|
| 64 |
* Enlarge your lower-level device. This is possible with any logical |
| 65 |
volume management facility (such as indexterm:[LVM]LVM) as long as |
| 66 |
you have free space available in the corresponding volume group. It |
| 67 |
may also be supported by hardware storage solutions. |
| 68 |
|
| 69 |
* Shrink your existing file system on your lower-level device. This |
| 70 |
may or may not be supported by your file system. |
| 71 |
|
| 72 |
* If neither of the two are possible, use |
| 73 |
<<s-external-meta-data,external meta data>> instead. |
| 74 |
|
| 75 |
To estimate the amount by which you must enlarge your lower-level |
| 76 |
device or shrink your file system, see <<s-meta-data-size>>. |
| 77 |
|
| 78 |
[[s-external-meta-data]] |
| 79 |
==== External meta data |
| 80 |
|
| 81 |
indexterm:[meta data]External meta data is simply stored on a |
| 82 |
separate, dedicated block device distinct from that which holds your |
| 83 |
production data. |
| 84 |
|
| 85 |
.Advantage |
| 86 |
For some write operations, using external meta data produces a |
| 87 |
somewhat improved latency behavior. |
| 88 |
|
| 89 |
.Disadvantage |
| 90 |
Meta data are not inextricably linked with the actual production |
| 91 |
data. This means that manual intervention is required in the case of a |
| 92 |
hardware failure destroying just the production data (but not DRBD |
| 93 |
meta data), to effect a full data sync from the surviving node onto |
| 94 |
the subsequently replaced disk. |
| 95 |
|
| 96 |
Use of external meta data is also the only viable option if _all_ of |
| 97 |
the following apply: |
| 98 |
|
| 99 |
* You are using DRBD to duplicate an existing device that already |
| 100 |
contains data you wish to preserve, _and_ |
| 101 |
|
| 102 |
* that existing device does not support enlargement, _and_ |
| 103 |
|
| 104 |
* the existing file system on the device does not support shrinking. |
| 105 |
|
| 106 |
To estimate the required size of the block device dedicated to hold |
| 107 |
your device meta data, see <<s-meta-data-size>>. |
| 108 |
|
| 109 |
NOTE: External meta data requires a minimum of a 1MB device size. |
| 110 |
|
| 111 |
[[s-meta-data-size]] |
| 112 |
==== Estimating meta data size |
| 113 |
|
| 114 |
indexterm:[meta data]You may calculate the exact space requirements |
| 115 |
for DRBD's meta data using the following formula: |
| 116 |
|
| 117 |
[[eq-metadata-size-exact]] |
| 118 |
.Calculating DRBD meta data size (exactly) |
| 119 |
image::images/metadata-size-exact.svg[] |
| 120 |
|
| 121 |
*_C~s~_* is the data device size in sectors, and *_N_* is the number of peers. |
| 122 |
|
| 123 |
//// |
| 124 |
If |
| 125 |
you are using the <<al-stripe,al-stripes>> setting, additional space of size |
| 126 |
_al-stripes_ times _al-strip-size_ is required. |
| 127 |
//// |
| 128 |
|
| 129 |
NOTE: You may retrieve the device size (in bytes) by issuing `blockdev --getsize64 |
| 130 |
<device>`; to convert to MB, divide by 1048576 (= 2^20^ or 1024^2^). |
| 131 |
|
| 132 |
In practice, you may use a reasonably good approximation, given |
| 133 |
below. Note that in this formula, the unit is megabytes, not sectors: |
| 134 |
|
| 135 |
[[eq-metadata-size-approx]] |
| 136 |
.Estimating DRBD meta data size (approximately) |
| 137 |
image::images/metadata-size-approx.svg[] |
| 138 |
|
| 139 |
[[s-gi]] |
| 140 |
=== Generation Identifiers |
| 141 |
|
| 142 |
indexterm:[generation identifiers]DRBD uses _generation identifiers_ |
| 143 |
(GIs) to identify "generations" of replicated data. |
| 144 |
|
| 145 |
This is DRBD's internal mechanism used for |
| 146 |
|
| 147 |
* determining whether the two nodes are in fact members of the same |
| 148 |
cluster (as opposed to two nodes that were connected accidentally), |
| 149 |
|
| 150 |
* determining the direction of background re-synchronization (if |
| 151 |
necessary), |
| 152 |
|
| 153 |
* determining whether full re-synchronization is necessary or whether |
| 154 |
partial re-synchronization is sufficient, |
| 155 |
|
| 156 |
* indexterm:[split brain]identifying split brain. |
| 157 |
|
| 158 |
[[s-data-generations]] |
| 159 |
==== Data generations |
| 160 |
|
| 161 |
DRBD marks the start of a new _data generation_ at each of the |
| 162 |
following occurrences: |
| 163 |
|
| 164 |
// FIXME PRe |
| 165 |
|
| 166 |
* The initial device full sync, |
| 167 |
|
| 168 |
* a disconnected resource switching to the primary role, |
| 169 |
|
| 170 |
* a resource in the primary role disconnecting. |
| 171 |
|
| 172 |
Thus, we can summarize that whenever a resource is in the _Connected_ |
| 173 |
connection state, and both nodes' disk state is _UpToDate_, the |
| 174 |
current data generation on both nodes is the same. The inverse is also |
| 175 |
true. Note that the current implementation uses the lowest bit to encode the |
| 176 |
role of the node (Primary/Secondary). Therefore, the lowest bit might be |
| 177 |
different on distinct nodes even if they are considered to have the same data |
| 178 |
generation. |
| 179 |
|
| 180 |
Every new data generation is identified by an 8-byte, universally |
| 181 |
unique identifier (UUID). |
| 182 |
|
| 183 |
[[s-gi-tuple]] |
| 184 |
==== The generation identifier tuple |
| 185 |
|
| 186 |
DRBD keeps some pieces of information about current and historical |
| 187 |
data generations in the local resource meta data: |
| 188 |
|
| 189 |
.Current UUID |
| 190 |
This is the generation identifier for the current data generation, as |
| 191 |
seen from the local node's perspective. When a resource is |
| 192 |
_Connected_ and fully synchronized, the current UUID is identical |
| 193 |
between nodes. |
| 194 |
|
| 195 |
.Bitmap UUIDs |
| 196 |
This is the UUID of the generation against which this on-disk |
| 197 |
bitmap is tracking changes (per remote host). Like the on-disk sync bitmap itself, this |
| 198 |
identifier is only relevant while the remote host is disconnected. |
| 199 |
|
| 200 |
// If the resource is _Connected_, this UUID is always empty (zero). FIXME |
| 201 |
|
| 202 |
.Historical UUIDs |
| 203 |
These are the identifiers of data generations preceding the |
| 204 |
current one, sized to have one slot per (possible) remote host. |
| 205 |
|
| 206 |
Collectively, these items are referred to as the _generation |
| 207 |
identifier tuple_, or "__GI tuple__" for short. |
| 208 |
|
| 209 |
[[s-gi-changes]] |
| 210 |
==== How generation identifiers change |
| 211 |
|
| 212 |
[[s-gi-changes-newgen]] |
| 213 |
===== Start of a new data generation |
| 214 |
|
| 215 |
When a node in _Primary_ role loses connection to its peer (either by network failure or |
| 216 |
manual intervention), DRBD modifies its local generation identifiers |
| 217 |
in the following manner: |
| 218 |
|
| 219 |
[[f-gi-changes-newgen]] |
| 220 |
.GI tuple changes at start of a new data generation |
| 221 |
image::images/gi-changes-newgen.svg[] |
| 222 |
|
| 223 |
. The primary creates a new UUID for the new data generation. This becomes the |
| 224 |
_new_ current UUID for the primary node. |
| 225 |
|
| 226 |
. The _previous_ current UUID now refers to the generation the bitmap is |
| 227 |
tracking changes against, so it becomes the new bitmap UUID for the |
| 228 |
primary node. |
| 229 |
|
| 230 |
. On the secondary node(s), the GI tuple remains unchanged. |
| 231 |
|
| 232 |
|
| 233 |
[[s-gi-changes-synccomplete]] |
| 234 |
===== Completion of re-synchronization |
| 235 |
|
| 236 |
When re-synchronization concludes, the synchronization target adopts the entire |
| 237 |
GI tuple from the synchronization source. |
| 238 |
|
| 239 |
The synchronization source keeps the same set, and doesn't generate new UUIDs. |
| 240 |
|
| 241 |
|
| 242 |
[[s-gi-use]] |
| 243 |
==== How DRBD uses generation identifiers |
| 244 |
|
| 245 |
When a connection between nodes is established, the two nodes exchange |
| 246 |
their currently available generation identifiers, and proceed |
| 247 |
accordingly. A number of possible outcomes exist: |
| 248 |
|
| 249 |
.Current UUIDs empty on both nodes |
| 250 |
The local node detects that both its current UUID and the peer's |
| 251 |
current UUID are empty. This is the normal occurrence for a freshly |
| 252 |
configured resource that has not had the initial full sync |
| 253 |
initiated. No synchronization takes place; it has to be started |
| 254 |
manually. |
| 255 |
|
| 256 |
.Current UUIDs empty on one node |
| 257 |
The local node detects that the peer's current UUID is empty, and its |
| 258 |
own is not. This is the normal case for a freshly configured resource |
| 259 |
on which the initial full sync has just been initiated, the local node |
| 260 |
having been selected as the initial synchronization source. DRBD now |
| 261 |
sets all bits in the on-disk sync bitmap (meaning it considers the |
| 262 |
entire device out-of-sync), and starts synchronizing as a |
| 263 |
synchronization source. In the opposite case (local current UUID |
| 264 |
empty, peer's non-empty), DRBD performs the same steps, except that |
| 265 |
the local node becomes the synchronization target. |
| 266 |
|
| 267 |
.Equal current UUIDs |
| 268 |
The local node detects that its current UUID and the peer's current |
| 269 |
UUID are non-empty and equal. This is the normal occurrence for a |
| 270 |
resource that went into disconnected mode at a time when it was in the |
| 271 |
secondary role, and was not promoted on either node while |
| 272 |
disconnected. No synchronization takes place, as none is necessary. |
| 273 |
|
| 274 |
.Bitmap UUID matches peer's current UUID |
| 275 |
The local node detects that its bitmap UUID matches the peer's current |
| 276 |
UUID, and that the peer's bitmap UUID is empty. This is the normal and |
| 277 |
expected occurrence after a secondary node failure, with the local |
| 278 |
node being in the primary role. It means that the peer never became |
| 279 |
primary in the meantime and worked on the basis of the same data |
| 280 |
generation all along. DRBD now initiates a normal, background |
| 281 |
re-synchronization, with the local node becoming the synchronization |
| 282 |
source. If, conversely, the local node detects that _its_ bitmap UUID |
| 283 |
is empty, and that the _peer's_ bitmap matches the local node's current |
| 284 |
UUID, then that is the normal and expected occurrence after a failure |
| 285 |
of the local node. Again, DRBD now initiates a normal, background |
| 286 |
re-synchronization, with the local node becoming the synchronization |
| 287 |
target. |
| 288 |
|
| 289 |
.Current UUID matches peer's historical UUID |
| 290 |
The local node detects that its current UUID matches one of the peer's |
| 291 |
historical UUIDs. This implies that while the two data sets share a |
| 292 |
common ancestor, and the peer node has the up-to-date data, the |
| 293 |
information kept in the peer node's bitmap is outdated and not |
| 294 |
usable. Thus, a normal synchronization would be insufficient. DRBD |
| 295 |
now marks the entire device as out-of-sync and initiates a full |
| 296 |
background re-synchronization, with the local node becoming the |
| 297 |
synchronization target. In the opposite case (one of the local node's |
| 298 |
historical UUID matches the peer's current UUID), DRBD performs the |
| 299 |
same steps, except that the local node becomes the synchronization |
| 300 |
source. |
| 301 |
|
| 302 |
// ERROR: FIXME |
| 303 |
|
| 304 |
.Bitmap UUIDs match, current UUIDs do not |
| 305 |
indexterm:[split brain]The local node detects that its current UUID |
| 306 |
differs from the peer's current UUID, and that the bitmap UUIDs |
| 307 |
match. This is split brain, but one where the data generations have |
| 308 |
the same parent. This means that DRBD invokes split brain |
| 309 |
auto-recovery strategies, if configured. Otherwise, DRBD disconnects |
| 310 |
and waits for manual split brain resolution. |
| 311 |
|
| 312 |
.Neither current nor bitmap UUIDs match |
| 313 |
The local node detects that its current UUID differs from the peer's |
| 314 |
current UUID, and that the bitmap UUIDs _do not_ match. This is split |
| 315 |
brain with unrelated ancestor generations, thus auto-recovery |
| 316 |
strategies, even if configured, are moot. DRBD disconnects and waits |
| 317 |
for manual split brain resolution. |
| 318 |
|
| 319 |
.No UUIDs match |
| 320 |
Finally, in case DRBD fails to detect even a single matching element |
| 321 |
in the two nodes' GI tuples, it logs a warning about unrelated data |
| 322 |
and disconnects. This is DRBD's safeguard against accidental |
| 323 |
connection of two cluster nodes that have never heard of each other |
| 324 |
before. |
| 325 |
|
| 326 |
|
| 327 |
[[s-activity-log]] |
| 328 |
=== The Activity Log |
| 329 |
|
| 330 |
[[s-al-purpose]] |
| 331 |
==== Purpose |
| 332 |
|
| 333 |
indexterm:[Activity Log]During a write operation DRBD forwards the |
| 334 |
write operation to the local backing block device, but also sends the |
| 335 |
data block over the network. These two actions occur, for all |
| 336 |
practical purposes, simultaneously. Random timing behavior may cause a |
| 337 |
situation where the write operation has been completed, but the |
| 338 |
transmission via the network has not yet taken place, or vice versa. |
| 339 |
|
| 340 |
If, at this moment, the active node fails and fail-over is being |
| 341 |
initiated, then this data block is out of sync between nodes -- it has |
| 342 |
been written on the failed node prior to the crash, but replication |
| 343 |
has not yet completed. Thus, when the node eventually recovers, this |
| 344 |
block must be removed from the data set during subsequent |
| 345 |
synchronization. Otherwise, the crashed node would be "one write |
| 346 |
ahead" of the surviving node, which would violate the "all or |
| 347 |
nothing" principle of replicated storage. This is an issue that is not |
| 348 |
limited to DRBD, in fact, this issue exists in practically all |
| 349 |
replicated storage configurations. Many other storage solutions (just |
| 350 |
as DRBD itself, prior to version 0.7) thus require that after a |
| 351 |
failure of the active node the data must be fully synchronized after |
| 352 |
its recovery. |
| 353 |
|
| 354 |
DRBD's approach, since version 0.7, is a different one. The _activity |
| 355 |
log_ (AL), stored in the meta data area, keeps track of those blocks |
| 356 |
that have "recently" been written to. Colloquially, these areas are |
| 357 |
referred to as _hot extents_. |
| 358 |
|
| 359 |
If a temporarily failed node that was in active mode at the time of |
| 360 |
failure is synchronized, only those hot extents highlighted in the AL |
| 361 |
need to be synchronized (plus any blocks marked in the bitmap on the now-active peer), |
| 362 |
rather than the full device. This drastically |
| 363 |
reduces synchronization time after an active node crash. |
| 364 |
|
| 365 |
[[s-active-extents]] |
| 366 |
==== Active extents |
| 367 |
|
| 368 |
indexterm:[Activity Log]The activity log has a configurable parameter, |
| 369 |
the number of active extents. Every active extent adds 4MiB to the |
| 370 |
amount of data being retransmitted after a Primary crash. This |
| 371 |
parameter must be understood as a compromise between the following |
| 372 |
opposites: |
| 373 |
|
| 374 |
.Many active extents |
| 375 |
Keeping a large activity log improves write throughput. Every time a |
| 376 |
new extent is activated, an old extent is reset to inactive. This |
| 377 |
transition requires a write operation to the meta data area. If the |
| 378 |
number of active extents is high, old active extents are swapped out |
| 379 |
fairly rarely, reducing meta data write operations and thereby |
| 380 |
improving performance. |
| 381 |
|
| 382 |
.Few active extents |
| 383 |
Keeping a small activity log reduces synchronization time after active |
| 384 |
node failure and subsequent recovery. |
| 385 |
|
| 386 |
|
| 387 |
[[s-suitable-al-size]] |
| 388 |
==== Selecting a suitable Activity Log size |
| 389 |
|
| 390 |
indexterm:[Activity Log]Consideration of the number of extents should |
| 391 |
be based on the desired synchronization time at a given |
| 392 |
synchronization rate. The number of active extents can be calculated |
| 393 |
as follows: |
| 394 |
|
| 395 |
[[eq-al-extents]] |
| 396 |
.Active extents calculation based on sync rate and target sync time |
| 397 |
image::images/al-extents.svg[] |
| 398 |
|
| 399 |
_R_ is the synchronization rate, given in MiB/s. _t~sync~_ is the target |
| 400 |
synchronization time, in seconds. _E_ is the resulting number of active |
| 401 |
extents. |
| 402 |
|
| 403 |
To provide an example, suppose the cluster has an I/O subsystem with a |
| 404 |
throughput rate of 200 MiByte/s that was configured to a |
| 405 |
synchronization rate (_R_) of 60 MiByte/s, and we want to keep the |
| 406 |
target synchronization time (_t~sync~_) at 4 minutes or 240 seconds: |
| 407 |
|
| 408 |
[[eq-al-extents-example]] |
| 409 |
.Active extents calculation based on sync rate and target sync time (example) |
| 410 |
image::images/al-extents-example.svg[] |
| 411 |
|
| 412 |
On a final note, DRBD 9 needs to keep an AL even on the Secondary nodes, as |
| 413 |
their data might be used to synchronize other Secondary nodes. |
| 414 |
|
| 415 |
|
| 416 |
[[s-quick-sync-bitmap]] |
| 417 |
=== The quick-sync bitmap |
| 418 |
|
| 419 |
indexterm:[quick-sync bitmap]indexterm:[bitmap (DRBD-specific |
| 420 |
concept)]The quick-sync bitmap is the internal data structure which |
| 421 |
DRBD uses, on a per-resource per-peer basis, to keep track of blocks being in |
| 422 |
sync (identical on both nodes) or out-of sync. It is only relevant |
| 423 |
when a resource is in disconnected mode. |
| 424 |
|
| 425 |
In the quick-sync bitmap, one bit represents a 4-KiB chunk of on-disk |
| 426 |
data. If the bit is cleared, it means that the corresponding block is |
| 427 |
still in sync with the peer node. That implies that the block has not |
| 428 |
been written to since the time of disconnection. Conversely, if the |
| 429 |
bit is set, it means that the block has been modified and needs to be |
| 430 |
re-synchronized whenever the connection becomes available again. |
| 431 |
|
| 432 |
As DRBD detects write I/O on a disconnected device, and hence starts |
| 433 |
setting bits in the quick-sync bitmap, it does so in RAM -- thus |
| 434 |
avoiding expensive synchronous metadata I/O operations. Only when the |
| 435 |
corresponding blocks turn cold (that is, expire from the |
| 436 |
<<s-activity-log,Activity Log>>), DRBD makes the appropriate |
| 437 |
modifications in an on-disk representation of the quick-sync |
| 438 |
bitmap. Likewise, if the resource happens to be manually shut down on |
| 439 |
the remaining node while disconnected, DRBD flushes the |
| 440 |
_complete_ quick-sync bitmap out to persistent storage. |
| 441 |
|
| 442 |
When the peer node recovers or the connection is re-established, DRBD |
| 443 |
combines the bitmap information from both nodes to determine the |
| 444 |
_total data set_ that it must re-synchronize. Simultaneously, DRBD |
| 445 |
<<s-gi-use,examines the generation identifiers>> to determine the |
| 446 |
_direction_ of synchronization. |
| 447 |
|
| 448 |
The node acting as the synchronization source then transmits the |
| 449 |
agreed-upon blocks to the peer node, clearing sync bits in the bitmap |
| 450 |
as the synchronization target acknowledges the modifications. If the |
| 451 |
re-synchronization is now interrupted (by another network outage, for |
| 452 |
example) and subsequently resumed it will continue where it left off |
| 453 |
-- with any additional blocks modified in the meantime being added to |
| 454 |
the re-synchronization data set, of course. |
| 455 |
|
| 456 |
NOTE: Re-synchronization may be also be paused and resumed manually |
| 457 |
with the indexterm:[drbdadm, pause-sync]`drbdadm pause-sync` and |
| 458 |
indexterm:[drbdadm, resume-sync]`drbdadm resume-sync` commands. You |
| 459 |
should, however, not do so light-heartedly -- interrupting |
| 460 |
re-synchronization leaves your secondary node's disk |
| 461 |
_Inconsistent_ longer than necessary. |
| 462 |
|
| 463 |
[[s-fence-peer]] |
| 464 |
=== The Peer-Fencing interface |
| 465 |
|
| 466 |
DRBD has an interface defined for fencingfootnote:[For a discussion about |
| 467 |
Fencing and STONITH, please see the corresponding Pacemaker page |
| 468 |
http://clusterlabs.org/doc/crm_fencing.html.] the peer |
| 469 |
node in case of the replication link being interrupted. The `fence-peer` |
| 470 |
should mark the disk(s) on the peer node as _Outdated_, or shut down |
| 471 |
the peer node. It has to fulfill these tasks under the assumption that |
| 472 |
the replication network is down. |
| 473 |
|
| 474 |
The fencing helper is invoked only in case |
| 475 |
|
| 476 |
. a `fence-peer` handler has been defined in the resource's (or `common`) |
| 477 |
`handlers` section, _and_ |
| 478 |
|
| 479 |
. the `fencing` option for the resource is set to either |
| 480 |
`resource-only` or `resource-and-stonith`, _and_ |
| 481 |
|
| 482 |
. the node was primary _and_ the replication link is interrupted long enough |
| 483 |
for DRBDfootnote:[That means e.g. a TCP timeout, the `ping-timeout`, or |
| 484 |
the kernel triggers a connection abort e.g. because the network link goes down.] |
| 485 |
to detect a network failure. _or_ |
| 486 |
|
| 487 |
. the node should promote to primary _and_ is not connected to the peer _and_ |
| 488 |
the peer's disks are not already marked as _Outdated_. |
| 489 |
|
| 490 |
|
| 491 |
The program or script specified as the `fence-peer` handler, when it is |
| 492 |
invoked, has the `DRBD_RESOURCE` and `DRBD_PEER` environment variables |
| 493 |
available. They contain the name of the affected DRBD resource and the |
| 494 |
peer's hostname, respectively. |
| 495 |
|
| 496 |
Any peer fencing helper program (or script) must return one of the |
| 497 |
following exit codes: |
| 498 |
|
| 499 |
.`fence-peer` handler exit codes |
| 500 |
[format="csv",separator=";",options="header"] |
| 501 |
|======================================= |
| 502 |
Exit code;Implication |
| 503 |
3;Peer's disk state was already _Inconsistent_. |
| 504 |
4;Peer's disk state was successfully set to _Outdated_ (or was _Outdated_ to begin with). |
| 505 |
5;Connection to the peer node failed, peer could not be reached. |
| 506 |
6;Peer refused to be outdated because the affected resource was in the primary role. |
| 507 |
7;Peer node was successfully fenced off the cluster. This should never occur unless `fencing` is set to `resource-and-stonith` for the affected resource. |
| 508 |
|======================================= |
| 509 |
|
| 510 |
[[s-drbd-client-internals]] |
| 511 |
=== The client mode |
| 512 |
|
| 513 |
Since version 9.0.13 DRBD supports clients. A client in DRBD speak is |
| 514 |
just a permanently diskless node. In the configuration, it is |
| 515 |
expressed by using the keyword `none` for the backing block device |
| 516 |
(the `disk` keyword). You will notice that in the `drbdsetup status` |
| 517 |
output you will see the `Diskless` disk status displayed in green |
| 518 |
color. (Usually, a disk state of `Diskless` is displayed in red). |
| 519 |
|
| 520 |
Internally all the peers of an intentional diskless node are |
| 521 |
configured with the `peer-device-option` `--bitmap=no`. That means |
| 522 |
that they will not allocate a bitmap slot in the meta-data for the |
| 523 |
intentional diskless peer. On the intentional diskless node the device |
| 524 |
gets marked with the option `--diskless=yes` while it is created with |
| 525 |
the `new-minor` sub-command of `drbdsetup`. |
| 526 |
|
| 527 |
These flags are visible through the `events2` status command: |
| 528 |
|
| 529 |
* a `device` might have the `client:` field. If it reports `yes` the |
| 530 |
local device was marked to be permanently diskless. |
| 531 |
|
| 532 |
* a `peer-device` might have the `peer-client` filed. If it says `yes` |
| 533 |
then there is no change-tracking bitmap to that peer. |
| 534 |
|
| 535 |
Relevant commands and implications: |
| 536 |
|
| 537 |
* You can only run `drbdsetup peer-device-options --bitmap=yes ...` if |
| 538 |
bitmap slots are available in the meta-data, since a bitmap-slot |
| 539 |
gets allocated. |
| 540 |
|
| 541 |
* The command `drbdsetup peer-device-options --bitmap=no ...` is only |
| 542 |
possible if the peer is diskless, it does _not_ unallocate the |
| 543 |
bitmap-slot. |
| 544 |
|
| 545 |
* `drbdsetup forget-peer ...` is used to irrevocable free the |
| 546 |
bitmap-slot assigned to a certain peer. |
| 547 |
|
| 548 |
* Connecting two peers with disk where one (or both) expect the peer |
| 549 |
to be permanently diskless fails. |
| 550 |
|
| 551 |
//- list divider. Keep it otherwise next included section is interpreted as list element |