Debian Codegraph @

UG9/en/internals.adoc

raw
1 [[ch-internals]]
2 == DRBD Internals
3
4 This chapter gives _some_ background information about some of DRBD's
5 internal algorithms and structures. It is intended for interested
6 users wishing to gain a certain degree of background knowledge about
7 DRBD. It does not dive into DRBD's inner workings deep enough to be a
8 reference for DRBD developers. For that purpose, please refer to the
9 papers listed in <<s-publications>>, and of course to the comments in
10 the DRBD source code.
11
12 [[s-metadata]]
13 === DRBD meta data
14
15 indexterm:[meta data]DRBD stores various pieces of information about
16 the data it keeps in a dedicated area. This metadata includes:
17
18 * the size of the DRBD device,
19 * the Generation Identifier (GI, described in detail in <<s-gi>>),
20 * the Activity Log (AL, described in detail in <<s-activity-log>>).
21 * the quick-sync bitmap (described in detail in <<s-quick-sync-bitmap>>),
22
23 This metadata may be stored _internally_ or _externally_. Which method
24 is used is configurable on a per-resource basis.
25
26 [[s-internal-meta-data]]
27 ==== Internal meta data
28
29 indexterm:[meta data]Configuring a resource to use internal meta data
30 means that DRBD stores its meta data on the same physical lower-level
31 device as the actual production data. It does so by setting aside an
32 area at the _end_ of the device for the specific purpose of storing
33 metadata.
34
35 .Advantage
36 Since the meta data are inextricably linked with the actual data, no
37 special action is required from the administrator in case of a hard
38 disk failure. The meta data are lost together with the actual data and
39 are also restored together.
40
41 .Disadvantage
42 In case of the lower-level device being a single physical hard disk
43 (as opposed to a RAID set), internal meta data may negatively affect
44 write throughput. The performance of write requests by the application
45 may trigger an update of the meta data in DRBD. If the meta data are
46 stored on the same magnetic disk of a hard disk, the write operation
47 may result in two additional movements of the write/read head of the
48 hard disk.
49
50 [CAUTION]
51 ============
52 If you are planning to use internal meta data in conjunction
53 with an existing lower-level device that already has data which you
54 wish to preserve, you _must_ account for the space required by DRBD's
55 meta data.
56
57 Otherwise, upon DRBD resource creation, the newly created metadata
58 would overwrite data at the end of the lower-level device, potentially
59 destroying existing files in the process.
60 ============
61
62 To avoid that, you must do one of the following things:
63
64 * Enlarge your lower-level device. This is possible with any logical
65 volume management facility (such as indexterm:[LVM]LVM) as long as
66 you have free space available in the corresponding volume group. It
67 may also be supported by hardware storage solutions.
68
69 * Shrink your existing file system on your lower-level device. This
70 may or may not be supported by your file system.
71
72 * If neither of the two are possible, use
73 <<s-external-meta-data,external meta data>> instead.
74
75 To estimate the amount by which you must enlarge your lower-level
76 device or shrink your file system, see <<s-meta-data-size>>.
77
78 [[s-external-meta-data]]
79 ==== External meta data
80
81 indexterm:[meta data]External meta data is simply stored on a
82 separate, dedicated block device distinct from that which holds your
83 production data.
84
85 .Advantage
86 For some write operations, using external meta data produces a
87 somewhat improved latency behavior.
88
89 .Disadvantage
90 Meta data are not inextricably linked with the actual production
91 data. This means that manual intervention is required in the case of a
92 hardware failure destroying just the production data (but not DRBD
93 meta data), to effect a full data sync from the surviving node onto
94 the subsequently replaced disk.
95
96 Use of external meta data is also the only viable option if _all_ of
97 the following apply:
98
99 * You are using DRBD to duplicate an existing device that already
100 contains data you wish to preserve, _and_
101
102 * that existing device does not support enlargement, _and_
103
104 * the existing file system on the device does not support shrinking.
105
106 To estimate the required size of the block device dedicated to hold
107 your device meta data, see <<s-meta-data-size>>.
108
109 NOTE: External meta data requires a minimum of a 1MB device size.
110
111 [[s-meta-data-size]]
112 ==== Estimating meta data size
113
114 indexterm:[meta data]You may calculate the exact space requirements
115 for DRBD's meta data using the following formula:
116
117 [[eq-metadata-size-exact]]
118 .Calculating DRBD meta data size (exactly)
119 image::images/metadata-size-exact.svg[]
120
121 *_C~s~_* is the data device size in sectors, and *_N_* is the number of peers.
122
123 ////
124 If
125 you are using the <<al-stripe,al-stripes>> setting, additional space of size
126 _al-stripes_ times _al-strip-size_ is required.
127 ////
128
129 NOTE: You may retrieve the device size (in bytes) by issuing `blockdev --getsize64
130 <device>`; to convert to MB, divide by 1048576 (= 2^20^ or 1024^2^).
131
132 In practice, you may use a reasonably good approximation, given
133 below. Note that in this formula, the unit is megabytes, not sectors:
134
135 [[eq-metadata-size-approx]]
136 .Estimating DRBD meta data size (approximately)
137 image::images/metadata-size-approx.svg[]
138
139 [[s-gi]]
140 === Generation Identifiers
141
142 indexterm:[generation identifiers]DRBD uses _generation identifiers_
143 (GIs) to identify "generations" of replicated data.
144
145 This is DRBD's internal mechanism used for
146
147 * determining whether the two nodes are in fact members of the same
148 cluster (as opposed to two nodes that were connected accidentally),
149
150 * determining the direction of background re-synchronization (if
151 necessary),
152
153 * determining whether full re-synchronization is necessary or whether
154 partial re-synchronization is sufficient,
155
156 * indexterm:[split brain]identifying split brain.
157
158 [[s-data-generations]]
159 ==== Data generations
160
161 DRBD marks the start of a new _data generation_ at each of the
162 following occurrences:
163
164 // FIXME PRe
165
166 * The initial device full sync,
167
168 * a disconnected resource switching to the primary role,
169
170 * a resource in the primary role disconnecting.
171
172 Thus, we can summarize that whenever a resource is in the _Connected_
173 connection state, and both nodes' disk state is _UpToDate_, the
174 current data generation on both nodes is the same. The inverse is also
175 true. Note that the current implementation uses the lowest bit to encode the
176 role of the node (Primary/Secondary). Therefore, the lowest bit might be
177 different on distinct nodes even if they are considered to have the same data
178 generation.
179
180 Every new data generation is identified by an 8-byte, universally
181 unique identifier (UUID).
182
183 [[s-gi-tuple]]
184 ==== The generation identifier tuple
185
186 DRBD keeps some pieces of information about current and historical
187 data generations in the local resource meta data:
188
189 .Current UUID
190 This is the generation identifier for the current data generation, as
191 seen from the local node's perspective. When a resource is
192 _Connected_ and fully synchronized, the current UUID is identical
193 between nodes.
194
195 .Bitmap UUIDs
196 This is the UUID of the generation against which this on-disk
197 bitmap is tracking changes (per remote host). Like the on-disk sync bitmap itself, this
198 identifier is only relevant while the remote host is disconnected.
199
200 // If the resource is _Connected_, this UUID is always empty (zero). FIXME
201
202 .Historical UUIDs
203 These are the identifiers of data generations preceding the
204 current one, sized to have one slot per (possible) remote host.
205
206 Collectively, these items are referred to as the _generation
207 identifier tuple_, or "__GI tuple__" for short.
208
209 [[s-gi-changes]]
210 ==== How generation identifiers change
211
212 [[s-gi-changes-newgen]]
213 ===== Start of a new data generation
214
215 When a node in _Primary_ role loses connection to its peer (either by network failure or
216 manual intervention), DRBD modifies its local generation identifiers
217 in the following manner:
218
219 [[f-gi-changes-newgen]]
220 .GI tuple changes at start of a new data generation
221 image::images/gi-changes-newgen.svg[]
222
223 . The primary creates a new UUID for the new data generation. This becomes the
224 _new_ current UUID for the primary node.
225
226 . The _previous_ current UUID now refers to the generation the bitmap is
227 tracking changes against, so it becomes the new bitmap UUID for the
228 primary node.
229
230 . On the secondary node(s), the GI tuple remains unchanged.
231
232
233 [[s-gi-changes-synccomplete]]
234 ===== Completion of re-synchronization
235
236 When re-synchronization concludes, the synchronization target adopts the entire
237 GI tuple from the synchronization source.
238
239 The synchronization source keeps the same set, and doesn't generate new UUIDs.
240
241
242 [[s-gi-use]]
243 ==== How DRBD uses generation identifiers
244
245 When a connection between nodes is established, the two nodes exchange
246 their currently available generation identifiers, and proceed
247 accordingly. A number of possible outcomes exist:
248
249 .Current UUIDs empty on both nodes
250 The local node detects that both its current UUID and the peer's
251 current UUID are empty. This is the normal occurrence for a freshly
252 configured resource that has not had the initial full sync
253 initiated. No synchronization takes place; it has to be started
254 manually.
255
256 .Current UUIDs empty on one node
257 The local node detects that the peer's current UUID is empty, and its
258 own is not. This is the normal case for a freshly configured resource
259 on which the initial full sync has just been initiated, the local node
260 having been selected as the initial synchronization source. DRBD now
261 sets all bits in the on-disk sync bitmap (meaning it considers the
262 entire device out-of-sync), and starts synchronizing as a
263 synchronization source. In the opposite case (local current UUID
264 empty, peer's non-empty), DRBD performs the same steps, except that
265 the local node becomes the synchronization target.
266
267 .Equal current UUIDs
268 The local node detects that its current UUID and the peer's current
269 UUID are non-empty and equal. This is the normal occurrence for a
270 resource that went into disconnected mode at a time when it was in the
271 secondary role, and was not promoted on either node while
272 disconnected. No synchronization takes place, as none is necessary.
273
274 .Bitmap UUID matches peer's current UUID
275 The local node detects that its bitmap UUID matches the peer's current
276 UUID, and that the peer's bitmap UUID is empty. This is the normal and
277 expected occurrence after a secondary node failure, with the local
278 node being in the primary role. It means that the peer never became
279 primary in the meantime and worked on the basis of the same data
280 generation all along. DRBD now initiates a normal, background
281 re-synchronization, with the local node becoming the synchronization
282 source. If, conversely, the local node detects that _its_ bitmap UUID
283 is empty, and that the _peer's_ bitmap matches the local node's current
284 UUID, then that is the normal and expected occurrence after a failure
285 of the local node. Again, DRBD now initiates a normal, background
286 re-synchronization, with the local node becoming the synchronization
287 target.
288
289 .Current UUID matches peer's historical UUID
290 The local node detects that its current UUID matches one of the peer's
291 historical UUIDs. This implies that while the two data sets share a
292 common ancestor, and the peer node has the up-to-date data, the
293 information kept in the peer node's bitmap is outdated and not
294 usable. Thus, a normal synchronization would be insufficient. DRBD
295 now marks the entire device as out-of-sync and initiates a full
296 background re-synchronization, with the local node becoming the
297 synchronization target. In the opposite case (one of the local node's
298 historical UUID matches the peer's current UUID), DRBD performs the
299 same steps, except that the local node becomes the synchronization
300 source.
301
302 // ERROR: FIXME
303
304 .Bitmap UUIDs match, current UUIDs do not
305 indexterm:[split brain]The local node detects that its current UUID
306 differs from the peer's current UUID, and that the bitmap UUIDs
307 match. This is split brain, but one where the data generations have
308 the same parent. This means that DRBD invokes split brain
309 auto-recovery strategies, if configured. Otherwise, DRBD disconnects
310 and waits for manual split brain resolution.
311
312 .Neither current nor bitmap UUIDs match
313 The local node detects that its current UUID differs from the peer's
314 current UUID, and that the bitmap UUIDs _do not_ match. This is split
315 brain with unrelated ancestor generations, thus auto-recovery
316 strategies, even if configured, are moot. DRBD disconnects and waits
317 for manual split brain resolution.
318
319 .No UUIDs match
320 Finally, in case DRBD fails to detect even a single matching element
321 in the two nodes' GI tuples, it logs a warning about unrelated data
322 and disconnects. This is DRBD's safeguard against accidental
323 connection of two cluster nodes that have never heard of each other
324 before.
325
326
327 [[s-activity-log]]
328 === The Activity Log
329
330 [[s-al-purpose]]
331 ==== Purpose
332
333 indexterm:[Activity Log]During a write operation DRBD forwards the
334 write operation to the local backing block device, but also sends the
335 data block over the network. These two actions occur, for all
336 practical purposes, simultaneously. Random timing behavior may cause a
337 situation where the write operation has been completed, but the
338 transmission via the network has not yet taken place, or vice versa.
339
340 If, at this moment, the active node fails and fail-over is being
341 initiated, then this data block is out of sync between nodes -- it has
342 been written on the failed node prior to the crash, but replication
343 has not yet completed. Thus, when the node eventually recovers, this
344 block must be removed from the data set during subsequent
345 synchronization. Otherwise, the crashed node would be "one write
346 ahead" of the surviving node, which would violate the "all or
347 nothing" principle of replicated storage. This is an issue that is not
348 limited to DRBD, in fact, this issue exists in practically all
349 replicated storage configurations. Many other storage solutions (just
350 as DRBD itself, prior to version 0.7) thus require that after a
351 failure of the active node the data must be fully synchronized after
352 its recovery.
353
354 DRBD's approach, since version 0.7, is a different one. The _activity
355 log_ (AL), stored in the meta data area, keeps track of those blocks
356 that have "recently" been written to. Colloquially, these areas are
357 referred to as _hot extents_.
358
359 If a temporarily failed node that was in active mode at the time of
360 failure is synchronized, only those hot extents highlighted in the AL
361 need to be synchronized (plus any blocks marked in the bitmap on the now-active peer),
362 rather than the full device. This drastically
363 reduces synchronization time after an active node crash.
364
365 [[s-active-extents]]
366 ==== Active extents
367
368 indexterm:[Activity Log]The activity log has a configurable parameter,
369 the number of active extents. Every active extent adds 4MiB to the
370 amount of data being retransmitted after a Primary crash. This
371 parameter must be understood as a compromise between the following
372 opposites:
373
374 .Many active extents
375 Keeping a large activity log improves write throughput. Every time a
376 new extent is activated, an old extent is reset to inactive. This
377 transition requires a write operation to the meta data area. If the
378 number of active extents is high, old active extents are swapped out
379 fairly rarely, reducing meta data write operations and thereby
380 improving performance.
381
382 .Few active extents
383 Keeping a small activity log reduces synchronization time after active
384 node failure and subsequent recovery.
385
386
387 [[s-suitable-al-size]]
388 ==== Selecting a suitable Activity Log size
389
390 indexterm:[Activity Log]Consideration of the number of extents should
391 be based on the desired synchronization time at a given
392 synchronization rate. The number of active extents can be calculated
393 as follows:
394
395 [[eq-al-extents]]
396 .Active extents calculation based on sync rate and target sync time
397 image::images/al-extents.svg[]
398
399 _R_ is the synchronization rate, given in MiB/s. _t~sync~_ is the target
400 synchronization time, in seconds. _E_ is the resulting number of active
401 extents.
402
403 To provide an example, suppose the cluster has an I/O subsystem with a
404 throughput rate of 200 MiByte/s that was configured to a
405 synchronization rate (_R_) of 60 MiByte/s, and we want to keep the
406 target synchronization time (_t~sync~_) at 4 minutes or 240 seconds:
407
408 [[eq-al-extents-example]]
409 .Active extents calculation based on sync rate and target sync time (example)
410 image::images/al-extents-example.svg[]
411
412 On a final note, DRBD 9 needs to keep an AL even on the Secondary nodes, as
413 their data might be used to synchronize other Secondary nodes.
414
415
416 [[s-quick-sync-bitmap]]
417 === The quick-sync bitmap
418
419 indexterm:[quick-sync bitmap]indexterm:[bitmap (DRBD-specific
420 concept)]The quick-sync bitmap is the internal data structure which
421 DRBD uses, on a per-resource per-peer basis, to keep track of blocks being in
422 sync (identical on both nodes) or out-of sync. It is only relevant
423 when a resource is in disconnected mode.
424
425 In the quick-sync bitmap, one bit represents a 4-KiB chunk of on-disk
426 data. If the bit is cleared, it means that the corresponding block is
427 still in sync with the peer node. That implies that the block has not
428 been written to since the time of disconnection. Conversely, if the
429 bit is set, it means that the block has been modified and needs to be
430 re-synchronized whenever the connection becomes available again.
431
432 As DRBD detects write I/O on a disconnected device, and hence starts
433 setting bits in the quick-sync bitmap, it does so in RAM -- thus
434 avoiding expensive synchronous metadata I/O operations. Only when the
435 corresponding blocks turn cold (that is, expire from the
436 <<s-activity-log,Activity Log>>), DRBD makes the appropriate
437 modifications in an on-disk representation of the quick-sync
438 bitmap. Likewise, if the resource happens to be manually shut down on
439 the remaining node while disconnected, DRBD flushes the
440 _complete_ quick-sync bitmap out to persistent storage.
441
442 When the peer node recovers or the connection is re-established, DRBD
443 combines the bitmap information from both nodes to determine the
444 _total data set_ that it must re-synchronize. Simultaneously, DRBD
445 <<s-gi-use,examines the generation identifiers>> to determine the
446 _direction_ of synchronization.
447
448 The node acting as the synchronization source then transmits the
449 agreed-upon blocks to the peer node, clearing sync bits in the bitmap
450 as the synchronization target acknowledges the modifications. If the
451 re-synchronization is now interrupted (by another network outage, for
452 example) and subsequently resumed it will continue where it left off
453 -- with any additional blocks modified in the meantime being added to
454 the re-synchronization data set, of course.
455
456 NOTE: Re-synchronization may be also be paused and resumed manually
457 with the indexterm:[drbdadm, pause-sync]`drbdadm pause-sync` and
458 indexterm:[drbdadm, resume-sync]`drbdadm resume-sync` commands. You
459 should, however, not do so light-heartedly -- interrupting
460 re-synchronization leaves your secondary node's disk
461 _Inconsistent_ longer than necessary.
462
463 [[s-fence-peer]]
464 === The Peer-Fencing interface
465
466 DRBD has an interface defined for fencingfootnote:[For a discussion about
467 Fencing and STONITH, please see the corresponding Pacemaker page
468 http://clusterlabs.org/doc/crm_fencing.html.] the peer
469 node in case of the replication link being interrupted. The `fence-peer`
470 should mark the disk(s) on the peer node as _Outdated_, or shut down
471 the peer node. It has to fulfill these tasks under the assumption that
472 the replication network is down.
473
474 The fencing helper is invoked only in case
475
476 . a `fence-peer` handler has been defined in the resource's (or `common`)
477 `handlers` section, _and_
478
479 . the `fencing` option for the resource is set to either
480 `resource-only` or `resource-and-stonith`, _and_
481
482 . the node was primary _and_ the replication link is interrupted long enough
483 for DRBDfootnote:[That means e.g. a TCP timeout, the `ping-timeout`, or
484 the kernel triggers a connection abort e.g. because the network link goes down.]
485 to detect a network failure. _or_
486
487 . the node should promote to primary _and_ is not connected to the peer _and_
488 the peer's disks are not already marked as _Outdated_.
489
490
491 The program or script specified as the `fence-peer` handler, when it is
492 invoked, has the `DRBD_RESOURCE` and `DRBD_PEER` environment variables
493 available. They contain the name of the affected DRBD resource and the
494 peer's hostname, respectively.
495
496 Any peer fencing helper program (or script) must return one of the
497 following exit codes:
498
499 .`fence-peer` handler exit codes
500 [format="csv",separator=";",options="header"]
501 |=======================================
502 Exit code;Implication
503 3;Peer's disk state was already _Inconsistent_.
504 4;Peer's disk state was successfully set to _Outdated_ (or was _Outdated_ to begin with).
505 5;Connection to the peer node failed, peer could not be reached.
506 6;Peer refused to be outdated because the affected resource was in the primary role.
507 7;Peer node was successfully fenced off the cluster. This should never occur unless `fencing` is set to `resource-and-stonith` for the affected resource.
508 |=======================================
509
510 [[s-drbd-client-internals]]
511 === The client mode
512
513 Since version 9.0.13 DRBD supports clients. A client in DRBD speak is
514 just a permanently diskless node. In the configuration, it is
515 expressed by using the keyword `none` for the backing block device
516 (the `disk` keyword). You will notice that in the `drbdsetup status`
517 output you will see the `Diskless` disk status displayed in green
518 color. (Usually, a disk state of `Diskless` is displayed in red).
519
520 Internally all the peers of an intentional diskless node are
521 configured with the `peer-device-option` `--bitmap=no`. That means
522 that they will not allocate a bitmap slot in the meta-data for the
523 intentional diskless peer. On the intentional diskless node the device
524 gets marked with the option `--diskless=yes` while it is created with
525 the `new-minor` sub-command of `drbdsetup`.
526
527 These flags are visible through the `events2` status command:
528
529 * a `device` might have the `client:` field. If it reports `yes` the
530 local device was marked to be permanently diskless.
531
532 * a `peer-device` might have the `peer-client` filed. If it says `yes`
533 then there is no change-tracking bitmap to that peer.
534
535 Relevant commands and implications:
536
537 * You can only run `drbdsetup peer-device-options --bitmap=yes ...` if
538 bitmap slots are available in the meta-data, since a bitmap-slot
539 gets allocated.
540
541 * The command `drbdsetup peer-device-options --bitmap=no ...` is only
542 possible if the peer is diskless, it does _not_ unallocate the
543 bitmap-slot.
544
545 * `drbdsetup forget-peer ...` is used to irrevocable free the
546 bitmap-slot assigned to a certain peer.
547
548 * Connecting two peers with disk where one (or both) expect the peer
549 to be permanently diskless fails.
550
551 //- list divider. Keep it otherwise next included section is interpreted as list element