Debian Codegraph @

UG9/en/configure.adoc

raw
1 [[ch-configure]]
2 === Configuring DRBD
3
4 [[s-prepare-storage]]
5 ==== Preparing your lower-level storage
6
7 After you have installed DRBD, you must set aside a roughly
8 identically sized storage area on both cluster nodes. This will
9 become the _lower-level device_ for your DRBD
10 resource. You may use any type of block device found on your
11 system for this purpose. Typical examples include:
12
13 * A hard drive partition (or a full physical hard drive),
14
15 * a software RAID device,
16
17 * an LVM Logical Volume or any other block device configured by the
18 Linux device-mapper infrastructure,
19
20 * any other block device type found on your system.
21
22 You may also use _resource stacking_, meaning you can use one DRBD
23 device as a lower-level device for another. Some specific
24 considerations apply to stacked resources; their configuration is
25 covered in detail in <<s-three-nodes>>.
26
27 NOTE: While it is possible to use loop devices as lower-level devices
28 for DRBD, doing so is not recommended due to deadlock issues.
29
30 It is _not_ necessary for this storage area to be empty before you
31 create a DRBD resource from it. In fact it is a common use case to
32 create a two-node cluster from a previously non-redundant
33 single-server system using DRBD (some caveats apply -- please refer to
34 <<s-metadata>> if you are planning to do this).
35
36 For the purposes of this guide, we assume a very simple setup:
37
38 * Both hosts have a free (currently unused) partition named
39 `/dev/sda7`.
40
41 * We are using <<s-internal-meta-data,internal meta data>>.
42
43
44 [[s-prepare-network]]
45 ==== Preparing your network configuration
46
47 It is recommended, though not strictly required, that you run your
48 DRBD replication over a dedicated connection. At the time of this
49 writing, the most reasonable choice for this is a direct,
50 back-to-back, Gigabit Ethernet connection. When DRBD is run
51 over switches, use of redundant components and the _bonding_ driver
52 (in _active-backup_ mode) is recommended.
53
54 It is generally not recommended to run DRBD replication via routers,
55 for reasons of fairly obvious performance drawbacks (adversely
56 affecting both throughput and latency).
57
58 In terms of local firewall considerations, it is important to
59 understand that DRBD (by convention) uses TCP ports from 7788 upwards,
60 with every resource listening on a separate port. DRBD uses _two_
61 TCP connections for every resource configured. For proper DRBD
62 functionality, it is required that these connections are allowed by
63 your firewall configuration.
64
65 Security considerations other than firewalling may also apply if a
66 Mandatory Access Control (MAC) scheme such as SELinux or AppArmor is
67 enabled. You may have to adjust your local security policy so it does
68 not keep DRBD from functioning properly.
69
70 You must, of course, also ensure that the TCP ports
71 for DRBD are not already used by another application.
72
73 It is not (yet) possible to configure a DRBD resource to support more than
74 one TCP connection pair. If you want to provide for DRBD connection
75 load-balancing or redundancy, you can easily do so at the Ethernet
76 level (again, using the _bonding_ driver).
77
78 For the purposes of this guide, we assume a
79 very simple setup:
80
81 * Our two DRBD hosts each have a currently unused network interface,
82 `eth1`, with IP addresses `10.1.1.31` and `10.1.1.32` assigned to it,
83 respectively.
84
85 * No other services are using TCP ports 7788 through 7799 on either
86 host.
87
88 * The local firewall configuration allows both inbound and outbound
89 TCP connections between the hosts over these ports.
90
91
92 [[s-configure-resource]]
93 ==== Configuring your resource
94
95 All aspects of DRBD are controlled in its configuration file,
96 `/etc/drbd.conf`. Normally, this configuration file is just a skeleton
97 with the following contents:
98
99 -------------------------------------
100 include "/etc/drbd.d/global_common.conf";
101 include "/etc/drbd.d/*.res";
102 -------------------------------------
103
104 By convention, `/etc/drbd.d/global_common.conf` contains the
105 <<s-drbdconf-global,`global`>> and <<s-drbdconf-common,`common`>>
106 sections of the DRBD configuration, whereas the `.res` files contain
107 one <<s-drbdconf-resource,`resource`>> section each.
108
109 It is also possible to use `drbd.conf` as a flat configuration file
110 without any `include` statements at all. Such a configuration,
111 however, quickly becomes cluttered and hard to manage, which is why
112 the multiple-file approach is the preferred one.
113
114 Regardless of which approach you employ, you should always make sure
115 that `drbd.conf`, and any other files it includes, are _exactly
116 identical_ on all participating cluster nodes.
117
118 The DRBD source tarball contains an example configuration file in the
119 `scripts` subdirectory. Binary installation packages will either
120 install this example configuration directly in `/etc`, or in a
121 package-specific documentation directory such as
122 `/usr/share/doc/packages/drbd`.
123
124 This section describes only those few aspects of the configuration
125 file which are absolutely necessary to understand in order to get DRBD
126 up and running. The configuration file's syntax and contents are
127 documented in great detail in the man page of `drbd.conf`.
128
129
130 [[s-drbdconf-example]]
131 ===== Example configuration
132
133 For the purposes of this guide, we assume a
134 minimal setup in line with the examples given in the
135 previous sections:
136
137 .Simple DRBD configuration (`/etc/drbd.d/global_common.conf`)
138 -------------------------------------
139 global {
140 usage-count yes;
141 }
142 common {
143 net {
144 protocol C;
145 }
146 }
147 -------------------------------------
148
149 .Simple DRBD resource configuration (`/etc/drbd.d/r0.res`)
150 -------------------------------------
151 resource "r0" {
152 device minor 1;
153 disk "/dev/sda7";
154 meta-disk internal;
155
156 on "alice" {
157 node-id 0;
158 }
159 on "bob" {
160 node-id 1;
161 }
162 connection {
163 host "alice" address 10.1.1.31:7789;
164 host "bob" address 10.1.1.32:7789;
165 }
166 }
167 -------------------------------------
168
169 This example configures DRBD in the following fashion:
170
171 * You "opt in" to be included in DRBD's usage statistics (see
172 <<fp-usage-count>>).
173
174 * Resources are configured to use fully synchronous replication
175 (<<s-replication-protocols,Protocol C>>) unless explicitly specified
176 otherwise.
177
178 * Our cluster consists of two nodes, 'alice' and 'bob'.
179
180 * We have a resource arbitrarily named `r0` which uses `/dev/sda7` as
181 the lower-level device, and is configured with
182 <<s-internal-meta-data,internal meta data>>.
183
184 * The resource uses TCP port 7789 for its network connections, and
185 binds to the IP addresses 10.1.1.31 and 10.1.1.32, respectively.
186
187 The configuration above implicitly creates one volume in the
188 resource, numbered zero (`0`). For multiple volumes in one resource,
189 modify the syntax as follows (assuming that the same lower-level storage block
190 devices are used on both nodes):
191
192 .Multi-volume DRBD resource configuration (`/etc/drbd.d/r0.res`)
193 -------------------------------------
194 resource "r0" {
195 volume 0 {
196 device minor 1;
197 disk "/dev/sda7";
198 meta-disk internal;
199 }
200 volume 1 {
201 device minor 2;
202 disk "/dev/sda7";
203 meta-disk internal;
204 }
205 on "alice" {
206 node-id 0;
207 }
208 on "bob" {
209 node-id 1;
210 volume 1 {
211 disk "/dev/sda9";
212 }
213 }
214 connection {
215 host "alice" address 10.1.1.31:7789;
216 host "bob" address 10.1.1.32:7789;
217 }
218 }
219 -------------------------------------
220
221 * Host sections ('on' keyword) inherit _volume_ sections from the resource
222 level. They may contain _volume_ themselves, these values have precedence
223 over inherited values.
224
225 NOTE: Volumes may also be added to existing resources on the fly. For
226 an example see <<s-lvm-add-pv>>.
227
228 For compatibility with older releases of DRBD it supports also drbd-8.4
229 like configuration files.
230
231 .An old(8.4) style configuration file
232 -------------------------------------
233 resource r0 {
234 on alice {
235 device /dev/drbd1;
236 disk /dev/sda7;
237 meta-disk internal;
238 address 10.1.1.31:7789;
239 }
240 on bob {
241 device /dev/drbd1;
242 disk /dev/sda7;
243 meta-disk internal;
244 address 10.1.1.32:7789;
245 }
246 -------------------------------------
247
248 * Strings that do not contain keywords, might be gives withouth
249 double quotes `"`.
250
251 * The old version to specify the device, was to give a string
252 containing the name of the resulting device file.
253
254 * Two node configurations get node numbers assigned by drbdadm.
255
256 * A pure two node configuration gets an implicit connection.
257
258 [[s-drbdconf-global]]
259 ===== The `global` section
260
261 This section is allowed only once in the configuration. It is normally
262 in the `/etc/drbd.d/global_common.conf` file. In a single-file
263 configuration, it should go to the very top of the configuration
264 file. Of the few options available in this section, only one is of
265 relevance to most users:
266
267 [[fp-usage-count]]
268 .`usage-count`
269 The DRBD project keeps statistics about the usage of various DRBD
270 versions. This is done by contacting an HTTP server every time a new
271 DRBD version is installed on a system. This can be disabled by setting
272 `usage-count no;`. The default is `usage-count ask;` which will
273 prompt you every time you upgrade DRBD.
274
275 DRBD's usage statistics are, of course, publicly available: see
276 http://usage.drbd.org.
277
278
279 [[s-drbdconf-common]]
280 ===== The `common` section
281
282 This section provides a shorthand method to define configuration
283 settings inherited by every resource. It is normally found in
284 `/etc/drbd.d/global_common.conf`. You may define any option you can
285 also define on a per-resource basis.
286
287 Including a `common` section is not strictly required, but strongly
288 recommended if you are using more than one resource. Otherwise, the
289 configuration quickly becomes convoluted by repeatedly-used options.
290
291 In the example above, we included `net { protocol C; }` in the
292 `common` section, so every resource configured (including `r0`)
293 inherits this option unless it has another `protocol` option
294 configured explicitly. For other synchronization protocols available,
295 see <<s-replication-protocols>>.
296
297 [[s-drbdconf-resource]]
298 ===== The `resource` sections
299
300 A per-resource configuration file is usually named
301 `/etc/drbd.d/__resource__.res`. Any DRBD resource you define must be
302 named by specifying a resource name in the configuration. The convention
303 is to use only letters, digits, and the underscore; while it is technically
304 possible to use other characters as well, you won't like the result if you ever
305 happen stumble to need the more specific `__resource__:___peer__/__volume__` syntax.
306
307 Every resource configuration must also have at least two `on _host_` sub-sections,
308 one for every cluster node. All other configuration settings are
309 either inherited from the `common` section (if it exists), or derived
310 from DRBD's default settings.
311
312 In addition, options with equal values on all hosts
313 can be specified directly in the `resource` section. Thus, we can
314 further condense our example configuration as follows:
315
316 -------------------------------------
317 resource "r0" {
318 device minor 1;
319 disk "/dev/sda7";
320 meta-disk internal;
321 on "alice" {
322 address 10.1.1.31:7789;
323 }
324 on "bob" {
325 address 10.1.1.32:7789;
326 }
327 }
328 -------------------------------------
329
330
331 [[s-drbdconf-conns]]
332 ==== Defining network connections
333
334 Currently the communication links in DRBD 9 must build a full mesh, i.e. in
335 every resource every node must have a direct connection to every other node
336 (excluding itself, of course).
337
338 For the simple case of two hosts `drbdadm` will insert the (single) network
339 connection by itself, for ease of use and backwards compatibility.
340
341 The net effect of this is a quadratic number of network connections over
342 hosts. For the "traditional" two nodes one connection is needed; for three hosts there are three node pairs; for four, six pairs;
343 5 hosts: 10 connections, and so on. For (the current)
344 maximum of 16 nodes there'll be 120 host pairs to connect.
345
346 [[eq-connection-mesh]]
347 .Number of connections for _N_ hosts
348 image::images/connection-mesh.svg[]
349
350
351 An example configuration file for three hosts would be this:
352
353 -------------------------------------
354 resource r0 {
355 device minor 1;
356 disk "/dev/sda7";
357 meta-disk internal;
358 on alice {
359 address 10.1.1.31:7000;
360 node-id 0;
361 }
362 on bob {
363 address 10.1.1.32:7000;
364 node-id 1;
365 }
366 on charlie {
367 address 10.1.1.33:7000;
368 node-id 2;
369 }
370 connection-mesh {
371 hosts alice bob charlie;
372 }
373 }
374 -------------------------------------
375
376
377 If you have got enough network cards in your servers, you can create direct
378 cross-over links between server pairs.
379 A single four-port ethernet card allows to have a single management interface,
380 and to connect 3 other servers, to get a full mesh for 4 cluster nodes.
381
382 In this case you can specify a different IP address to use the direct link:
383
384 --------------------------------
385 resource r0 {
386 ...
387 connection {
388 host alice address 10.1.2.1:7010;
389 host bob address 10.1.2.2:7001;
390 }
391 connection {
392 host alice address 10.1.3.1:7020;
393 host charlie address 10.1.3.2:7002;
394 }
395 connection {
396 host bob address 10.1.4.1:7021;
397 host charlie address 10.1.4.2:7012;
398 }
399 }
400 --------------------------------
401
402 For easier maintenance and debugging it's recommend to have different ports for
403 each endpoint - looking at a `tcpdump` trace the packets can be associated easily.
404
405 The examples below will still be using two servers only; please see
406 <<s-4node-example>> for a four-node example.
407
408
409 [[s-configuring-multiple-paths]]
410 ==== Configuring transport implementations
411 DRBD allows configuring multiple paths per connection, by introducing
412 multiple path sections in a connection. Please see the following example:
413
414 ---------------------
415 resource <resource> {
416 ...
417 connection {
418 path {
419 host alpha address 192.168.41.1:7900;
420 host bravo address 192.168.41.2:7900;
421 }
422 path {
423 host alpha address 192.168.42.1:7900;
424 host bravo address 192.168.42.2:7900;
425 }
426 }
427 ...
428 }
429 ---------------------
430
431 Obviously the two endpoint hostname need to be equal in all paths of
432 a connection. Paths may be on different IPs (potential different NICs)
433 or may only be on different ports.
434
435 The TCP transport uses one path at a time. In case the backing TCP
436 connections get dropped, or show timeouts, the TCP transport implementation
437 tries to establish a connection over the next path. It goes over all paths
438 in a round-robin fashion until a connection gets established.
439
440 The RDMA transport uses all paths of a connection concurrently and it
441 balances the network traffic between the paths evenly.
442
443 [[s-configuring-transports]]
444 ==== Configuring transport implementations
445 DRBD supports multiple network transports. A transport implementation can be
446 configured for each connection of a resource.
447
448 [[s-tcp_ip]]
449 ===== TCP/IP
450 ---------------------
451 resource <resource> {
452 net {
453 transport "tcp";
454 }
455 ...
456 }
457 ---------------------
458 `tcp` is the default transport. I.e. each connection that lacks a transport
459 option uses the `tcp` transport.
460
461 The `tcp` transport can be configured with the net options: `sndbuf-size`, `rcvbuf-size`,
462 `connect-int`, `sock-check-timeo`, `ping-timeo`, `timeout`.
463
464 [[s-rdma]]
465 ===== RDMA
466 ---------------------
467 resource <resource> {
468 net {
469 transport "rdma";
470 }
471 ...
472 }
473 ---------------------
474 The `rdma` transport can be configured with the net options: `sndbuf-size`, `rcvbuf-size`,
475 `max_buffers`, `connect-int`, `sock-check-timeo`, `ping-timeo`, `timeout`.
476
477 The `rdma` transport is a zero-copy-receive transport. One implication of that is that
478 the `max_buffers` configuration options must be set to a value big enough to hold all
479 `rcvbuf-size`.
480
481 NOTE: `rcvbuf-size` is configured in bytes, while `max_buffers` is configured in pages. For
482 optimal performance `max_buffers` should be big enough to hold all of `rcvbuf-size` and
483 the amount of data that might be in flight to the backend device at any point in time.
484
485 TIP: In case you are using InfiniBand HCAs with the `rdma` transport, you need to
486 configure IPoIB as well. The IP address is not used for data transfer, but it is used
487 to find the right adapters and ports while establishing the connection.
488
489 CAUTION: The configuration options `sndbuf-size`, `rcvbuf-size` are only considered at
490 the time a connection is established. I.e. you can change them while the connection is
491 established. They will take effect in the moment the connection is re-established.
492
493 [[s-performance_considerations_for_rdma]]
494 ===== Performance considerations for RDMA
495
496 By looking at the pseudo file _/sys/kernel/debug/drbd/<resource>/connections/<peer>/transport_,
497 the counts of available receive descriptors (rx_desc) and transmit descriptors (tx_desc)
498 can be monitored. In case one of the descriptor kinds becomes depleted you should increase
499 `sndbuf-size` or `rcvbuf-size`.
500
501 [[s-first-time-up]]
502 ==== Enabling your resource for the first time
503
504 After you have completed initial resource configuration as outlined in
505 the previous sections, you can bring up your resource.
506
507 Each of the following steps must be completed on both nodes.
508
509 Please note that with our example config snippets (`resource r0 { ... }`), `<resource>` would be `r0`.
510
511 .Create device metadata
512 This step must be completed only on initial device
513 creation. It initializes DRBD's metadata:
514
515 -------------------------------------
516 # drbdadm create-md <resource>
517 v09 Magic number not found
518 Writing meta data...
519 initialising activity log
520 NOT initializing bitmap
521 New drbd meta data block successfully created.
522 -------------------------------------
523
524 Please note that the number of bitmap slots that are allocated in the meta-data
525 depends on the number of hosts for this resource; per default the hosts in the
526 resource configuration are counted.
527 If all hosts are specified _before_ creating the meta-data, this will "just work";
528 adding bitmap slots for further nodes is possible later, but incurs some manual work.
529
530
531 .Enable the resource
532 This step associates the resource with its backing device (or devices,
533 in case of a multi-volume resource), sets replication parameters, and
534 connects the resource to its peer:
535 -------------------------------------
536 # drbdadm up <resource>
537 -------------------------------------
538
539 .Observe the status via `drbdadm status`
540 ``drbdsetup``'s status output
541 should now contain information similar to the following:
542
543 -------------------------------------
544 # drbdadm status r0
545 r0 role:Secondary
546 disk:Inconsistent
547 bob role:Secondary
548 disk:Inconsistent
549 -------------------------------------
550
551 NOTE: The _Inconsistent/Inconsistent_ disk state is expected at this
552 point.
553
554 By now, DRBD has successfully allocated both disk and network
555 resources and is ready for operation. What it does not know yet is
556 which of your nodes should be used as the source of the initial device
557 synchronization.
558
559 [[s-initial-full-sync]]
560 ==== The initial device synchronization
561
562 There are two more steps required for DRBD to become fully
563 operational:
564
565 .Select an initial sync source
566 If you are dealing with newly-initialized, empty disks, this choice is
567 entirely arbitrary. If one of your nodes already has valuable data
568 that you need to preserve, however, _it is of crucial importance_ that
569 you select that node as your synchronization source. If you do
570 initial device synchronization in the wrong direction, you will lose
571 that data. Exercise caution.
572
573
574 .Start the initial full synchronization
575 This step must be performed on only one node, only on initial resource
576 configuration, and only on the node you selected as the
577 synchronization source. To perform this step, issue this command:
578
579 -------------------------------------
580 # drbdadm primary --force <resource>
581 -------------------------------------
582
583 After issuing this command, the initial full synchronization will
584 commence. You will be able to monitor its progress via
585 `drbdadm status`. It may take some time depending on the size of the
586 device.
587
588 By now, your DRBD device is fully operational, even before the initial
589 synchronization has completed (albeit with slightly reduced
590 performance). If you started with empty disks you may now already
591 create a filesystem on the device, use it as
592 a raw block device, mount it, and perform any other operation you
593 would with an accessible block device.
594
595 You will now probably want to continue with <<p-work>>, which
596 describes common administrative tasks to perform on your resource.
597
598 [[s-skip-initial-resync]]
599 ==== Skipping initial resynchronization
600
601 If (and only if) you are starting DRBD resources from scratch (with
602 no valueable data on them) you can use following command sequence
603 to skip initial resync (don't do that with data you want to keep on
604 the devices):
605
606 On all nodes:
607
608 -------------------------------------
609 # drbdadm create-md <res>
610 # drbdadm up <res>
611 -------------------------------------
612
613 The command `drbdadm status` should now show all disks as _Inconsistent_.
614
615 Then, on one node execute the following command:
616 --
617 -------------------------------------
618 # drbdadm new-current-uuid --clear-bitmap <resource>/<volume>
619 -------------------------------------
620 or
621 -------------------------------------
622 # drbdsetup new-current-uuid --clear-bitmap <minor>
623 -------------------------------------
624 --
625
626 Running `drbdadm status` now shows the disks as _UpToDate_ (even tough the
627 backing devices might be out of sync). You can now create a file
628 system on the disk and start using it.
629
630 IMPORTANT: Don't do the above with data you want to keep or it gets
631 corrupted.
632
633 [[s-using-truck-based-replication]]
634 ==== Using truck based replication
635
636 In order to preseed a remote node with data which is then to be kept
637 synchronized, and to skip the initial full device synchronization, follow
638 these steps.
639
640 This assumes that your local node has a configured, but disconnected
641 DRBD resource in the _Primary_ role. That is to say, device
642 configuration is completed, identical `drbd.conf` copies exist on both
643 nodes, and you have issued the commands for
644 <<s-initial-full-sync,initial resource promotion>> on your local node
645 -- but the remote node is not connected yet.
646
647
648 * On the local node, issue the following command:
649 +
650 --
651 -------------------------------------
652 # drbdadm new-current-uuid --clear-bitmap <resource>/<volume>
653 -------------------------------------
654 or
655 -------------------------------------
656 # drbdsetup new-current-uuid --clear-bitmap <minor>
657 -------------------------------------
658 --
659
660 * Create a consistent, verbatim copy of the resource's data _and its
661 metadata_. You may do so, for example, by removing a hot-swappable
662 drive from a RAID-1 mirror. You would, of course, replace it with a
663 fresh drive, and rebuild the RAID set, to ensure continued
664 redundancy. But the removed drive is a verbatim copy that can now be
665 shipped off site. If your local block device supports snapshot
666 copies (such as when using DRBD on top of LVM), you may also create
667 a bitwise copy of that snapshot using `dd`.
668
669
670 * On the local node, issue:
671 +
672 --
673 -------------------------------------
674 # drbdadm new-current-uuid <resource>
675 -------------------------------------
676 or the matching `drbdsetup` command.
677
678 Note the absence of the `--clear-bitmap` option in this second
679 invocation.
680 --
681
682 * Physically transport the copies to the remote peer location.
683
684 * Add the copies to the remote node. This may again be a matter of
685 plugging a physical disk, or grafting a bitwise copy of your shipped
686 data onto existing storage on the remote node. Be sure to restore
687 or copy not only your replicated data, but also the associated DRBD
688 metadata. If you fail to do so, the disk shipping process is moot.
689
690 * On the new node we need to fix the node ID in the meta data, and exchange
691 the peer-node info for the two nodes. Please see the following lines as
692 example for changing node id from 2 to 1 on a resource `r0` volume `0`.
693 +
694 --
695
696 This must be done while the volume is not in use.
697
698 You need to edit the first 4 lines to match your needs. V is the
699 resource name with the volume number. NODE_FROM is the node ID of
700 the node the data originates from. NOTE_TO is the node ID of the
701 node where data will be replicated to. META_DATA_LOCATION is the
702 location of the meta data which might be internal or flex-external.
703
704 -----------
705 V=r0/0
706 NODE_FROM=2
707 NODE_TO=1
708 META_DATA_LOCATION=internal
709
710 drbdadm -- --force dump-md $V > /tmp/md_orig.txt
711 sed -e "s/node-id $NODE_FROM/node-id $NODE_TO/" \
712 -e "s/^peer.$NODE_FROM. /peer-NEW /" \
713 -e "s/^peer.$NODE_TO. /peer[$NODE_FROM] /" \
714 -e "s/^peer-NEW /peer[$NODE_TO] /" \
715 < /tmp/md_orig.txt > /tmp/md.txt
716
717 drbdmeta --force $(drbdadm sh-minor $V) v09 $(drbdadm sh-md-dev $V) $META_DATA_LOCATION restore-md /tmp/md.txt
718 -----------
719
720 .NOTE
721 `drbdmeta` before 8.9.7 cannot cope with out-of-order `peer` sections; you'll
722 need to exchange the blocks via an editor.
723
724 --
725
726 * Bring up the resource on the remote node:
727 +
728 -------------------------------------
729 # drbdadm up <resource>
730 -------------------------------------
731
732 After the two peers connect, they will not initiate a full device
733 synchronization. Instead, the automatic synchronization that now
734 commences only covers those blocks that changed since the invocation
735 of `drbdadm{nbsp}--clear-bitmap{nbsp}new-current-uuid`.
736
737 Even if there were _no_ changes whatsoever since then, there may still
738 be a brief synchronization period due to areas covered by the
739 <<s-activity-log,Activity Log>> being rolled back on the new
740 Secondary. This may be mitigated by the use of
741 <<p-checksum-sync,checksum-based synchronization>>.
742
743 You may use this same procedure regardless of whether the resource is
744 a regular DRBD resource, or a stacked resource. For stacked resources,
745 simply add the `-S` or `--stacked` option to `drbdadm`.
746
747
748 [[s-4node-example]]
749 ==== Example configuration for four nodes
750
751 Here is an example for a four-node cluster.
752
753 [[s-connection-mesh]]
754 ----------------------
755 resource r0 {
756 device minor 0;
757 disk /dev/vg/r0;
758 meta-disk internal;
759
760 on store1 {
761 address 10.1.10.1:7100;
762 node-id 1;
763 }
764 on store2 {
765 address 10.1.10.2:7100;
766 node-id 2;
767 }
768 on store3 {
769 address 10.1.10.3:7100;
770 node-id 3;
771 }
772 on store4 {
773 address 10.1.10.4:7100;
774 node-id 4;
775 }
776
777 connection-mesh {
778 hosts store1 store2 store3 store4;
779 }
780 }
781 ----------------------
782
783 In case you want to see the `connection-mesh` configuration expanded, try `drbdadm dump _<resource>_ -v`.
784
785
786 [[s-connection-mesh-distinct-interfaces]]
787 As another example, if the four nodes have enough interfaces to provide
788 a complete mesh via direct linksfootnote:[i.e. three crossover and at least one
789 outgoing/management interface], you can specify the IP addresses of the
790 interfaces:
791
792 ---------------------------
793 resource r0 {
794 ...
795
796 # store1 has crossover links like 10.99.1x.y
797 connection {
798 host store1 address 10.99.12.1 port 7012;
799 host store2 address 10.99.12.2 port 7021;
800 }
801 connection {
802 host store1 address 10.99.13.1 port 7013;
803 host store3 address 10.99.13.3 port 7031;
804 }
805 connection {
806 host store1 address 10.99.14.1 port 7014;
807 host store4 address 10.99.14.4 port 7041;
808 }
809
810 # store2 has crossover links like 10.99.2x.y
811 connection {
812 host store2 address 10.99.23.2 port 7023;
813 host store3 address 10.99.23.3 port 7032;
814 }
815 connection {
816 host store2 address 10.99.24.2 port 7024;
817 host store4 address 10.99.24.4 port 7042;
818 }
819
820 # store3 has crossover links like 10.99.3x.y
821 connection {
822 host store3 address 10.99.34.3 port 7034;
823 host store4 address 10.99.34.4 port 7043;
824 }
825 }
826 ---------------------------
827
828 Please note the numbering scheme used for the IP addresses and ports. Another
829 resource could use the same IP addresses, but ports `71__xy__`, the next one
830 `72__xy__`, and so on.