| 1 |
[[ch-configure]] |
| 2 |
=== Configuring DRBD |
| 3 |
|
| 4 |
[[s-prepare-storage]] |
| 5 |
==== Preparing your lower-level storage |
| 6 |
|
| 7 |
After you have installed DRBD, you must set aside a roughly |
| 8 |
identically sized storage area on both cluster nodes. This will |
| 9 |
become the _lower-level device_ for your DRBD |
| 10 |
resource. You may use any type of block device found on your |
| 11 |
system for this purpose. Typical examples include: |
| 12 |
|
| 13 |
* A hard drive partition (or a full physical hard drive), |
| 14 |
|
| 15 |
* a software RAID device, |
| 16 |
|
| 17 |
* an LVM Logical Volume or any other block device configured by the |
| 18 |
Linux device-mapper infrastructure, |
| 19 |
|
| 20 |
* any other block device type found on your system. |
| 21 |
|
| 22 |
You may also use _resource stacking_, meaning you can use one DRBD |
| 23 |
device as a lower-level device for another. Some specific |
| 24 |
considerations apply to stacked resources; their configuration is |
| 25 |
covered in detail in <<s-three-nodes>>. |
| 26 |
|
| 27 |
NOTE: While it is possible to use loop devices as lower-level devices |
| 28 |
for DRBD, doing so is not recommended due to deadlock issues. |
| 29 |
|
| 30 |
It is _not_ necessary for this storage area to be empty before you |
| 31 |
create a DRBD resource from it. In fact it is a common use case to |
| 32 |
create a two-node cluster from a previously non-redundant |
| 33 |
single-server system using DRBD (some caveats apply -- please refer to |
| 34 |
<<s-metadata>> if you are planning to do this). |
| 35 |
|
| 36 |
For the purposes of this guide, we assume a very simple setup: |
| 37 |
|
| 38 |
* Both hosts have a free (currently unused) partition named |
| 39 |
`/dev/sda7`. |
| 40 |
|
| 41 |
* We are using <<s-internal-meta-data,internal meta data>>. |
| 42 |
|
| 43 |
|
| 44 |
[[s-prepare-network]] |
| 45 |
==== Preparing your network configuration |
| 46 |
|
| 47 |
It is recommended, though not strictly required, that you run your |
| 48 |
DRBD replication over a dedicated connection. At the time of this |
| 49 |
writing, the most reasonable choice for this is a direct, |
| 50 |
back-to-back, Gigabit Ethernet connection. When DRBD is run |
| 51 |
over switches, use of redundant components and the _bonding_ driver |
| 52 |
(in _active-backup_ mode) is recommended. |
| 53 |
|
| 54 |
It is generally not recommended to run DRBD replication via routers, |
| 55 |
for reasons of fairly obvious performance drawbacks (adversely |
| 56 |
affecting both throughput and latency). |
| 57 |
|
| 58 |
In terms of local firewall considerations, it is important to |
| 59 |
understand that DRBD (by convention) uses TCP ports from 7788 upwards, |
| 60 |
with every resource listening on a separate port. DRBD uses _two_ |
| 61 |
TCP connections for every resource configured. For proper DRBD |
| 62 |
functionality, it is required that these connections are allowed by |
| 63 |
your firewall configuration. |
| 64 |
|
| 65 |
Security considerations other than firewalling may also apply if a |
| 66 |
Mandatory Access Control (MAC) scheme such as SELinux or AppArmor is |
| 67 |
enabled. You may have to adjust your local security policy so it does |
| 68 |
not keep DRBD from functioning properly. |
| 69 |
|
| 70 |
You must, of course, also ensure that the TCP ports |
| 71 |
for DRBD are not already used by another application. |
| 72 |
|
| 73 |
It is not (yet) possible to configure a DRBD resource to support more than |
| 74 |
one TCP connection pair. If you want to provide for DRBD connection |
| 75 |
load-balancing or redundancy, you can easily do so at the Ethernet |
| 76 |
level (again, using the _bonding_ driver). |
| 77 |
|
| 78 |
For the purposes of this guide, we assume a |
| 79 |
very simple setup: |
| 80 |
|
| 81 |
* Our two DRBD hosts each have a currently unused network interface, |
| 82 |
`eth1`, with IP addresses `10.1.1.31` and `10.1.1.32` assigned to it, |
| 83 |
respectively. |
| 84 |
|
| 85 |
* No other services are using TCP ports 7788 through 7799 on either |
| 86 |
host. |
| 87 |
|
| 88 |
* The local firewall configuration allows both inbound and outbound |
| 89 |
TCP connections between the hosts over these ports. |
| 90 |
|
| 91 |
|
| 92 |
[[s-configure-resource]] |
| 93 |
==== Configuring your resource |
| 94 |
|
| 95 |
All aspects of DRBD are controlled in its configuration file, |
| 96 |
`/etc/drbd.conf`. Normally, this configuration file is just a skeleton |
| 97 |
with the following contents: |
| 98 |
|
| 99 |
------------------------------------- |
| 100 |
include "/etc/drbd.d/global_common.conf"; |
| 101 |
include "/etc/drbd.d/*.res"; |
| 102 |
------------------------------------- |
| 103 |
|
| 104 |
By convention, `/etc/drbd.d/global_common.conf` contains the |
| 105 |
<<s-drbdconf-global,`global`>> and <<s-drbdconf-common,`common`>> |
| 106 |
sections of the DRBD configuration, whereas the `.res` files contain |
| 107 |
one <<s-drbdconf-resource,`resource`>> section each. |
| 108 |
|
| 109 |
It is also possible to use `drbd.conf` as a flat configuration file |
| 110 |
without any `include` statements at all. Such a configuration, |
| 111 |
however, quickly becomes cluttered and hard to manage, which is why |
| 112 |
the multiple-file approach is the preferred one. |
| 113 |
|
| 114 |
Regardless of which approach you employ, you should always make sure |
| 115 |
that `drbd.conf`, and any other files it includes, are _exactly |
| 116 |
identical_ on all participating cluster nodes. |
| 117 |
|
| 118 |
The DRBD source tarball contains an example configuration file in the |
| 119 |
`scripts` subdirectory. Binary installation packages will either |
| 120 |
install this example configuration directly in `/etc`, or in a |
| 121 |
package-specific documentation directory such as |
| 122 |
`/usr/share/doc/packages/drbd`. |
| 123 |
|
| 124 |
This section describes only those few aspects of the configuration |
| 125 |
file which are absolutely necessary to understand in order to get DRBD |
| 126 |
up and running. The configuration file's syntax and contents are |
| 127 |
documented in great detail in the man page of `drbd.conf`. |
| 128 |
|
| 129 |
|
| 130 |
[[s-drbdconf-example]] |
| 131 |
===== Example configuration |
| 132 |
|
| 133 |
For the purposes of this guide, we assume a |
| 134 |
minimal setup in line with the examples given in the |
| 135 |
previous sections: |
| 136 |
|
| 137 |
.Simple DRBD configuration (`/etc/drbd.d/global_common.conf`) |
| 138 |
------------------------------------- |
| 139 |
global { |
| 140 |
usage-count yes; |
| 141 |
} |
| 142 |
common { |
| 143 |
net { |
| 144 |
protocol C; |
| 145 |
} |
| 146 |
} |
| 147 |
------------------------------------- |
| 148 |
|
| 149 |
.Simple DRBD resource configuration (`/etc/drbd.d/r0.res`) |
| 150 |
------------------------------------- |
| 151 |
resource "r0" { |
| 152 |
device minor 1; |
| 153 |
disk "/dev/sda7"; |
| 154 |
meta-disk internal; |
| 155 |
|
| 156 |
on "alice" { |
| 157 |
node-id 0; |
| 158 |
} |
| 159 |
on "bob" { |
| 160 |
node-id 1; |
| 161 |
} |
| 162 |
connection { |
| 163 |
host "alice" address 10.1.1.31:7789; |
| 164 |
host "bob" address 10.1.1.32:7789; |
| 165 |
} |
| 166 |
} |
| 167 |
------------------------------------- |
| 168 |
|
| 169 |
This example configures DRBD in the following fashion: |
| 170 |
|
| 171 |
* You "opt in" to be included in DRBD's usage statistics (see |
| 172 |
<<fp-usage-count>>). |
| 173 |
|
| 174 |
* Resources are configured to use fully synchronous replication |
| 175 |
(<<s-replication-protocols,Protocol C>>) unless explicitly specified |
| 176 |
otherwise. |
| 177 |
|
| 178 |
* Our cluster consists of two nodes, 'alice' and 'bob'. |
| 179 |
|
| 180 |
* We have a resource arbitrarily named `r0` which uses `/dev/sda7` as |
| 181 |
the lower-level device, and is configured with |
| 182 |
<<s-internal-meta-data,internal meta data>>. |
| 183 |
|
| 184 |
* The resource uses TCP port 7789 for its network connections, and |
| 185 |
binds to the IP addresses 10.1.1.31 and 10.1.1.32, respectively. |
| 186 |
|
| 187 |
The configuration above implicitly creates one volume in the |
| 188 |
resource, numbered zero (`0`). For multiple volumes in one resource, |
| 189 |
modify the syntax as follows (assuming that the same lower-level storage block |
| 190 |
devices are used on both nodes): |
| 191 |
|
| 192 |
.Multi-volume DRBD resource configuration (`/etc/drbd.d/r0.res`) |
| 193 |
------------------------------------- |
| 194 |
resource "r0" { |
| 195 |
volume 0 { |
| 196 |
device minor 1; |
| 197 |
disk "/dev/sda7"; |
| 198 |
meta-disk internal; |
| 199 |
} |
| 200 |
volume 1 { |
| 201 |
device minor 2; |
| 202 |
disk "/dev/sda7"; |
| 203 |
meta-disk internal; |
| 204 |
} |
| 205 |
on "alice" { |
| 206 |
node-id 0; |
| 207 |
} |
| 208 |
on "bob" { |
| 209 |
node-id 1; |
| 210 |
volume 1 { |
| 211 |
disk "/dev/sda9"; |
| 212 |
} |
| 213 |
} |
| 214 |
connection { |
| 215 |
host "alice" address 10.1.1.31:7789; |
| 216 |
host "bob" address 10.1.1.32:7789; |
| 217 |
} |
| 218 |
} |
| 219 |
------------------------------------- |
| 220 |
|
| 221 |
* Host sections ('on' keyword) inherit _volume_ sections from the resource |
| 222 |
level. They may contain _volume_ themselves, these values have precedence |
| 223 |
over inherited values. |
| 224 |
|
| 225 |
NOTE: Volumes may also be added to existing resources on the fly. For |
| 226 |
an example see <<s-lvm-add-pv>>. |
| 227 |
|
| 228 |
For compatibility with older releases of DRBD it supports also drbd-8.4 |
| 229 |
like configuration files. |
| 230 |
|
| 231 |
.An old(8.4) style configuration file |
| 232 |
------------------------------------- |
| 233 |
resource r0 { |
| 234 |
on alice { |
| 235 |
device /dev/drbd1; |
| 236 |
disk /dev/sda7; |
| 237 |
meta-disk internal; |
| 238 |
address 10.1.1.31:7789; |
| 239 |
} |
| 240 |
on bob { |
| 241 |
device /dev/drbd1; |
| 242 |
disk /dev/sda7; |
| 243 |
meta-disk internal; |
| 244 |
address 10.1.1.32:7789; |
| 245 |
} |
| 246 |
------------------------------------- |
| 247 |
|
| 248 |
* Strings that do not contain keywords, might be gives withouth |
| 249 |
double quotes `"`. |
| 250 |
|
| 251 |
* The old version to specify the device, was to give a string |
| 252 |
containing the name of the resulting device file. |
| 253 |
|
| 254 |
* Two node configurations get node numbers assigned by drbdadm. |
| 255 |
|
| 256 |
* A pure two node configuration gets an implicit connection. |
| 257 |
|
| 258 |
[[s-drbdconf-global]] |
| 259 |
===== The `global` section |
| 260 |
|
| 261 |
This section is allowed only once in the configuration. It is normally |
| 262 |
in the `/etc/drbd.d/global_common.conf` file. In a single-file |
| 263 |
configuration, it should go to the very top of the configuration |
| 264 |
file. Of the few options available in this section, only one is of |
| 265 |
relevance to most users: |
| 266 |
|
| 267 |
[[fp-usage-count]] |
| 268 |
.`usage-count` |
| 269 |
The DRBD project keeps statistics about the usage of various DRBD |
| 270 |
versions. This is done by contacting an HTTP server every time a new |
| 271 |
DRBD version is installed on a system. This can be disabled by setting |
| 272 |
`usage-count no;`. The default is `usage-count ask;` which will |
| 273 |
prompt you every time you upgrade DRBD. |
| 274 |
|
| 275 |
DRBD's usage statistics are, of course, publicly available: see |
| 276 |
http://usage.drbd.org. |
| 277 |
|
| 278 |
|
| 279 |
[[s-drbdconf-common]] |
| 280 |
===== The `common` section |
| 281 |
|
| 282 |
This section provides a shorthand method to define configuration |
| 283 |
settings inherited by every resource. It is normally found in |
| 284 |
`/etc/drbd.d/global_common.conf`. You may define any option you can |
| 285 |
also define on a per-resource basis. |
| 286 |
|
| 287 |
Including a `common` section is not strictly required, but strongly |
| 288 |
recommended if you are using more than one resource. Otherwise, the |
| 289 |
configuration quickly becomes convoluted by repeatedly-used options. |
| 290 |
|
| 291 |
In the example above, we included `net { protocol C; }` in the |
| 292 |
`common` section, so every resource configured (including `r0`) |
| 293 |
inherits this option unless it has another `protocol` option |
| 294 |
configured explicitly. For other synchronization protocols available, |
| 295 |
see <<s-replication-protocols>>. |
| 296 |
|
| 297 |
[[s-drbdconf-resource]] |
| 298 |
===== The `resource` sections |
| 299 |
|
| 300 |
A per-resource configuration file is usually named |
| 301 |
`/etc/drbd.d/__resource__.res`. Any DRBD resource you define must be |
| 302 |
named by specifying a resource name in the configuration. The convention |
| 303 |
is to use only letters, digits, and the underscore; while it is technically |
| 304 |
possible to use other characters as well, you won't like the result if you ever |
| 305 |
happen stumble to need the more specific `__resource__:___peer__/__volume__` syntax. |
| 306 |
|
| 307 |
Every resource configuration must also have at least two `on _host_` sub-sections, |
| 308 |
one for every cluster node. All other configuration settings are |
| 309 |
either inherited from the `common` section (if it exists), or derived |
| 310 |
from DRBD's default settings. |
| 311 |
|
| 312 |
In addition, options with equal values on all hosts |
| 313 |
can be specified directly in the `resource` section. Thus, we can |
| 314 |
further condense our example configuration as follows: |
| 315 |
|
| 316 |
------------------------------------- |
| 317 |
resource "r0" { |
| 318 |
device minor 1; |
| 319 |
disk "/dev/sda7"; |
| 320 |
meta-disk internal; |
| 321 |
on "alice" { |
| 322 |
address 10.1.1.31:7789; |
| 323 |
} |
| 324 |
on "bob" { |
| 325 |
address 10.1.1.32:7789; |
| 326 |
} |
| 327 |
} |
| 328 |
------------------------------------- |
| 329 |
|
| 330 |
|
| 331 |
[[s-drbdconf-conns]] |
| 332 |
==== Defining network connections |
| 333 |
|
| 334 |
Currently the communication links in DRBD 9 must build a full mesh, i.e. in |
| 335 |
every resource every node must have a direct connection to every other node |
| 336 |
(excluding itself, of course). |
| 337 |
|
| 338 |
For the simple case of two hosts `drbdadm` will insert the (single) network |
| 339 |
connection by itself, for ease of use and backwards compatibility. |
| 340 |
|
| 341 |
The net effect of this is a quadratic number of network connections over |
| 342 |
hosts. For the "traditional" two nodes one connection is needed; for three hosts there are three node pairs; for four, six pairs; |
| 343 |
5 hosts: 10 connections, and so on. For (the current) |
| 344 |
maximum of 16 nodes there'll be 120 host pairs to connect. |
| 345 |
|
| 346 |
[[eq-connection-mesh]] |
| 347 |
.Number of connections for _N_ hosts |
| 348 |
image::images/connection-mesh.svg[] |
| 349 |
|
| 350 |
|
| 351 |
An example configuration file for three hosts would be this: |
| 352 |
|
| 353 |
------------------------------------- |
| 354 |
resource r0 { |
| 355 |
device minor 1; |
| 356 |
disk "/dev/sda7"; |
| 357 |
meta-disk internal; |
| 358 |
on alice { |
| 359 |
address 10.1.1.31:7000; |
| 360 |
node-id 0; |
| 361 |
} |
| 362 |
on bob { |
| 363 |
address 10.1.1.32:7000; |
| 364 |
node-id 1; |
| 365 |
} |
| 366 |
on charlie { |
| 367 |
address 10.1.1.33:7000; |
| 368 |
node-id 2; |
| 369 |
} |
| 370 |
connection-mesh { |
| 371 |
hosts alice bob charlie; |
| 372 |
} |
| 373 |
} |
| 374 |
------------------------------------- |
| 375 |
|
| 376 |
|
| 377 |
If you have got enough network cards in your servers, you can create direct |
| 378 |
cross-over links between server pairs. |
| 379 |
A single four-port ethernet card allows to have a single management interface, |
| 380 |
and to connect 3 other servers, to get a full mesh for 4 cluster nodes. |
| 381 |
|
| 382 |
In this case you can specify a different IP address to use the direct link: |
| 383 |
|
| 384 |
-------------------------------- |
| 385 |
resource r0 { |
| 386 |
... |
| 387 |
connection { |
| 388 |
host alice address 10.1.2.1:7010; |
| 389 |
host bob address 10.1.2.2:7001; |
| 390 |
} |
| 391 |
connection { |
| 392 |
host alice address 10.1.3.1:7020; |
| 393 |
host charlie address 10.1.3.2:7002; |
| 394 |
} |
| 395 |
connection { |
| 396 |
host bob address 10.1.4.1:7021; |
| 397 |
host charlie address 10.1.4.2:7012; |
| 398 |
} |
| 399 |
} |
| 400 |
-------------------------------- |
| 401 |
|
| 402 |
For easier maintenance and debugging it's recommend to have different ports for |
| 403 |
each endpoint - looking at a `tcpdump` trace the packets can be associated easily. |
| 404 |
|
| 405 |
The examples below will still be using two servers only; please see |
| 406 |
<<s-4node-example>> for a four-node example. |
| 407 |
|
| 408 |
|
| 409 |
[[s-configuring-multiple-paths]] |
| 410 |
==== Configuring transport implementations |
| 411 |
DRBD allows configuring multiple paths per connection, by introducing |
| 412 |
multiple path sections in a connection. Please see the following example: |
| 413 |
|
| 414 |
--------------------- |
| 415 |
resource <resource> { |
| 416 |
... |
| 417 |
connection { |
| 418 |
path { |
| 419 |
host alpha address 192.168.41.1:7900; |
| 420 |
host bravo address 192.168.41.2:7900; |
| 421 |
} |
| 422 |
path { |
| 423 |
host alpha address 192.168.42.1:7900; |
| 424 |
host bravo address 192.168.42.2:7900; |
| 425 |
} |
| 426 |
} |
| 427 |
... |
| 428 |
} |
| 429 |
--------------------- |
| 430 |
|
| 431 |
Obviously the two endpoint hostname need to be equal in all paths of |
| 432 |
a connection. Paths may be on different IPs (potential different NICs) |
| 433 |
or may only be on different ports. |
| 434 |
|
| 435 |
The TCP transport uses one path at a time. In case the backing TCP |
| 436 |
connections get dropped, or show timeouts, the TCP transport implementation |
| 437 |
tries to establish a connection over the next path. It goes over all paths |
| 438 |
in a round-robin fashion until a connection gets established. |
| 439 |
|
| 440 |
The RDMA transport uses all paths of a connection concurrently and it |
| 441 |
balances the network traffic between the paths evenly. |
| 442 |
|
| 443 |
[[s-configuring-transports]] |
| 444 |
==== Configuring transport implementations |
| 445 |
DRBD supports multiple network transports. A transport implementation can be |
| 446 |
configured for each connection of a resource. |
| 447 |
|
| 448 |
[[s-tcp_ip]] |
| 449 |
===== TCP/IP |
| 450 |
--------------------- |
| 451 |
resource <resource> { |
| 452 |
net { |
| 453 |
transport "tcp"; |
| 454 |
} |
| 455 |
... |
| 456 |
} |
| 457 |
--------------------- |
| 458 |
`tcp` is the default transport. I.e. each connection that lacks a transport |
| 459 |
option uses the `tcp` transport. |
| 460 |
|
| 461 |
The `tcp` transport can be configured with the net options: `sndbuf-size`, `rcvbuf-size`, |
| 462 |
`connect-int`, `sock-check-timeo`, `ping-timeo`, `timeout`. |
| 463 |
|
| 464 |
[[s-rdma]] |
| 465 |
===== RDMA |
| 466 |
--------------------- |
| 467 |
resource <resource> { |
| 468 |
net { |
| 469 |
transport "rdma"; |
| 470 |
} |
| 471 |
... |
| 472 |
} |
| 473 |
--------------------- |
| 474 |
The `rdma` transport can be configured with the net options: `sndbuf-size`, `rcvbuf-size`, |
| 475 |
`max_buffers`, `connect-int`, `sock-check-timeo`, `ping-timeo`, `timeout`. |
| 476 |
|
| 477 |
The `rdma` transport is a zero-copy-receive transport. One implication of that is that |
| 478 |
the `max_buffers` configuration options must be set to a value big enough to hold all |
| 479 |
`rcvbuf-size`. |
| 480 |
|
| 481 |
NOTE: `rcvbuf-size` is configured in bytes, while `max_buffers` is configured in pages. For |
| 482 |
optimal performance `max_buffers` should be big enough to hold all of `rcvbuf-size` and |
| 483 |
the amount of data that might be in flight to the backend device at any point in time. |
| 484 |
|
| 485 |
TIP: In case you are using InfiniBand HCAs with the `rdma` transport, you need to |
| 486 |
configure IPoIB as well. The IP address is not used for data transfer, but it is used |
| 487 |
to find the right adapters and ports while establishing the connection. |
| 488 |
|
| 489 |
CAUTION: The configuration options `sndbuf-size`, `rcvbuf-size` are only considered at |
| 490 |
the time a connection is established. I.e. you can change them while the connection is |
| 491 |
established. They will take effect in the moment the connection is re-established. |
| 492 |
|
| 493 |
[[s-performance_considerations_for_rdma]] |
| 494 |
===== Performance considerations for RDMA |
| 495 |
|
| 496 |
By looking at the pseudo file _/sys/kernel/debug/drbd/<resource>/connections/<peer>/transport_, |
| 497 |
the counts of available receive descriptors (rx_desc) and transmit descriptors (tx_desc) |
| 498 |
can be monitored. In case one of the descriptor kinds becomes depleted you should increase |
| 499 |
`sndbuf-size` or `rcvbuf-size`. |
| 500 |
|
| 501 |
[[s-first-time-up]] |
| 502 |
==== Enabling your resource for the first time |
| 503 |
|
| 504 |
After you have completed initial resource configuration as outlined in |
| 505 |
the previous sections, you can bring up your resource. |
| 506 |
|
| 507 |
Each of the following steps must be completed on both nodes. |
| 508 |
|
| 509 |
Please note that with our example config snippets (`resource r0 { ... }`), `<resource>` would be `r0`. |
| 510 |
|
| 511 |
.Create device metadata |
| 512 |
This step must be completed only on initial device |
| 513 |
creation. It initializes DRBD's metadata: |
| 514 |
|
| 515 |
------------------------------------- |
| 516 |
# drbdadm create-md <resource> |
| 517 |
v09 Magic number not found |
| 518 |
Writing meta data... |
| 519 |
initialising activity log |
| 520 |
NOT initializing bitmap |
| 521 |
New drbd meta data block successfully created. |
| 522 |
------------------------------------- |
| 523 |
|
| 524 |
Please note that the number of bitmap slots that are allocated in the meta-data |
| 525 |
depends on the number of hosts for this resource; per default the hosts in the |
| 526 |
resource configuration are counted. |
| 527 |
If all hosts are specified _before_ creating the meta-data, this will "just work"; |
| 528 |
adding bitmap slots for further nodes is possible later, but incurs some manual work. |
| 529 |
|
| 530 |
|
| 531 |
.Enable the resource |
| 532 |
This step associates the resource with its backing device (or devices, |
| 533 |
in case of a multi-volume resource), sets replication parameters, and |
| 534 |
connects the resource to its peer: |
| 535 |
------------------------------------- |
| 536 |
# drbdadm up <resource> |
| 537 |
------------------------------------- |
| 538 |
|
| 539 |
.Observe the status via `drbdadm status` |
| 540 |
``drbdsetup``'s status output |
| 541 |
should now contain information similar to the following: |
| 542 |
|
| 543 |
------------------------------------- |
| 544 |
# drbdadm status r0 |
| 545 |
r0 role:Secondary |
| 546 |
disk:Inconsistent |
| 547 |
bob role:Secondary |
| 548 |
disk:Inconsistent |
| 549 |
------------------------------------- |
| 550 |
|
| 551 |
NOTE: The _Inconsistent/Inconsistent_ disk state is expected at this |
| 552 |
point. |
| 553 |
|
| 554 |
By now, DRBD has successfully allocated both disk and network |
| 555 |
resources and is ready for operation. What it does not know yet is |
| 556 |
which of your nodes should be used as the source of the initial device |
| 557 |
synchronization. |
| 558 |
|
| 559 |
[[s-initial-full-sync]] |
| 560 |
==== The initial device synchronization |
| 561 |
|
| 562 |
There are two more steps required for DRBD to become fully |
| 563 |
operational: |
| 564 |
|
| 565 |
.Select an initial sync source |
| 566 |
If you are dealing with newly-initialized, empty disks, this choice is |
| 567 |
entirely arbitrary. If one of your nodes already has valuable data |
| 568 |
that you need to preserve, however, _it is of crucial importance_ that |
| 569 |
you select that node as your synchronization source. If you do |
| 570 |
initial device synchronization in the wrong direction, you will lose |
| 571 |
that data. Exercise caution. |
| 572 |
|
| 573 |
|
| 574 |
.Start the initial full synchronization |
| 575 |
This step must be performed on only one node, only on initial resource |
| 576 |
configuration, and only on the node you selected as the |
| 577 |
synchronization source. To perform this step, issue this command: |
| 578 |
|
| 579 |
------------------------------------- |
| 580 |
# drbdadm primary --force <resource> |
| 581 |
------------------------------------- |
| 582 |
|
| 583 |
After issuing this command, the initial full synchronization will |
| 584 |
commence. You will be able to monitor its progress via |
| 585 |
`drbdadm status`. It may take some time depending on the size of the |
| 586 |
device. |
| 587 |
|
| 588 |
By now, your DRBD device is fully operational, even before the initial |
| 589 |
synchronization has completed (albeit with slightly reduced |
| 590 |
performance). If you started with empty disks you may now already |
| 591 |
create a filesystem on the device, use it as |
| 592 |
a raw block device, mount it, and perform any other operation you |
| 593 |
would with an accessible block device. |
| 594 |
|
| 595 |
You will now probably want to continue with <<p-work>>, which |
| 596 |
describes common administrative tasks to perform on your resource. |
| 597 |
|
| 598 |
[[s-skip-initial-resync]] |
| 599 |
==== Skipping initial resynchronization |
| 600 |
|
| 601 |
If (and only if) you are starting DRBD resources from scratch (with |
| 602 |
no valueable data on them) you can use following command sequence |
| 603 |
to skip initial resync (don't do that with data you want to keep on |
| 604 |
the devices): |
| 605 |
|
| 606 |
On all nodes: |
| 607 |
|
| 608 |
------------------------------------- |
| 609 |
# drbdadm create-md <res> |
| 610 |
# drbdadm up <res> |
| 611 |
------------------------------------- |
| 612 |
|
| 613 |
The command `drbdadm status` should now show all disks as _Inconsistent_. |
| 614 |
|
| 615 |
Then, on one node execute the following command: |
| 616 |
-- |
| 617 |
------------------------------------- |
| 618 |
# drbdadm new-current-uuid --clear-bitmap <resource>/<volume> |
| 619 |
------------------------------------- |
| 620 |
or |
| 621 |
------------------------------------- |
| 622 |
# drbdsetup new-current-uuid --clear-bitmap <minor> |
| 623 |
------------------------------------- |
| 624 |
-- |
| 625 |
|
| 626 |
Running `drbdadm status` now shows the disks as _UpToDate_ (even tough the |
| 627 |
backing devices might be out of sync). You can now create a file |
| 628 |
system on the disk and start using it. |
| 629 |
|
| 630 |
IMPORTANT: Don't do the above with data you want to keep or it gets |
| 631 |
corrupted. |
| 632 |
|
| 633 |
[[s-using-truck-based-replication]] |
| 634 |
==== Using truck based replication |
| 635 |
|
| 636 |
In order to preseed a remote node with data which is then to be kept |
| 637 |
synchronized, and to skip the initial full device synchronization, follow |
| 638 |
these steps. |
| 639 |
|
| 640 |
This assumes that your local node has a configured, but disconnected |
| 641 |
DRBD resource in the _Primary_ role. That is to say, device |
| 642 |
configuration is completed, identical `drbd.conf` copies exist on both |
| 643 |
nodes, and you have issued the commands for |
| 644 |
<<s-initial-full-sync,initial resource promotion>> on your local node |
| 645 |
-- but the remote node is not connected yet. |
| 646 |
|
| 647 |
|
| 648 |
* On the local node, issue the following command: |
| 649 |
+ |
| 650 |
-- |
| 651 |
------------------------------------- |
| 652 |
# drbdadm new-current-uuid --clear-bitmap <resource>/<volume> |
| 653 |
------------------------------------- |
| 654 |
or |
| 655 |
------------------------------------- |
| 656 |
# drbdsetup new-current-uuid --clear-bitmap <minor> |
| 657 |
------------------------------------- |
| 658 |
-- |
| 659 |
|
| 660 |
* Create a consistent, verbatim copy of the resource's data _and its |
| 661 |
metadata_. You may do so, for example, by removing a hot-swappable |
| 662 |
drive from a RAID-1 mirror. You would, of course, replace it with a |
| 663 |
fresh drive, and rebuild the RAID set, to ensure continued |
| 664 |
redundancy. But the removed drive is a verbatim copy that can now be |
| 665 |
shipped off site. If your local block device supports snapshot |
| 666 |
copies (such as when using DRBD on top of LVM), you may also create |
| 667 |
a bitwise copy of that snapshot using `dd`. |
| 668 |
|
| 669 |
|
| 670 |
* On the local node, issue: |
| 671 |
+ |
| 672 |
-- |
| 673 |
------------------------------------- |
| 674 |
# drbdadm new-current-uuid <resource> |
| 675 |
------------------------------------- |
| 676 |
or the matching `drbdsetup` command. |
| 677 |
|
| 678 |
Note the absence of the `--clear-bitmap` option in this second |
| 679 |
invocation. |
| 680 |
-- |
| 681 |
|
| 682 |
* Physically transport the copies to the remote peer location. |
| 683 |
|
| 684 |
* Add the copies to the remote node. This may again be a matter of |
| 685 |
plugging a physical disk, or grafting a bitwise copy of your shipped |
| 686 |
data onto existing storage on the remote node. Be sure to restore |
| 687 |
or copy not only your replicated data, but also the associated DRBD |
| 688 |
metadata. If you fail to do so, the disk shipping process is moot. |
| 689 |
|
| 690 |
* On the new node we need to fix the node ID in the meta data, and exchange |
| 691 |
the peer-node info for the two nodes. Please see the following lines as |
| 692 |
example for changing node id from 2 to 1 on a resource `r0` volume `0`. |
| 693 |
+ |
| 694 |
-- |
| 695 |
|
| 696 |
This must be done while the volume is not in use. |
| 697 |
|
| 698 |
You need to edit the first 4 lines to match your needs. V is the |
| 699 |
resource name with the volume number. NODE_FROM is the node ID of |
| 700 |
the node the data originates from. NOTE_TO is the node ID of the |
| 701 |
node where data will be replicated to. META_DATA_LOCATION is the |
| 702 |
location of the meta data which might be internal or flex-external. |
| 703 |
|
| 704 |
----------- |
| 705 |
V=r0/0 |
| 706 |
NODE_FROM=2 |
| 707 |
NODE_TO=1 |
| 708 |
META_DATA_LOCATION=internal |
| 709 |
|
| 710 |
drbdadm -- --force dump-md $V > /tmp/md_orig.txt |
| 711 |
sed -e "s/node-id $NODE_FROM/node-id $NODE_TO/" \ |
| 712 |
-e "s/^peer.$NODE_FROM. /peer-NEW /" \ |
| 713 |
-e "s/^peer.$NODE_TO. /peer[$NODE_FROM] /" \ |
| 714 |
-e "s/^peer-NEW /peer[$NODE_TO] /" \ |
| 715 |
< /tmp/md_orig.txt > /tmp/md.txt |
| 716 |
|
| 717 |
drbdmeta --force $(drbdadm sh-minor $V) v09 $(drbdadm sh-md-dev $V) $META_DATA_LOCATION restore-md /tmp/md.txt |
| 718 |
----------- |
| 719 |
|
| 720 |
.NOTE |
| 721 |
`drbdmeta` before 8.9.7 cannot cope with out-of-order `peer` sections; you'll |
| 722 |
need to exchange the blocks via an editor. |
| 723 |
|
| 724 |
-- |
| 725 |
|
| 726 |
* Bring up the resource on the remote node: |
| 727 |
+ |
| 728 |
------------------------------------- |
| 729 |
# drbdadm up <resource> |
| 730 |
------------------------------------- |
| 731 |
|
| 732 |
After the two peers connect, they will not initiate a full device |
| 733 |
synchronization. Instead, the automatic synchronization that now |
| 734 |
commences only covers those blocks that changed since the invocation |
| 735 |
of `drbdadm{nbsp}--clear-bitmap{nbsp}new-current-uuid`. |
| 736 |
|
| 737 |
Even if there were _no_ changes whatsoever since then, there may still |
| 738 |
be a brief synchronization period due to areas covered by the |
| 739 |
<<s-activity-log,Activity Log>> being rolled back on the new |
| 740 |
Secondary. This may be mitigated by the use of |
| 741 |
<<p-checksum-sync,checksum-based synchronization>>. |
| 742 |
|
| 743 |
You may use this same procedure regardless of whether the resource is |
| 744 |
a regular DRBD resource, or a stacked resource. For stacked resources, |
| 745 |
simply add the `-S` or `--stacked` option to `drbdadm`. |
| 746 |
|
| 747 |
|
| 748 |
[[s-4node-example]] |
| 749 |
==== Example configuration for four nodes |
| 750 |
|
| 751 |
Here is an example for a four-node cluster. |
| 752 |
|
| 753 |
[[s-connection-mesh]] |
| 754 |
---------------------- |
| 755 |
resource r0 { |
| 756 |
device minor 0; |
| 757 |
disk /dev/vg/r0; |
| 758 |
meta-disk internal; |
| 759 |
|
| 760 |
on store1 { |
| 761 |
address 10.1.10.1:7100; |
| 762 |
node-id 1; |
| 763 |
} |
| 764 |
on store2 { |
| 765 |
address 10.1.10.2:7100; |
| 766 |
node-id 2; |
| 767 |
} |
| 768 |
on store3 { |
| 769 |
address 10.1.10.3:7100; |
| 770 |
node-id 3; |
| 771 |
} |
| 772 |
on store4 { |
| 773 |
address 10.1.10.4:7100; |
| 774 |
node-id 4; |
| 775 |
} |
| 776 |
|
| 777 |
connection-mesh { |
| 778 |
hosts store1 store2 store3 store4; |
| 779 |
} |
| 780 |
} |
| 781 |
---------------------- |
| 782 |
|
| 783 |
In case you want to see the `connection-mesh` configuration expanded, try `drbdadm dump _<resource>_ -v`. |
| 784 |
|
| 785 |
|
| 786 |
[[s-connection-mesh-distinct-interfaces]] |
| 787 |
As another example, if the four nodes have enough interfaces to provide |
| 788 |
a complete mesh via direct linksfootnote:[i.e. three crossover and at least one |
| 789 |
outgoing/management interface], you can specify the IP addresses of the |
| 790 |
interfaces: |
| 791 |
|
| 792 |
--------------------------- |
| 793 |
resource r0 { |
| 794 |
... |
| 795 |
|
| 796 |
# store1 has crossover links like 10.99.1x.y |
| 797 |
connection { |
| 798 |
host store1 address 10.99.12.1 port 7012; |
| 799 |
host store2 address 10.99.12.2 port 7021; |
| 800 |
} |
| 801 |
connection { |
| 802 |
host store1 address 10.99.13.1 port 7013; |
| 803 |
host store3 address 10.99.13.3 port 7031; |
| 804 |
} |
| 805 |
connection { |
| 806 |
host store1 address 10.99.14.1 port 7014; |
| 807 |
host store4 address 10.99.14.4 port 7041; |
| 808 |
} |
| 809 |
|
| 810 |
# store2 has crossover links like 10.99.2x.y |
| 811 |
connection { |
| 812 |
host store2 address 10.99.23.2 port 7023; |
| 813 |
host store3 address 10.99.23.3 port 7032; |
| 814 |
} |
| 815 |
connection { |
| 816 |
host store2 address 10.99.24.2 port 7024; |
| 817 |
host store4 address 10.99.24.4 port 7042; |
| 818 |
} |
| 819 |
|
| 820 |
# store3 has crossover links like 10.99.3x.y |
| 821 |
connection { |
| 822 |
host store3 address 10.99.34.3 port 7034; |
| 823 |
host store4 address 10.99.34.4 port 7043; |
| 824 |
} |
| 825 |
} |
| 826 |
--------------------------- |
| 827 |
|
| 828 |
Please note the numbering scheme used for the IP addresses and ports. Another |
| 829 |
resource could use the same IP addresses, but ports `71__xy__`, the next one |
| 830 |
`72__xy__`, and so on. |