[[ch-configure]] === Configuring DRBD [[s-prepare-storage]] ==== Preparing your lower-level storage After you have installed DRBD, you must set aside a roughly identically sized storage area on both cluster nodes. This will become the _lower-level device_ for your DRBD resource. You may use any type of block device found on your system for this purpose. Typical examples include: * A hard drive partition (or a full physical hard drive), * a software RAID device, * an LVM Logical Volume or any other block device configured by the Linux device-mapper infrastructure, * any other block device type found on your system. You may also use _resource stacking_, meaning you can use one DRBD device as a lower-level device for another. Some specific considerations apply to stacked resources; their configuration is covered in detail in <>. NOTE: While it is possible to use loop devices as lower-level devices for DRBD, doing so is not recommended due to deadlock issues. It is _not_ necessary for this storage area to be empty before you create a DRBD resource from it. In fact it is a common use case to create a two-node cluster from a previously non-redundant single-server system using DRBD (some caveats apply -- please refer to <> if you are planning to do this). For the purposes of this guide, we assume a very simple setup: * Both hosts have a free (currently unused) partition named `/dev/sda7`. * We are using <>. [[s-prepare-network]] ==== Preparing your network configuration It is recommended, though not strictly required, that you run your DRBD replication over a dedicated connection. At the time of this writing, the most reasonable choice for this is a direct, back-to-back, Gigabit Ethernet connection. When DRBD is run over switches, use of redundant components and the _bonding_ driver (in _active-backup_ mode) is recommended. It is generally not recommended to run DRBD replication via routers, for reasons of fairly obvious performance drawbacks (adversely affecting both throughput and latency). In terms of local firewall considerations, it is important to understand that DRBD (by convention) uses TCP ports from 7788 upwards, with every resource listening on a separate port. DRBD uses _two_ TCP connections for every resource configured. For proper DRBD functionality, it is required that these connections are allowed by your firewall configuration. Security considerations other than firewalling may also apply if a Mandatory Access Control (MAC) scheme such as SELinux or AppArmor is enabled. You may have to adjust your local security policy so it does not keep DRBD from functioning properly. You must, of course, also ensure that the TCP ports for DRBD are not already used by another application. It is not (yet) possible to configure a DRBD resource to support more than one TCP connection pair. If you want to provide for DRBD connection load-balancing or redundancy, you can easily do so at the Ethernet level (again, using the _bonding_ driver). For the purposes of this guide, we assume a very simple setup: * Our two DRBD hosts each have a currently unused network interface, `eth1`, with IP addresses `10.1.1.31` and `10.1.1.32` assigned to it, respectively. * No other services are using TCP ports 7788 through 7799 on either host. * The local firewall configuration allows both inbound and outbound TCP connections between the hosts over these ports. [[s-configure-resource]] ==== Configuring your resource All aspects of DRBD are controlled in its configuration file, `/etc/drbd.conf`. Normally, this configuration file is just a skeleton with the following contents: ------------------------------------- include "/etc/drbd.d/global_common.conf"; include "/etc/drbd.d/*.res"; ------------------------------------- By convention, `/etc/drbd.d/global_common.conf` contains the <> and <> sections of the DRBD configuration, whereas the `.res` files contain one <> section each. It is also possible to use `drbd.conf` as a flat configuration file without any `include` statements at all. Such a configuration, however, quickly becomes cluttered and hard to manage, which is why the multiple-file approach is the preferred one. Regardless of which approach you employ, you should always make sure that `drbd.conf`, and any other files it includes, are _exactly identical_ on all participating cluster nodes. The DRBD source tarball contains an example configuration file in the `scripts` subdirectory. Binary installation packages will either install this example configuration directly in `/etc`, or in a package-specific documentation directory such as `/usr/share/doc/packages/drbd`. This section describes only those few aspects of the configuration file which are absolutely necessary to understand in order to get DRBD up and running. The configuration file's syntax and contents are documented in great detail in the man page of `drbd.conf`. [[s-drbdconf-example]] ===== Example configuration For the purposes of this guide, we assume a minimal setup in line with the examples given in the previous sections: .Simple DRBD configuration (`/etc/drbd.d/global_common.conf`) ------------------------------------- global { usage-count yes; } common { net { protocol C; } } ------------------------------------- .Simple DRBD resource configuration (`/etc/drbd.d/r0.res`) ------------------------------------- resource "r0" { device minor 1; disk "/dev/sda7"; meta-disk internal; on "alice" { node-id 0; } on "bob" { node-id 1; } connection { host "alice" address 10.1.1.31:7789; host "bob" address 10.1.1.32:7789; } } ------------------------------------- This example configures DRBD in the following fashion: * You "opt in" to be included in DRBD's usage statistics (see <>). * Resources are configured to use fully synchronous replication (<>) unless explicitly specified otherwise. * Our cluster consists of two nodes, 'alice' and 'bob'. * We have a resource arbitrarily named `r0` which uses `/dev/sda7` as the lower-level device, and is configured with <>. * The resource uses TCP port 7789 for its network connections, and binds to the IP addresses 10.1.1.31 and 10.1.1.32, respectively. The configuration above implicitly creates one volume in the resource, numbered zero (`0`). For multiple volumes in one resource, modify the syntax as follows (assuming that the same lower-level storage block devices are used on both nodes): .Multi-volume DRBD resource configuration (`/etc/drbd.d/r0.res`) ------------------------------------- resource "r0" { volume 0 { device minor 1; disk "/dev/sda7"; meta-disk internal; } volume 1 { device minor 2; disk "/dev/sda7"; meta-disk internal; } on "alice" { node-id 0; } on "bob" { node-id 1; volume 1 { disk "/dev/sda9"; } } connection { host "alice" address 10.1.1.31:7789; host "bob" address 10.1.1.32:7789; } } ------------------------------------- * Host sections ('on' keyword) inherit _volume_ sections from the resource level. They may contain _volume_ themselves, these values have precedence over inherited values. NOTE: Volumes may also be added to existing resources on the fly. For an example see <>. For compatibility with older releases of DRBD it supports also drbd-8.4 like configuration files. .An old(8.4) style configuration file ------------------------------------- resource r0 { on alice { device /dev/drbd1; disk /dev/sda7; meta-disk internal; address 10.1.1.31:7789; } on bob { device /dev/drbd1; disk /dev/sda7; meta-disk internal; address 10.1.1.32:7789; } ------------------------------------- * Strings that do not contain keywords, might be gives withouth double quotes `"`. * The old version to specify the device, was to give a string containing the name of the resulting device file. * Two node configurations get node numbers assigned by drbdadm. * A pure two node configuration gets an implicit connection. [[s-drbdconf-global]] ===== The `global` section This section is allowed only once in the configuration. It is normally in the `/etc/drbd.d/global_common.conf` file. In a single-file configuration, it should go to the very top of the configuration file. Of the few options available in this section, only one is of relevance to most users: [[fp-usage-count]] .`usage-count` The DRBD project keeps statistics about the usage of various DRBD versions. This is done by contacting an HTTP server every time a new DRBD version is installed on a system. This can be disabled by setting `usage-count no;`. The default is `usage-count ask;` which will prompt you every time you upgrade DRBD. DRBD's usage statistics are, of course, publicly available: see http://usage.drbd.org. [[s-drbdconf-common]] ===== The `common` section This section provides a shorthand method to define configuration settings inherited by every resource. It is normally found in `/etc/drbd.d/global_common.conf`. You may define any option you can also define on a per-resource basis. Including a `common` section is not strictly required, but strongly recommended if you are using more than one resource. Otherwise, the configuration quickly becomes convoluted by repeatedly-used options. In the example above, we included `net { protocol C; }` in the `common` section, so every resource configured (including `r0`) inherits this option unless it has another `protocol` option configured explicitly. For other synchronization protocols available, see <>. [[s-drbdconf-resource]] ===== The `resource` sections A per-resource configuration file is usually named `/etc/drbd.d/__resource__.res`. Any DRBD resource you define must be named by specifying a resource name in the configuration. The convention is to use only letters, digits, and the underscore; while it is technically possible to use other characters as well, you won't like the result if you ever happen stumble to need the more specific `__resource__:___peer__/__volume__` syntax. Every resource configuration must also have at least two `on _host_` sub-sections, one for every cluster node. All other configuration settings are either inherited from the `common` section (if it exists), or derived from DRBD's default settings. In addition, options with equal values on all hosts can be specified directly in the `resource` section. Thus, we can further condense our example configuration as follows: ------------------------------------- resource "r0" { device minor 1; disk "/dev/sda7"; meta-disk internal; on "alice" { address 10.1.1.31:7789; } on "bob" { address 10.1.1.32:7789; } } ------------------------------------- [[s-drbdconf-conns]] ==== Defining network connections Currently the communication links in DRBD 9 must build a full mesh, i.e. in every resource every node must have a direct connection to every other node (excluding itself, of course). For the simple case of two hosts `drbdadm` will insert the (single) network connection by itself, for ease of use and backwards compatibility. The net effect of this is a quadratic number of network connections over hosts. For the "traditional" two nodes one connection is needed; for three hosts there are three node pairs; for four, six pairs; 5 hosts: 10 connections, and so on. For (the current) maximum of 16 nodes there'll be 120 host pairs to connect. [[eq-connection-mesh]] .Number of connections for _N_ hosts image::images/connection-mesh.svg[] An example configuration file for three hosts would be this: ------------------------------------- resource r0 { device minor 1; disk "/dev/sda7"; meta-disk internal; on alice { address 10.1.1.31:7000; node-id 0; } on bob { address 10.1.1.32:7000; node-id 1; } on charlie { address 10.1.1.33:7000; node-id 2; } connection-mesh { hosts alice bob charlie; } } ------------------------------------- If you have got enough network cards in your servers, you can create direct cross-over links between server pairs. A single four-port ethernet card allows to have a single management interface, and to connect 3 other servers, to get a full mesh for 4 cluster nodes. In this case you can specify a different IP address to use the direct link: -------------------------------- resource r0 { ... connection { host alice address 10.1.2.1:7010; host bob address 10.1.2.2:7001; } connection { host alice address 10.1.3.1:7020; host charlie address 10.1.3.2:7002; } connection { host bob address 10.1.4.1:7021; host charlie address 10.1.4.2:7012; } } -------------------------------- For easier maintenance and debugging it's recommend to have different ports for each endpoint - looking at a `tcpdump` trace the packets can be associated easily. The examples below will still be using two servers only; please see <> for a four-node example. [[s-configuring-multiple-paths]] ==== Configuring transport implementations DRBD allows configuring multiple paths per connection, by introducing multiple path sections in a connection. Please see the following example: --------------------- resource { ... connection { path { host alpha address 192.168.41.1:7900; host bravo address 192.168.41.2:7900; } path { host alpha address 192.168.42.1:7900; host bravo address 192.168.42.2:7900; } } ... } --------------------- Obviously the two endpoint hostname need to be equal in all paths of a connection. Paths may be on different IPs (potential different NICs) or may only be on different ports. The TCP transport uses one path at a time. In case the backing TCP connections get dropped, or show timeouts, the TCP transport implementation tries to establish a connection over the next path. It goes over all paths in a round-robin fashion until a connection gets established. The RDMA transport uses all paths of a connection concurrently and it balances the network traffic between the paths evenly. [[s-configuring-transports]] ==== Configuring transport implementations DRBD supports multiple network transports. A transport implementation can be configured for each connection of a resource. [[s-tcp_ip]] ===== TCP/IP --------------------- resource { net { transport "tcp"; } ... } --------------------- `tcp` is the default transport. I.e. each connection that lacks a transport option uses the `tcp` transport. The `tcp` transport can be configured with the net options: `sndbuf-size`, `rcvbuf-size`, `connect-int`, `sock-check-timeo`, `ping-timeo`, `timeout`. [[s-rdma]] ===== RDMA --------------------- resource { net { transport "rdma"; } ... } --------------------- The `rdma` transport can be configured with the net options: `sndbuf-size`, `rcvbuf-size`, `max_buffers`, `connect-int`, `sock-check-timeo`, `ping-timeo`, `timeout`. The `rdma` transport is a zero-copy-receive transport. One implication of that is that the `max_buffers` configuration options must be set to a value big enough to hold all `rcvbuf-size`. NOTE: `rcvbuf-size` is configured in bytes, while `max_buffers` is configured in pages. For optimal performance `max_buffers` should be big enough to hold all of `rcvbuf-size` and the amount of data that might be in flight to the backend device at any point in time. TIP: In case you are using InfiniBand HCAs with the `rdma` transport, you need to configure IPoIB as well. The IP address is not used for data transfer, but it is used to find the right adapters and ports while establishing the connection. CAUTION: The configuration options `sndbuf-size`, `rcvbuf-size` are only considered at the time a connection is established. I.e. you can change them while the connection is established. They will take effect in the moment the connection is re-established. [[s-performance_considerations_for_rdma]] ===== Performance considerations for RDMA By looking at the pseudo file _/sys/kernel/debug/drbd//connections//transport_, the counts of available receive descriptors (rx_desc) and transmit descriptors (tx_desc) can be monitored. In case one of the descriptor kinds becomes depleted you should increase `sndbuf-size` or `rcvbuf-size`. [[s-first-time-up]] ==== Enabling your resource for the first time After you have completed initial resource configuration as outlined in the previous sections, you can bring up your resource. Each of the following steps must be completed on both nodes. Please note that with our example config snippets (`resource r0 { ... }`), `` would be `r0`. .Create device metadata This step must be completed only on initial device creation. It initializes DRBD's metadata: ------------------------------------- # drbdadm create-md v09 Magic number not found Writing meta data... initialising activity log NOT initializing bitmap New drbd meta data block successfully created. ------------------------------------- Please note that the number of bitmap slots that are allocated in the meta-data depends on the number of hosts for this resource; per default the hosts in the resource configuration are counted. If all hosts are specified _before_ creating the meta-data, this will "just work"; adding bitmap slots for further nodes is possible later, but incurs some manual work. .Enable the resource This step associates the resource with its backing device (or devices, in case of a multi-volume resource), sets replication parameters, and connects the resource to its peer: ------------------------------------- # drbdadm up ------------------------------------- .Observe the status via `drbdadm status` ``drbdsetup``'s status output should now contain information similar to the following: ------------------------------------- # drbdadm status r0 r0 role:Secondary disk:Inconsistent bob role:Secondary disk:Inconsistent ------------------------------------- NOTE: The _Inconsistent/Inconsistent_ disk state is expected at this point. By now, DRBD has successfully allocated both disk and network resources and is ready for operation. What it does not know yet is which of your nodes should be used as the source of the initial device synchronization. [[s-initial-full-sync]] ==== The initial device synchronization There are two more steps required for DRBD to become fully operational: .Select an initial sync source If you are dealing with newly-initialized, empty disks, this choice is entirely arbitrary. If one of your nodes already has valuable data that you need to preserve, however, _it is of crucial importance_ that you select that node as your synchronization source. If you do initial device synchronization in the wrong direction, you will lose that data. Exercise caution. .Start the initial full synchronization This step must be performed on only one node, only on initial resource configuration, and only on the node you selected as the synchronization source. To perform this step, issue this command: ------------------------------------- # drbdadm primary --force ------------------------------------- After issuing this command, the initial full synchronization will commence. You will be able to monitor its progress via `drbdadm status`. It may take some time depending on the size of the device. By now, your DRBD device is fully operational, even before the initial synchronization has completed (albeit with slightly reduced performance). If you started with empty disks you may now already create a filesystem on the device, use it as a raw block device, mount it, and perform any other operation you would with an accessible block device. You will now probably want to continue with <>, which describes common administrative tasks to perform on your resource. [[s-skip-initial-resync]] ==== Skipping initial resynchronization If (and only if) you are starting DRBD resources from scratch (with no valueable data on them) you can use following command sequence to skip initial resync (don't do that with data you want to keep on the devices): On all nodes: ------------------------------------- # drbdadm create-md # drbdadm up ------------------------------------- The command `drbdadm status` should now show all disks as _Inconsistent_. Then, on one node execute the following command: -- ------------------------------------- # drbdadm new-current-uuid --clear-bitmap / ------------------------------------- or ------------------------------------- # drbdsetup new-current-uuid --clear-bitmap ------------------------------------- -- Running `drbdadm status` now shows the disks as _UpToDate_ (even tough the backing devices might be out of sync). You can now create a file system on the disk and start using it. IMPORTANT: Don't do the above with data you want to keep or it gets corrupted. [[s-using-truck-based-replication]] ==== Using truck based replication In order to preseed a remote node with data which is then to be kept synchronized, and to skip the initial full device synchronization, follow these steps. This assumes that your local node has a configured, but disconnected DRBD resource in the _Primary_ role. That is to say, device configuration is completed, identical `drbd.conf` copies exist on both nodes, and you have issued the commands for <> on your local node -- but the remote node is not connected yet. * On the local node, issue the following command: + -- ------------------------------------- # drbdadm new-current-uuid --clear-bitmap / ------------------------------------- or ------------------------------------- # drbdsetup new-current-uuid --clear-bitmap ------------------------------------- -- * Create a consistent, verbatim copy of the resource's data _and its metadata_. You may do so, for example, by removing a hot-swappable drive from a RAID-1 mirror. You would, of course, replace it with a fresh drive, and rebuild the RAID set, to ensure continued redundancy. But the removed drive is a verbatim copy that can now be shipped off site. If your local block device supports snapshot copies (such as when using DRBD on top of LVM), you may also create a bitwise copy of that snapshot using `dd`. * On the local node, issue: + -- ------------------------------------- # drbdadm new-current-uuid ------------------------------------- or the matching `drbdsetup` command. Note the absence of the `--clear-bitmap` option in this second invocation. -- * Physically transport the copies to the remote peer location. * Add the copies to the remote node. This may again be a matter of plugging a physical disk, or grafting a bitwise copy of your shipped data onto existing storage on the remote node. Be sure to restore or copy not only your replicated data, but also the associated DRBD metadata. If you fail to do so, the disk shipping process is moot. * On the new node we need to fix the node ID in the meta data, and exchange the peer-node info for the two nodes. Please see the following lines as example for changing node id from 2 to 1 on a resource `r0` volume `0`. + -- This must be done while the volume is not in use. You need to edit the first 4 lines to match your needs. V is the resource name with the volume number. NODE_FROM is the node ID of the node the data originates from. NOTE_TO is the node ID of the node where data will be replicated to. META_DATA_LOCATION is the location of the meta data which might be internal or flex-external. ----------- V=r0/0 NODE_FROM=2 NODE_TO=1 META_DATA_LOCATION=internal drbdadm -- --force dump-md $V > /tmp/md_orig.txt sed -e "s/node-id $NODE_FROM/node-id $NODE_TO/" \ -e "s/^peer.$NODE_FROM. /peer-NEW /" \ -e "s/^peer.$NODE_TO. /peer[$NODE_FROM] /" \ -e "s/^peer-NEW /peer[$NODE_TO] /" \ < /tmp/md_orig.txt > /tmp/md.txt drbdmeta --force $(drbdadm sh-minor $V) v09 $(drbdadm sh-md-dev $V) $META_DATA_LOCATION restore-md /tmp/md.txt ----------- .NOTE `drbdmeta` before 8.9.7 cannot cope with out-of-order `peer` sections; you'll need to exchange the blocks via an editor. -- * Bring up the resource on the remote node: + ------------------------------------- # drbdadm up ------------------------------------- After the two peers connect, they will not initiate a full device synchronization. Instead, the automatic synchronization that now commences only covers those blocks that changed since the invocation of `drbdadm{nbsp}--clear-bitmap{nbsp}new-current-uuid`. Even if there were _no_ changes whatsoever since then, there may still be a brief synchronization period due to areas covered by the <> being rolled back on the new Secondary. This may be mitigated by the use of <>. You may use this same procedure regardless of whether the resource is a regular DRBD resource, or a stacked resource. For stacked resources, simply add the `-S` or `--stacked` option to `drbdadm`. [[s-4node-example]] ==== Example configuration for four nodes Here is an example for a four-node cluster. [[s-connection-mesh]] ---------------------- resource r0 { device minor 0; disk /dev/vg/r0; meta-disk internal; on store1 { address 10.1.10.1:7100; node-id 1; } on store2 { address 10.1.10.2:7100; node-id 2; } on store3 { address 10.1.10.3:7100; node-id 3; } on store4 { address 10.1.10.4:7100; node-id 4; } connection-mesh { hosts store1 store2 store3 store4; } } ---------------------- In case you want to see the `connection-mesh` configuration expanded, try `drbdadm dump __ -v`. [[s-connection-mesh-distinct-interfaces]] As another example, if the four nodes have enough interfaces to provide a complete mesh via direct linksfootnote:[i.e. three crossover and at least one outgoing/management interface], you can specify the IP addresses of the interfaces: --------------------------- resource r0 { ... # store1 has crossover links like 10.99.1x.y connection { host store1 address 10.99.12.1 port 7012; host store2 address 10.99.12.2 port 7021; } connection { host store1 address 10.99.13.1 port 7013; host store3 address 10.99.13.3 port 7031; } connection { host store1 address 10.99.14.1 port 7014; host store4 address 10.99.14.4 port 7041; } # store2 has crossover links like 10.99.2x.y connection { host store2 address 10.99.23.2 port 7023; host store3 address 10.99.23.3 port 7032; } connection { host store2 address 10.99.24.2 port 7024; host store4 address 10.99.24.4 port 7042; } # store3 has crossover links like 10.99.3x.y connection { host store3 address 10.99.34.3 port 7034; host store4 address 10.99.34.4 port 7043; } } --------------------------- Please note the numbering scheme used for the IP addresses and ports. Another resource could use the same IP addresses, but ports `71__xy__`, the next one `72__xy__`, and so on.