[[ch-features]] == DRBD Features This chapter discusses various useful DRBD features, and gives some background information about them. Some of these features will be important to most users, some will only be relevant in very specific deployment scenarios. <> and <> contain instructions on how to enable and use these features in day-to-day operation. [[s-single-primary-mode]] === Single-primary mode In single-primary mode, a resource is, at any given time, in the primary role on only one cluster member. Since it is guaranteed that only one cluster node manipulates the data at any moment, this mode can be used with any conventional file system (ext3, ext4, XFS etc.). Deploying DRBD in single-primary mode is the canonical approach for High-Availability (fail-over capable) clusters. [[s-dual-primary-mode]] === Dual-primary mode In dual-primary mode a resource can be in the primary role on two nodes at a time. Since concurrent access to the data is thus possible, this mode usually requires the use of a shared cluster file system that utilizes a distributed lock manager. ifndef::drbd-only[] Examples include <> and <>. endif::drbd-only[] Deploying DRBD in dual-primary mode is the preferred approach for load-balancing clusters which require concurrent data access from two nodes, e.g. virtualization environments with a need for live-migration. This mode is disabled by default, and must be enabled explicitly in DRBD's configuration file. See <> for information on enabling dual-primary mode for specific resources. [[s-replication-protocols]] === Replication modes DRBD supports three distinct replication modes, allowing three degrees of replication synchronicity. [[fp-protocol-a]] .Protocol A Asynchronous replication protocol. Local write operations on the primary node are considered completed as soon as the local disk write has finished, and the replication packet has been placed in the local TCP send buffer. In the event of forced fail-over, data loss may occur. The data on the standby node is consistent after fail-over; however, the most recent updates performed prior to the crash could be lost. Protocol A is most often used in long distance replication scenarios. ifndef::drbd-only,de-brand[] When used in combination with DRBD Proxy it makes an effective disaster recovery solution. See <>, for more information. endif::[] [[fp-protocol-b]] .Protocol B Memory synchronous (semi-synchronous) replication protocol. Local write operations on the primary node are considered completed as soon as the local disk write has occurred, and the replication packet has reached the peer node. Normally, no writes are lost in case of forced fail-over. However, in the event of simultaneous power failure on both nodes *and* concurrent, irreversible destruction of the primary's data store, the most recent writes completed on the primary may be lost. [[fp-protocol-c]] .Protocol C Synchronous replication protocol. Local write operations on the primary node are considered completed only after both the local and the remote disk write(s) have been confirmed. As a result, loss of a single node is guaranteed not to lead to any data loss. Data loss is, of course, inevitable even with this replication protocol if all nodes (resp. their storage subsystems) are irreversibly destroyed at the same time. By far, the most commonly used replication protocol in DRBD setups is protocol C. The choice of replication protocol influences two factors of your deployment: _protection_ and _latency_. _Throughput_, by contrast, is largely independent of the replication protocol selected. See <> for an example resource configuration which demonstrates replication protocol configuration. [[s-multi-node]] === More than 2-way redundancy With DRBD 9 it's possible to have the data stored simultaneously on more than two cluster nodes. While this has been possible before via <>, in DRBD 9 this is supported out-of-the-box for (currently) up to 16 nodes. (In practice, using 3-, 4- or perhaps 5-way redundancy via DRBD will make other things the leading cause of downtime.) The major difference to the stacking solution is that there's less performance loss, because only one level of data replication is being used. // E.g. if availability for a single node is 99%, for two nodes it might // be 99.99%, for three nodes 99.999% [[s-automatic-promotion]] === Automatic Promotion of Resources Prior to DRBD 9, promoting a resource would be done with the `drbdadm primary` command. With DRBD 9, DRBD will automatically promote a resource to primary role when the `auto-promote` option is enabled, and one of its volumes is mounted or opened for writing. As soon as all volumes are unmounted or closed, the role of the resource changes back to secondary. Automatic promotion will only succeed if the cluster state allows it (that is, if an explicit `drbdadm primary` command would succeed). Otherwise, mounting or opening the device fails as it did prior to DRBD 9. [[s-replication-transports]] === Multiple replication transports DRBD supports multiple network transports. As of now two transport implementations are available: TCP and RDMA. Each transport implementation comes as its own kernel module. [[s-tcp_transport]] ==== TCP Transport The `drbd_transport_tcp.ko` transport implementation is included with the distribution files of drbd itself. As the name implies, this transport implementation uses the TCP/IP protocol to move data between machines. DRBD's replication and synchronization framework socket layer supports multiple low-level transports: .TCP over IPv4 This is the canonical implementation, and DRBD's default. It may be used on any system that has IPv4 enabled. .TCP over IPv6 When configured to use standard TCP sockets for replication and synchronization, DRBD can use also IPv6 as its network protocol. This is equivalent in semantics and performance to IPv4, albeit using a different addressing scheme. .SDP SDP is an implementation of BSD-style sockets for RDMA capable transports such as InfiniBand. SDP was available as part of the OFED stack of most distributions but is now *considered deprecated*. SDP uses an IPv4-style addressing scheme. Employed over an InfiniBand interconnect, SDP provides a high-throughput, low-latency replication network to DRBD. .SuperSockets SuperSockets replace the TCP/IP portions of the stack with a single, monolithic, highly efficient and RDMA capable socket implementation. DRBD can use this socket type for very low latency replication. SuperSockets must run on specific hardware which is currently available from a single vendor, Dolphin Interconnect Solutions. [[s-rdma_transport]] ==== RDMA Transport Alternatively the `drbd_transport_rdma.ko` kernel module is available ifndef::de-brand[] from LINBIT endif::de-brand[] . This transport uses the verbs/RDMA API to move data over InfiniBand HCAs, iWARP capable NICs or RoCE capable NICs. In contrast to the BSD sockets API (used by TCP/IP) the verbs/RDMA API allows data movement with very little CPU involvement. At high transfer rates it might be possible that the CPU load/memory bandwidth of the tcp transport becomes the limiting factor. You can probably achieve higher transfer rates using the rdma transport with appropriate hardware. A transport implementation can be configured for each connection of a resource. See <> for more details. [[s-multiple-paths]] === Multiple paths DRBD allows configuring multiple paths per connection. The TCP transport uses only one path at a time for a connection. The RDMA transport is capable of balancing the network traffic over multiple paths of a single connection. see <> for more details. [[s-resync]] === Efficient synchronization (Re-)synchronization is distinct from device replication. While replication occurs on any write event to a resource in the primary role, synchronization is decoupled from incoming writes. Rather, it affects the device as a whole. Synchronization is necessary if the replication link has been interrupted for any reason, be it due to failure of the primary node, failure of the secondary node, or interruption of the replication link. Synchronization is efficient in the sense that DRBD does not synchronize modified blocks in the order they were originally written, but in linear order, which has the following consequences: * Synchronization is fast, since blocks in which several successive write operations occurred are only synchronized once. * Synchronization is also associated with few disk seeks, as blocks are synchronized according to the natural on-disk block layout. * During synchronization, the data set on the standby node is partly obsolete and partly already updated. This state of data is called _inconsistent_. The service continues to run uninterrupted on the active node, while background synchronization is in progress. IMPORTANT: A node with inconsistent data generally cannot be put into operation, thus it is desirable to keep the time period during which a node is inconsistent as short as possible. DRBD does, however, ship with an LVM integration facility that automates the creation of LVM snapshots immediately before synchronization. This ensures that a _consistent_ copy of the data is always available on the peer, even while synchronization is running. See <> for details on using this facility. [[s-variable-rate-sync]] ==== Variable-rate synchronization In variable-rate synchronization (the default since 8.4), DRBD detects the available bandwidth on the synchronization network, compares it to incoming foreground application I/O, and selects an appropriate synchronization rate based on a fully automatic control loop. See <> for configuration suggestions with regard to variable-rate synchronization. [[s-fixed_rate_synchronization]] ==== Fixed-rate synchronization In fixed-rate synchronization, the amount of data shipped to the synchronizing peer per second (the _synchronization rate_) has a configurable, static upper limit. Based on this limit, you may estimate the expected sync time based on the following simple formula: [[eq-resync-time]] [equation] .Synchronization time image::images/resync-time.svg[] _t~sync~_ is the expected sync time. _D_ is the amount of data to be synchronized, which you are unlikely to have any influence over (this is the amount of data that was modified by your application while the replication link was broken). _R_ is the rate of synchronization, which is configurable -- bounded by the throughput limitations of the replication network and I/O subsystem. See <> for configuration suggestions with regard to fixed-rate synchronization. [[s-checksum-sync]] ==== Checksum-based synchronization [[p-checksum-sync]] The efficiency of DRBD's synchronization algorithm may be further enhanced by using data digests, also known as checksums. When using checksum-based synchronization, then rather than performing a brute-force overwrite of blocks marked out of sync, DRBD _reads_ blocks before synchronizing them and computes a hash of the contents currently found on disk. It then compares this hash with one computed from the same sector on the peer, and omits re-writing this block if the hashes match. This can dramatically cut down synchronization times in situation where a filesystem re-writes a sector with identical contents while DRBD is in disconnected mode. See <> for configuration suggestions with regard to synchronization. [[s-suspended-replication]] === Suspended replication If properly configured, DRBD can detect if the replication network is congested, and _suspend_ replication in this case. In this mode, the primary node "pulls ahead" of the secondary -- temporarily going out of sync, but still leaving a consistent copy on the secondary. When more bandwidth becomes available, replication automatically resumes and a background synchronization takes place. Suspended replication is typically enabled over links with variable bandwidth, such as wide area replication over shared connections between data centers or cloud instances. See <> for details on congestion policies and suspended replication. [[s-online-verify]] === On-line device verification On-line device verification enables users to do a block-by-block data integrity check between nodes in a very efficient manner. Note that _efficient_ refers to efficient use of network bandwidth here, and to the fact that verification does not break redundancy in any way. On-line verification is still a resource-intensive operation, with a noticeable impact on CPU utilization and load average. It works by one node (the _verification source_) sequentially calculating a cryptographic digest of every block stored on the lower-level storage device of a particular resource. DRBD then transmits that digest to the peer node(s) (the _verification target(s)_), where it is checked against a digest of the local copy of the affected block. If the digests do not match, the block is marked out-of-sync and may later be synchronized. Because DRBD transmits just the digests, not the full blocks, on-line verification uses network bandwidth very efficiently. The process is termed _on-line_ verification because it does not require that the DRBD resource being verified is unused at the time of verification. Thus, though it does carry a slight performance penalty while it is running, on-line verification does not cause service interruption or system down time -- neither during the verification run nor during subsequent synchronization. It is a common use case to have on-line verification managed by the local cron daemon, running it, for example, once a week or once a month. See <> for information on how to enable, invoke, and automate on-line verification. [[s-integrity-check]] === Replication traffic integrity checking DRBD optionally performs end-to-end message integrity checking using cryptographic message digest algorithms such as MD5, SHA-1, or CRC-32C. These message digest algorithms are *not* provided by DRBD, but by the Linux kernel crypto API; DRBD merely uses them. Thus, DRBD is capable of utilizing any message digest algorithm available in a particular system's kernel configuration. With this feature enabled, DRBD generates a message digest of every data block it replicates to the peer, which the peer then uses to verify the integrity of the replication packet. If the replicated block can not be verified against the digest, the connection is dropped and immediately re-established; because of the bitmap the typical result is a retransmission. Thus, DRBD replication is protected against several error sources, all of which, if unchecked, would potentially lead to data corruption during the replication process: * Bitwise errors ("bit flips") occurring on data in transit between main memory and the network interface on the sending node (which goes undetected by TCP checksumming if it is offloaded to the network card, as is common in recent implementations); * Bit flips occurring on data in transit from the network interface to main memory on the receiving node (the same considerations apply for TCP checksum offloading); * Any form of corruption due to a race conditions or bugs in network interface firmware or drivers; * Bit flips or random corruption injected by some reassembling network component between nodes (if not using direct, back-to-back connections). See <> for information on how to enable replication traffic integrity checking. [[s-split-brain-notification-and-recovery]] === Split brain notification and automatic recovery Split brain is a situation where, due to temporary failure of all network links between cluster nodes, and possibly due to intervention by a cluster management software or human error, both nodes switched to the _Primary_ role while disconnected. This is a potentially harmful state, as it implies that modifications to the data might have been made on either node, without having been replicated to the peer. Thus, it is likely in this situation that two diverging sets of data have been created, which cannot be trivially merged. DRBD split brain is distinct from cluster split brain, which is the loss of all connectivity between hosts managed by a distributed cluster management application such as Pacemaker. To avoid confusion, this guide uses the following convention: * _Split brain_ refers to DRBD split brain as described in the paragraph above. * Loss of all cluster connectivity is referred to as a _cluster partition_, an alternative term for cluster split brain. DRBD allows for automatic operator notification (by email or other means) when it detects split brain. See <> for details on how to configure this feature. While the recommended course of action in this scenario is to <> the split brain and then eliminate its root cause, it may be desirable, in some cases, to automate the process. DRBD has several resolution algorithms available for doing so: * *Discarding modifications made on the younger primary.* In this mode, when the network connection is re-established and split brain is discovered, DRBD will discard modifications made, in the meantime, on the node which switched to the primary role _last_. * *Discarding modifications made on the older primary.* In this mode, DRBD will discard modifications made, in the meantime, on the node which switched to the primary role _first_. * *Discarding modifications on the primary with fewer changes.* In this mode, DRBD will check which of the two nodes has recorded fewer modifications, and will then discard _all_ modifications made on that host. * *Graceful recovery from split brain if one host has had no intermediate changes.* In this mode, if one of the hosts has made no modifications at all during split brain, DRBD will simply recover gracefully and declare the split brain resolved. Note that this is a fairly unlikely scenario. Even if both hosts only mounted the file system on the DRBD block device (even read-only), the device contents typically would be modified (e.g. by filesystem journal replay), ruling out the possibility of automatic recovery. Whether or not automatic split brain recovery is acceptable depends largely on the individual application. Consider the example of DRBD hosting a database. The discard modifications from host with fewer changes approach may be fine for a web application click-through database. By contrast, it may be totally unacceptable to automatically discard _any_ modifications made to a financial database, requiring manual recovery in any split brain event. Consider your application's requirements carefully before enabling automatic split brain recovery. Refer to <> for details on configuring DRBD's automatic split brain recovery policies. [[s-disk-flush-support]] === Support for disk flushes When local block devices such as hard drives or RAID logical disks have write caching enabled, writes to these devices are considered completed as soon as they have reached the volatile cache. Controller manufacturers typically refer to this as write-back mode, the opposite being write-through. If a power outage occurs on a controller in write-back mode, the last writes are never committed to the disk, potentially causing data loss. To counteract this, DRBD makes use of disk flushes. A disk flush is a write operation that completes only when the associated data has been committed to stable (non-volatile) storage -- that is to say, it has effectively been written to disk, rather than to the cache. DRBD uses disk flushes for write operations both to its replicated data set and to its meta data. In effect, DRBD circumvents the write cache in situations it deems necessary, as in <> updates or enforcement of implicit write-after-write dependencies. This means additional reliability even in the face of power failure. It is important to understand that DRBD can use disk flushes only when layered on top of backing devices that support them. Most reasonably recent kernels support disk flushes for most SCSI and SATA devices. Linux software RAID (md) supports disk flushes for RAID-1 provided that all component devices support them too. The same is true for device-mapper devices (LVM2, dm-raid, multipath). Controllers with battery-backed write cache (BBWC) use a battery to back up their volatile storage. On such devices, when power is restored after an outage, the controller flushes all pending writes out to disk from the battery-backed cache, ensuring that all writes committed to the volatile cache are actually transferred to stable storage. When running DRBD on top of such devices, it may be acceptable to disable disk flushes, thereby improving DRBD's write performance. See <> for details. [[s-trim-discard]] === Trim/Discard support _Trim_/_Discard_ are two names for the same feature: a request to a storage system, telling it that some data range is not being used anymorefootnote:[For example, a deleted file's data.] and can get recycled. + This call originates in Flash-based storages (SSDs, FusionIO cards, etc.), which cannot easily _rewrite_ a sector but instead have to _erase_ and write the (new) data again (incurring some latency cost). For more details, see e.g. the https://en.wikipedia.org/wiki/Trim_%28computing%29[wikipedia page]. Since 8.4.3 DRBD includes support for _Trim_/_Discard_. You don't need to configure or enable anything; if DRBD detects that the local (underlying) storage system allows using these commands, it will transparently enable them and pass such requests through. The effect is that e.g. a recent-enough `mkfs.ext4` on a multi-TB volume can shorten the initial sync time to a few seconds to minutes - just by telling DRBD (which will relay that information to all connected nodes) that most/all of the storage is now to be seen as invalidated. Nodes that connect to that resource later on will not have seen the _Trim_/_Discard_ requests, and will therefore start a full resync; depending on kernel version and file system a call to `fstrim` might give the wanted result, though. NOTE: even if you don't have storage with _Trim_/_Discard_ support, some virtual block devices will provide you with the same feature, for example Thin LVM. [[s-handling-disk-errors]] === Disk error handling strategies If a hard drive fails which is used as a backing block device for DRBD on one of the nodes, DRBD may either pass on the I/O error to the upper layer (usually the file system) or it can mask I/O errors from upper layers. [[fp-io-error-pass-on]] .Passing on I/O errors If DRBD is configured to pass on I/O errors, any such errors occurring on the lower-level device are transparently passed to upper I/O layers. Thus, it is left to upper layers to deal with such errors (this may result in a file system being remounted read-only, for example). This strategy does not ensure service continuity, and is hence not recommended for most users. [[fp-io-error-detach]] .Masking I/O errors If DRBD is configured to _detach_ on lower-level I/O error, DRBD will do so, automatically, upon occurrence of the first lower-level I/O error. The I/O error is masked from upper layers while DRBD transparently fetches the affected block from a peer node, over the network. From then onwards, DRBD is said to operate in diskless mode, and carries out all subsequent I/O operations, read and write, on the peer node(s) only. Performance in this mode will be reduced, but the service continues without interruption, and can be moved to the peer node in a deliberate fashion at a convenient time. See <> for information on configuring I/O error handling strategies for DRBD. [[s-outdate]] === Strategies for dealing with outdated data DRBD distinguishes between _inconsistent_ and _outdated_ data. Inconsistent data is data that cannot be expected to be accessible and useful in any manner. The prime example for this is data on a node that is currently the target of an on-going synchronization. Data on such a node is part obsolete, part up to date, and impossible to identify as either. Thus, for example, if the device holds a filesystem (as is commonly the case), that filesystem would be unexpected to mount or even pass an automatic filesystem check. Outdated data, by contrast, is data on a secondary node that is consistent, but no longer in sync with the primary node. This would occur in any interruption of the replication link, whether temporary or permanent. Data on an outdated, disconnected secondary node is expected to be clean, but it reflects a state of the peer node some time past. In order to avoid services using outdated data, DRBD disallows <> that is in the outdated state. DRBD has interfaces that allow an external application to outdate a secondary node as soon as a network interruption occurs. DRBD will then refuse to switch the node to the primary role, preventing applications from using the outdated data. ifndef::drbd-only[] A complete implementation of this functionality exists for the <> (where it uses a communication channel separate from the DRBD replication link). However, the interfaces are generic and may be easily used by any other cluster management application. endif::drbd-only[] Whenever an outdated resource has its replication link re-established, its outdated flag is automatically cleared. A <> then follows. [[s-three-way-repl]] === Three-way replication via stacking NOTE: Available in DRBD version 8.3.0 and above; deprecated in DRBD version 9.x, as more nodes can be implemented on a single level. See <> for details. When using three-way replication, DRBD adds a third node to an existing 2-node cluster and replicates data to that node, where it can be used for backup and disaster recovery purposes. ifndef::drbd-only,de-brand[] This type of configuration generally involves <>. endif::[] Three-way replication works by adding another, _stacked_ DRBD resource on top of the existing resource holding your production data, as seen in this illustration: .DRBD resource stacking image::images/drbd-resource-stacking.svg[] The stacked resource is replicated using asynchronous replication (DRBD protocol A), whereas the production data would usually make use of synchronous replication (DRBD protocol C). Three-way replication can be used permanently, where the third node is continuously updated with data from the production cluster. Alternatively, it may also be employed on demand, where the production cluster is normally disconnected from the backup site, and site-to-site synchronization is performed on a regular basis, for example by running a nightly cron job. ifndef::drbd-only,de-brand[] [[s-drbd-proxy]] === Long-distance replication via DRBD Proxy DRBD's <> is asynchronous, but the writing application will block as soon as the socket output buffer is full (see the `sndbuf-size` option in the man page of `drbd.conf`). In that event, the writing application has to wait until some of the data written runs off through a possibly small bandwidth network link. The average write bandwidth is limited by available bandwidth of the network link. Write bursts can only be handled gracefully if they fit into the limited socket output buffer. You can mitigate this by DRBD Proxy's buffering mechanism. DRBD Proxy will place changed data from the DRBD device on the primary node into its buffers. DRBD Proxy's buffer size is freely configurable, only limited by the address room size and available physical RAM. Optionally DRBD Proxy can be configured to compress and decompress the data it forwards. Compression and decompression of DRBD's data packets might slightly increase latency. However, when the bandwidth of the network link is the limiting factor, the gain in shortening transmit time outweighs the compression and decompression overhead. Compression and decompression were implemented with multi core SMP systems in mind, and can utilize multiple CPU cores. The fact that most block I/O data compresses very well and therefore the effective bandwidth increases justifies the use of the DRBD Proxy even with DRBD protocols B and C. See <> for information on configuring DRBD Proxy. NOTE: DRBD Proxy is one of the few parts of the DRBD product family that is not published under an open source license. Please contact sales@linbit.com or sales_us@linbit.com for an evaluation license. endif::[] [[s-truck-based-replication]] === Truck based replication Truck based replication, also known as disk shipping, is a means of preseeding a remote site with data to be replicated, by physically shipping storage media to the remote site. This is particularly suited for situations where * the total amount of data to be replicated is fairly large (more than a few hundreds of gigabytes); * the expected rate of change of the data to be replicated is less than enormous; * the available network bandwidth between sites is limited. In such situations, without truck based replication, DRBD would require a very long initial device synchronization (on the order of weeks, months, or years). Truck based replication allows to ship a data seed to the remote site, and so drastically reduces the initial synchronization time. See <> for details on this use case. [[s-floating-peers]] === Floating peers NOTE: This feature is available in DRBD versions 8.3.2 and above. A somewhat special use case for DRBD is the _floating peers_ configuration. In floating peer setups, DRBD peers are not tied to specific named hosts (as in conventional configurations), but instead have the ability to float between several hosts. In such a configuration, DRBD identifies peers by IP address, rather than by host name. ifndef::drbd-only[] For more information about managing floating peer configurations, see <>. endif::drbd-only[] [[s-rebalance]] === Data rebalancing (horizontal storage scaling) If your company's policy says that 3-way redundancy is needed, you need at least 3 servers for your setup. Now, as your storage demands grow, you will encounter the need for additional servers. Rather than having to buy 3 more servers at the same time, you can _rebalance_ your data across a single additional node. .DRBD data rebalancing image::images/rebalance.svg[] //[scaledwidth="75"] In the figure above you can see the _before_ and _after_ states: from 3 nodes with three 25TiB volumes each (for a net 75TiB), to 4 nodes, with net 100TiB. DRBD 9 makes it possible to do an online, live migration of the data; please see <> for the exact steps needed. [[s-drbd-client]] === DRBD client With the multiple-peer feature of DRBD a number of interesting use-cases have been added, for example the _DRBD client_. The basic idea is that the DRBD _backend_ can consist of 3, 4, or more nodes (depending on the policy of required redundancy); but, as DRBD 9 can connect more nodes than that. DRBD works then as a storage access protocol in addition to storage replication. All write requests executed on a primary _DRBD client_ gets shipped to all nodes equipped with storage. Read requests are only shipped to one of the server nodes. The _DRBD client_ will evenly distribute the read requests among all available server nodes. See <> for more information. [[s-feature-quorum]] === Quorum In order to avoid split brain or diverging data of replicas one has to configure fencing. It turns out that in real world deployments node fencing is not popular because often mistakes happen in planning or deploying it. In the moment a data-set has 3 replicas one can rely on the quorum implementation within DRBD instead of cluster manager level fencing. ifndef::drbd-only[] Pacemaker gets informed about quorum or loss-of-quorum via the master score of the resource. endif::drbd-only[] DRBD's quorum can be used with any kind of Linux based service. In case the service terminates in the moment it is exposed to an IO-error the on quorum loss behavior is very elegant. In case the service does not terminate upon IO-error the systems needs to be configured to reboot a primary node that looses quorum. See <> for more information. [[s-feature-quorum-tiebreaker]] ==== Tiebreaker [NOTE] ==================== The quorum tiebreaker feature is available in DRBD versions 9.0.18 and above. ==================== The fundamental problem with two node clusters is that in the moment they lose connectivity we have two partitions and none of them has quorum, which results in the cluster halting the service. This problem can be mitigated by adding a third, diskless node to the cluster which will then act as a quorum tiebreaker. See <> for more information. ifndef::drbd-only[] [[s-feature-VCS]] === DRBD integration for VCS Veritas Cluster Server (or Veritas Infoscale Availability) is a commercial alternative to the Pacemaker open source software. In case you need to integrate DRBD resources into a VCS setup please see the README in https://github.com/LINBIT/drbd-utils/tree/master/scripts/VCS[drbd-utils/scripts/VCS] on github. endif::drbd-only[]