| 1 |
// FIXME |
| 2 |
[[ch-pacemaker]] |
| 3 |
== Integrating DRBD with Pacemaker clusters |
| 4 |
|
| 5 |
indexterm:[Pacemaker]Using DRBD in conjunction with the Pacemaker |
| 6 |
cluster stack is arguably DRBD's most frequently found use |
| 7 |
case. Pacemaker is also one of the applications that make DRBD |
| 8 |
extremely powerful in a wide variety of usage scenarios. |
| 9 |
|
| 10 |
DRBD can be used in Pacemaker clusters in different ways: |
| 11 |
|
| 12 |
* DRBD running as a background-service, used like a SAN; or |
| 13 |
* DRBD completely managed by Pacemaker via the DRBD OCF resource agent |
| 14 |
|
| 15 |
Both have a few advantages and disadvantages, these will be discussed below. |
| 16 |
|
| 17 |
NOTE: It's recommended to have either fencing configured or quorum enabled. |
| 18 |
(But not both. External fencing handler results may interact in conflicting ways |
| 19 |
with DRBD internal quorum.) |
| 20 |
If your cluster has communication issues (e.g. network switch loses |
| 21 |
power) and gets split, the parts might start the services (failover) |
| 22 |
and cause a *Split-Brain* when the communication resumes again. |
| 23 |
|
| 24 |
|
| 25 |
|
| 26 |
[[s-pacemaker-primer]] |
| 27 |
=== Pacemaker primer |
| 28 |
|
| 29 |
Pacemaker is a sophisticated, feature-rich, and widely deployed |
| 30 |
cluster resource manager for the Linux platform. It comes with a rich |
| 31 |
set of documentation. In order to understand this chapter, reading the |
| 32 |
following documents is highly recommended: |
| 33 |
|
| 34 |
* http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf[Clusters |
| 35 |
From Scratch], a step-by-step guide to configuring high availability |
| 36 |
clusters; |
| 37 |
* http://crmsh.github.io/documentation/index.html[CRM CLI (command line |
| 38 |
interface) tool], a manual for the CRM shell, a simple and intuitive |
| 39 |
command line interface bundled with Pacemaker; |
| 40 |
* http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/[Pacemaker |
| 41 |
Configuration Explained], a reference document explaining the |
| 42 |
concept and design behind Pacemaker. |
| 43 |
|
| 44 |
|
| 45 |
[[s-pacemaker-drbd-background]] |
| 46 |
=== Using DRBD as a background service in a pacemaker cluster |
| 47 |
|
| 48 |
In this section you will see that using autonomous DRBD storage can look |
| 49 |
like local storage; so integrating in a Pacemaker cluster is done by |
| 50 |
pointing your mount points at DRBD. |
| 51 |
|
| 52 |
First of all, we will use the `auto-promote` feature of DRBD, so that |
| 53 |
DRBD automatically sets itself _Primary_ when needed. This will probably |
| 54 |
apply to all of your resources, so setting that a default in the |
| 55 |
`common` section makes sense: |
| 56 |
|
| 57 |
[source,drbd] |
| 58 |
---------------------------- |
| 59 |
common { |
| 60 |
options { |
| 61 |
auto-promote yes; |
| 62 |
... |
| 63 |
} |
| 64 |
} |
| 65 |
---------------------------- |
| 66 |
|
| 67 |
Now you just need to use your storage, e.g. via a filesystem: |
| 68 |
|
| 69 |
.Pacemaker configuration for DRBD-backed MySQL service, using `auto-promote` |
| 70 |
---------------------------- |
| 71 |
crm configure |
| 72 |
crm(live)configure# primitive fs_mysql ocf:heartbeat:Filesystem \ |
| 73 |
params device="/dev/drbd/by-res/mysql/0" \ |
| 74 |
directory="/var/lib/mysql" fstype="ext3" |
| 75 |
crm(live)configure# primitive ip_mysql ocf:heartbeat:IPaddr2 \ |
| 76 |
params ip="10.9.42.1" nic="eth0" |
| 77 |
crm(live)configure# primitive mysqld lsb:mysqld |
| 78 |
crm(live)configure# group mysql fs_mysql ip_mysql mysqld |
| 79 |
crm(live)configure# commit |
| 80 |
crm(live)configure# exit |
| 81 |
bye |
| 82 |
---------------------------- |
| 83 |
|
| 84 |
Essentially all that is needed is a mountpoint (`/var/lib/mysql` in this |
| 85 |
example) where the DRBD resource gets mounted. |
| 86 |
|
| 87 |
As long as Pacemaker has control, it will only allow a single instance |
| 88 |
of that mount across your cluster. |
| 89 |
|
| 90 |
See also <<s-pacemaker-drbd-attr>> for additional information on ordering |
| 91 |
constraints for system startup and more. |
| 92 |
|
| 93 |
[[s-pacemaker-crm-drbd-backed-service]] |
| 94 |
=== Adding a DRBD-backed service to the cluster configuration, including a master-slave resource |
| 95 |
|
| 96 |
This section explains how to enable a DRBD-backed service in a |
| 97 |
Pacemaker cluster. |
| 98 |
|
| 99 |
NOTE: If you are employing the DRBD OCF resource agent, it is |
| 100 |
recommended that you defer DRBD startup, shutdown, promotion, and |
| 101 |
demotion _exclusively_ to the OCF resource agent. That means that you |
| 102 |
should disable the DRBD init script: |
| 103 |
|
| 104 |
---------------------------- |
| 105 |
chkconfig drbd off |
| 106 |
---------------------------- |
| 107 |
|
| 108 |
The `ocf:linbit:drbd` OCF resource agent provides Master/Slave |
| 109 |
capability, allowing Pacemaker to start and monitor the DRBD resource |
| 110 |
on multiple nodes and promoting and demoting as needed. You must, |
| 111 |
however, understand that the DRBD RA disconnects and detaches all |
| 112 |
DRBD resources it manages on Pacemaker shutdown, and also upon |
| 113 |
enabling standby mode for a node. |
| 114 |
|
| 115 |
|
| 116 |
IMPORTANT: The OCF resource agent which ships with DRBD belongs to the |
| 117 |
`linbit` provider, and hence installs as |
| 118 |
`/usr/lib/ocf/resource.d/linbit/drbd`. There is a legacy resource |
| 119 |
agent that ships as part of the OCF resource agents package, which |
| 120 |
uses the `heartbeat` provider and installs into |
| 121 |
`/usr/lib/ocf/resource.d/heartbeat/drbd`. The legacy OCF RA is |
| 122 |
deprecated and should no longer be used. |
| 123 |
|
| 124 |
In order to enable a DRBD-backed configuration for a MySQL database in |
| 125 |
a Pacemaker CRM cluster with the `drbd` OCF resource agent, you must |
| 126 |
create both the necessary resources, and Pacemaker constraints to |
| 127 |
ensure your service only starts on a previously promoted DRBD |
| 128 |
resource. You may do so using the `crm` shell, as outlined in the |
| 129 |
following example: |
| 130 |
|
| 131 |
.Pacemaker configuration for DRBD-backed MySQL service, using a `master-slave` resource |
| 132 |
---------------------------- |
| 133 |
crm configure |
| 134 |
crm(live)configure# primitive drbd_mysql ocf:linbit:drbd \ |
| 135 |
params drbd_resource="mysql" \ |
| 136 |
op monitor interval="29s" role="Master" \ |
| 137 |
op monitor interval="31s" role="Slave" |
| 138 |
crm(live)configure# ms ms_drbd_mysql drbd_mysql \ |
| 139 |
meta master-max="1" master-node-max="1" \ |
| 140 |
clone-max="2" clone-node-max="1" \ |
| 141 |
notify="true" |
| 142 |
crm(live)configure# primitive fs_mysql ocf:heartbeat:Filesystem \ |
| 143 |
params device="/dev/drbd/by-res/mysql/0" \ |
| 144 |
directory="/var/lib/mysql" fstype="ext3" |
| 145 |
crm(live)configure# primitive ip_mysql ocf:heartbeat:IPaddr2 \ |
| 146 |
params ip="10.9.42.1" nic="eth0" |
| 147 |
crm(live)configure# primitive mysqld lsb:mysqld |
| 148 |
crm(live)configure# group mysql fs_mysql ip_mysql mysqld |
| 149 |
crm(live)configure# colocation mysql_on_drbd \ |
| 150 |
inf: mysql ms_drbd_mysql:Master |
| 151 |
crm(live)configure# order mysql_after_drbd \ |
| 152 |
inf: ms_drbd_mysql:promote mysql:start |
| 153 |
crm(live)configure# commit |
| 154 |
crm(live)configure# exit |
| 155 |
bye |
| 156 |
---------------------------- |
| 157 |
|
| 158 |
After this, your configuration should be enabled. Pacemaker now |
| 159 |
selects a node on which it promotes the DRBD resource, and then starts |
| 160 |
the DRBD-backed resource group on that same node. |
| 161 |
|
| 162 |
See also <<s-pacemaker-drbd-attr>> for additional information on |
| 163 |
location contraints for placing the Master role. |
| 164 |
|
| 165 |
[[s-pacemaker-fencing]] |
| 166 |
=== Using resource-level fencing in Pacemaker clusters |
| 167 |
|
| 168 |
This section outlines the steps necessary to prevent Pacemaker from |
| 169 |
promoting a DRBD Master/Slave resource when its DRBD replication link |
| 170 |
has been interrupted. This keeps Pacemaker from starting a service |
| 171 |
with outdated data and causing an unwanted "time warp" in the |
| 172 |
process. |
| 173 |
|
| 174 |
In order to enable any resource-level fencing for DRBD, you must add |
| 175 |
the following lines to your resource configuration: |
| 176 |
|
| 177 |
[source,drbd] |
| 178 |
---------------------------- |
| 179 |
resource <resource> { |
| 180 |
net { |
| 181 |
fencing resource-only; |
| 182 |
... |
| 183 |
} |
| 184 |
} |
| 185 |
---------------------------- |
| 186 |
|
| 187 |
You will also have to make changes to the `handlers` section depending |
| 188 |
on the cluster infrastructure being used. |
| 189 |
|
| 190 |
Corosync-based Pacemaker clusters can use the |
| 191 |
functionality explained in <<s-pacemaker-fencing-cib>>. |
| 192 |
|
| 193 |
IMPORTANT: It is absolutely vital to configure at least two independent cluster |
| 194 |
communications channels for this functionality to work correctly. Corosync |
| 195 |
clusters should list at least two redundant rings in `corosync.conf`, |
| 196 |
respectively several paths for knet. |
| 197 |
|
| 198 |
|
| 199 |
[[s-pacemaker-fencing-cib]] |
| 200 |
==== Resource-level fencing using the Cluster Information Base (CIB) |
| 201 |
|
| 202 |
In order to enable resource-level fencing for Pacemaker, you will have |
| 203 |
to set two options in `drbd.conf`: |
| 204 |
|
| 205 |
[source,drbd] |
| 206 |
---------------------------- |
| 207 |
resource <resource> { |
| 208 |
net { |
| 209 |
fencing resource-only; |
| 210 |
... |
| 211 |
} |
| 212 |
handlers { |
| 213 |
fence-peer "/usr/lib/drbd/crm-fence-peer.9.sh"; |
| 214 |
unfence-peer "/usr/lib/drbd/crm-unfence-peer.9.sh"; |
| 215 |
# Note: we used to abuse the after-resync-target handler to do the |
| 216 |
# unfence, but since 2016 have a dedicated unfence-peer handler. |
| 217 |
# Using the after-resync-target handler is wrong in some corner cases. |
| 218 |
... |
| 219 |
} |
| 220 |
... |
| 221 |
} |
| 222 |
---------------------------- |
| 223 |
|
| 224 |
Thus, if the DRBD replication link becomes disconnected, the |
| 225 |
`crm-fence-peer.9.sh` script contacts the cluster manager, determines the |
| 226 |
Pacemaker Master/Slave resource associated with this DRBD resource, |
| 227 |
and ensures that the Master/Slave resource no longer gets promoted on |
| 228 |
any node other than the currently active one. Conversely, when the |
| 229 |
connection is re-established and DRBD completes its synchronization |
| 230 |
process, then that constraint is removed and the cluster manager is |
| 231 |
free to promote the resource on any node again. |
| 232 |
|
| 233 |
[[s-pacemaker-stacked-resources]] |
| 234 |
=== Using stacked DRBD resources in Pacemaker clusters |
| 235 |
|
| 236 |
NOTE: Stacking is deprecated in DRBD version |
| 237 |
9.x, as more nodes can be implemented on a single level. See |
| 238 |
<<s-drbdconf-conns>> for details. |
| 239 |
|
| 240 |
Stacked resources allow DRBD to be used for multi-level redundancy in |
| 241 |
multiple-node clusters, or to establish off-site disaster recovery |
| 242 |
capability. This section describes how to configure DRBD and Pacemaker |
| 243 |
in such configurations. |
| 244 |
|
| 245 |
|
| 246 |
[[s-pacemaker-stacked-dr]] |
| 247 |
==== Adding off-site disaster recovery to Pacemaker clusters |
| 248 |
|
| 249 |
In this configuration scenario, we would deal with a two-node high |
| 250 |
availability cluster in one site, plus a separate node which would |
| 251 |
presumably be housed off-site. The third node acts as a disaster |
| 252 |
recovery node and is a standalone server. Consider the following |
| 253 |
illustration to describe the concept. |
| 254 |
|
| 255 |
.DRBD resource stacking in Pacemaker clusters |
| 256 |
image::images/drbd-resource-stacking-pacemaker-3nodes.svg[] |
| 257 |
|
| 258 |
In this example, 'alice' and 'bob' form a two-node Pacemaker cluster, |
| 259 |
whereas 'charlie' is an off-site node not managed by Pacemaker. |
| 260 |
|
| 261 |
To create such a configuration, you would first configure and |
| 262 |
initialize DRBD resources as described in <<s-three-nodes>>. Then, |
| 263 |
configure Pacemaker with the following CRM configuration: |
| 264 |
|
| 265 |
[source,drbd] |
| 266 |
---------------------------- |
| 267 |
primitive p_drbd_r0 ocf:linbit:drbd \ |
| 268 |
params drbd_resource="r0" |
| 269 |
|
| 270 |
primitive p_drbd_r0-U ocf:linbit:drbd \ |
| 271 |
params drbd_resource="r0-U" |
| 272 |
|
| 273 |
primitive p_ip_stacked ocf:heartbeat:IPaddr2 \ |
| 274 |
params ip="192.168.42.1" nic="eth0" |
| 275 |
|
| 276 |
ms ms_drbd_r0 p_drbd_r0 \ |
| 277 |
meta master-max="1" master-node-max="1" \ |
| 278 |
clone-max="2" clone-node-max="1" \ |
| 279 |
notify="true" globally-unique="false" |
| 280 |
|
| 281 |
ms ms_drbd_r0-U p_drbd_r0-U \ |
| 282 |
meta master-max="1" clone-max="1" \ |
| 283 |
clone-node-max="1" master-node-max="1" \ |
| 284 |
notify="true" globally-unique="false" |
| 285 |
|
| 286 |
colocation c_drbd_r0-U_on_drbd_r0 \ |
| 287 |
inf: ms_drbd_r0-U ms_drbd_r0:Master |
| 288 |
|
| 289 |
colocation c_drbd_r0-U_on_ip \ |
| 290 |
inf: ms_drbd_r0-U p_ip_stacked |
| 291 |
|
| 292 |
colocation c_ip_on_r0_master \ |
| 293 |
inf: p_ip_stacked ms_drbd_r0:Master |
| 294 |
|
| 295 |
order o_ip_before_r0-U \ |
| 296 |
inf: p_ip_stacked ms_drbd_r0-U:start |
| 297 |
|
| 298 |
order o_drbd_r0_before_r0-U \ |
| 299 |
inf: ms_drbd_r0:promote ms_drbd_r0-U:start |
| 300 |
---------------------------- |
| 301 |
|
| 302 |
Assuming you created this configuration in a temporary file named |
| 303 |
`/tmp/crm.txt`, you may import it into the live cluster configuration |
| 304 |
with the following command: |
| 305 |
|
| 306 |
---------------------------- |
| 307 |
crm configure < /tmp/crm.txt |
| 308 |
---------------------------- |
| 309 |
|
| 310 |
This configuration will ensure that the following actions occur in the |
| 311 |
correct order on the 'alice'/'bob' cluster: |
| 312 |
|
| 313 |
. Pacemaker starts the DRBD resource `r0` on both cluster nodes, and |
| 314 |
promotes one node to the Master (DRBD Primary) role. |
| 315 |
|
| 316 |
. Pacemaker then starts the IP address 192.168.42.1, which the stacked |
| 317 |
resource is to use for replication to the third node. It does so on |
| 318 |
the node it has previously promoted to the Master role for `r0` DRBD |
| 319 |
resource. |
| 320 |
|
| 321 |
. On the node which now has the Primary role for `r0` and also the |
| 322 |
replication IP address for `r0-U`, Pacemaker now starts the |
| 323 |
`r0-U` DRBD resource, which connects and replicates to the off-site |
| 324 |
node. |
| 325 |
|
| 326 |
. Pacemaker then promotes the `r0-U` resource to the Primary role too, |
| 327 |
so it can be used by an application. |
| 328 |
|
| 329 |
Thus, this Pacemaker configuration ensures that there is not only full |
| 330 |
data redundancy between cluster nodes, but also to the third, off-site |
| 331 |
node. |
| 332 |
|
| 333 |
NOTE: This type of setup is usually deployed together with |
| 334 |
<<s-drbd-proxy,DRBD Proxy>>. |
| 335 |
|
| 336 |
[[s-pacemaker-stacked-4way]] |
| 337 |
==== Using stacked resources to achieve 4-way redundancy in Pacemaker clusters |
| 338 |
|
| 339 |
In this configuration, a total of three DRBD resources (two unstacked, |
| 340 |
one stacked) are used to achieve 4-way storage redundancy. This means |
| 341 |
that of a 4-node cluster, up to three nodes can fail while still |
| 342 |
providing service availability. |
| 343 |
|
| 344 |
Consider the following illustration to explain the concept. |
| 345 |
|
| 346 |
.DRBD resource stacking in Pacemaker clusters |
| 347 |
image::images/drbd-resource-stacking-pacemaker-4nodes.svg[] |
| 348 |
|
| 349 |
In this example, 'alice', 'bob', 'charlie', and 'daisy' form two |
| 350 |
two-node Pacemaker clusters. 'alice' and 'bob' form the cluster named |
| 351 |
`left` and replicate data using a DRBD resource between them, while |
| 352 |
'charlie' and 'daisy' do the same with a separate DRBD resource, in a |
| 353 |
cluster named `right`. A third, stacked DRBD resource connects the two |
| 354 |
clusters. |
| 355 |
|
| 356 |
NOTE: Due to limitations in the Pacemaker cluster manager as of |
| 357 |
Pacemaker version 1.0.5, it is not possible to create this setup in a |
| 358 |
single four-node cluster without disabling CIB validation, which is an |
| 359 |
advanced process not recommended for general-purpose use. It is |
| 360 |
anticipated that this is being addressed in future Pacemaker releases. |
| 361 |
|
| 362 |
To create such a configuration, you would first configure and |
| 363 |
initialize DRBD resources as described in <<s-three-nodes>> (except |
| 364 |
that the remote half of the DRBD configuration is also stacked, not |
| 365 |
just the local cluster). Then, configure Pacemaker with the following |
| 366 |
CRM configuration, starting with the cluster `left`: |
| 367 |
|
| 368 |
[source,drbd] |
| 369 |
---------------------------- |
| 370 |
primitive p_drbd_left ocf:linbit:drbd \ |
| 371 |
params drbd_resource="left" |
| 372 |
|
| 373 |
primitive p_drbd_stacked ocf:linbit:drbd \ |
| 374 |
params drbd_resource="stacked" |
| 375 |
|
| 376 |
primitive p_ip_stacked_left ocf:heartbeat:IPaddr2 \ |
| 377 |
params ip="10.9.9.100" nic="eth0" |
| 378 |
|
| 379 |
ms ms_drbd_left p_drbd_left \ |
| 380 |
meta master-max="1" master-node-max="1" \ |
| 381 |
clone-max="2" clone-node-max="1" \ |
| 382 |
notify="true" |
| 383 |
|
| 384 |
ms ms_drbd_stacked p_drbd_stacked \ |
| 385 |
meta master-max="1" clone-max="1" \ |
| 386 |
clone-node-max="1" master-node-max="1" \ |
| 387 |
notify="true" target-role="Master" |
| 388 |
|
| 389 |
colocation c_ip_on_left_master \ |
| 390 |
inf: p_ip_stacked_left ms_drbd_left:Master |
| 391 |
|
| 392 |
colocation c_drbd_stacked_on_ip_left \ |
| 393 |
inf: ms_drbd_stacked p_ip_stacked_left |
| 394 |
|
| 395 |
order o_ip_before_stacked_left \ |
| 396 |
inf: p_ip_stacked_left ms_drbd_stacked:start |
| 397 |
|
| 398 |
order o_drbd_left_before_stacked_left \ |
| 399 |
inf: ms_drbd_left:promote ms_drbd_stacked:start |
| 400 |
|
| 401 |
---------------------------- |
| 402 |
|
| 403 |
Assuming you created this configuration in a temporary file named |
| 404 |
`/tmp/crm.txt`, you may import it into the live cluster configuration |
| 405 |
with the following command: |
| 406 |
|
| 407 |
---------------------------- |
| 408 |
crm configure < /tmp/crm.txt |
| 409 |
---------------------------- |
| 410 |
|
| 411 |
After adding this configuration to the CIB, Pacemaker will execute the |
| 412 |
following actions: |
| 413 |
|
| 414 |
. Bring up the DRBD resource `left` replicating between 'alice' and |
| 415 |
'bob' promoting the resource to the Master role on one of these nodes. |
| 416 |
|
| 417 |
. Bring up the IP address 10.9.9.100 (on either 'alice' or 'bob', |
| 418 |
depending on which of these holds the Master role for the resource |
| 419 |
`left`). |
| 420 |
|
| 421 |
. Bring up the DRBD resource `stacked` on the same node that holds the |
| 422 |
just-configured IP address. |
| 423 |
|
| 424 |
. Promote the stacked DRBD resource to the Primary role. |
| 425 |
|
| 426 |
Now, proceed on the cluster `right` by creating the following |
| 427 |
configuration: |
| 428 |
|
| 429 |
[source,drbd] |
| 430 |
---------------------------- |
| 431 |
primitive p_drbd_right ocf:linbit:drbd \ |
| 432 |
params drbd_resource="right" |
| 433 |
|
| 434 |
primitive p_drbd_stacked ocf:linbit:drbd \ |
| 435 |
params drbd_resource="stacked" |
| 436 |
|
| 437 |
primitive p_ip_stacked_right ocf:heartbeat:IPaddr2 \ |
| 438 |
params ip="10.9.10.101" nic="eth0" |
| 439 |
|
| 440 |
ms ms_drbd_right p_drbd_right \ |
| 441 |
meta master-max="1" master-node-max="1" \ |
| 442 |
clone-max="2" clone-node-max="1" \ |
| 443 |
notify="true" |
| 444 |
|
| 445 |
ms ms_drbd_stacked p_drbd_stacked \ |
| 446 |
meta master-max="1" clone-max="1" \ |
| 447 |
clone-node-max="1" master-node-max="1" \ |
| 448 |
notify="true" target-role="Slave" |
| 449 |
|
| 450 |
colocation c_drbd_stacked_on_ip_right \ |
| 451 |
inf: ms_drbd_stacked p_ip_stacked_right |
| 452 |
|
| 453 |
colocation c_ip_on_right_master \ |
| 454 |
inf: p_ip_stacked_right ms_drbd_right:Master |
| 455 |
|
| 456 |
order o_ip_before_stacked_right \ |
| 457 |
inf: p_ip_stacked_right ms_drbd_stacked:start |
| 458 |
|
| 459 |
order o_drbd_right_before_stacked_right \ |
| 460 |
inf: ms_drbd_right:promote ms_drbd_stacked:start |
| 461 |
---------------------------- |
| 462 |
|
| 463 |
After adding this configuration to the CIB, Pacemaker will execute the |
| 464 |
following actions: |
| 465 |
|
| 466 |
. Bring up the DRBD resource `right` replicating between 'charlie' and |
| 467 |
'daisy', promoting the resource to the Master role on one of these |
| 468 |
nodes. |
| 469 |
|
| 470 |
. Bring up the IP address 10.9.10.101 (on either 'charlie' or 'daisy', |
| 471 |
depending on which of these holds the Master role for the resource |
| 472 |
`right`). |
| 473 |
|
| 474 |
. Bring up the DRBD resource `stacked` on the same node that holds the |
| 475 |
just-configured IP address. |
| 476 |
|
| 477 |
. Leave the stacked DRBD resource in the Secondary role (due to |
| 478 |
`target-role="Slave"`). |
| 479 |
|
| 480 |
|
| 481 |
[[s-pacemaker-floating-peers]] |
| 482 |
=== Configuring DRBD to replicate between two SAN-backed Pacemaker clusters |
| 483 |
|
| 484 |
indexterm:[IP address, floating peers]This is a somewhat advanced setup usually employed in split-site |
| 485 |
configurations. It involves two separate Pacemaker clusters, where |
| 486 |
each cluster has access to a separate Storage Area Network (SAN). DRBD |
| 487 |
is then used to replicate data stored on that SAN, across an IP link |
| 488 |
between sites. |
| 489 |
|
| 490 |
Consider the following illustration to describe the concept. |
| 491 |
|
| 492 |
.Using DRBD to replicate between SAN-based clusters |
| 493 |
image::images/drbd-pacemaker-floating-peers.svg[] |
| 494 |
|
| 495 |
Which of the individual nodes in each site currently acts as the DRBD |
| 496 |
peer is not explicitly defined -- the DRBD peers |
| 497 |
<<s-floating-peers,are said to _float_>>; that is, DRBD binds to |
| 498 |
virtual IP addresses not tied to a specific physical machine. |
| 499 |
|
| 500 |
|
| 501 |
NOTE: This type of setup is usually deployed together with |
| 502 |
<<s-drbd-proxy,DRBD Proxy>> and/or <<s-truck-based-replication,truck |
| 503 |
based replication>>. |
| 504 |
|
| 505 |
Since this type of setup deals with shared storage, configuring and |
| 506 |
testing STONITH is absolutely vital for it to work properly. |
| 507 |
|
| 508 |
|
| 509 |
[[s-pacemaker-floating-peers-drbd-config]] |
| 510 |
==== DRBD resource configuration |
| 511 |
|
| 512 |
To enable your DRBD resource to float, configure it in `drbd.conf` in |
| 513 |
the following fashion: |
| 514 |
|
| 515 |
[source,drbd] |
| 516 |
---------------------------- |
| 517 |
resource <resource> { |
| 518 |
... |
| 519 |
device /dev/drbd0; |
| 520 |
disk /dev/sda1; |
| 521 |
meta-disk internal; |
| 522 |
floating 10.9.9.100:7788; |
| 523 |
floating 10.9.10.101:7788; |
| 524 |
} |
| 525 |
---------------------------- |
| 526 |
|
| 527 |
The `floating` keyword replaces the `on <host>` sections normally |
| 528 |
found in the resource configuration. In this mode, DRBD identifies |
| 529 |
peers by IP address and TCP port, rather than by host name. It is |
| 530 |
important to note that the addresses specified must be virtual cluster |
| 531 |
IP addresses, rather than physical node IP addresses, for floating to |
| 532 |
function properly. As shown in the example, in split-site |
| 533 |
configurations the two floating addresses can be expected to belong to |
| 534 |
two separate IP networks -- it is thus vital for routers and firewalls |
| 535 |
to properly allow DRBD replication traffic between the nodes. |
| 536 |
|
| 537 |
|
| 538 |
[[s-pacemaker-floating-peers-crm-config]] |
| 539 |
==== Pacemaker resource configuration |
| 540 |
|
| 541 |
A DRBD floating peers setup, in terms of Pacemaker configuration, |
| 542 |
involves the following items (in each of the two Pacemaker clusters |
| 543 |
involved): |
| 544 |
|
| 545 |
* A virtual cluster IP address. |
| 546 |
|
| 547 |
* A master/slave DRBD resource (using the DRBD OCF resource agent). |
| 548 |
|
| 549 |
* Pacemaker constraints ensuring that resources are started on the |
| 550 |
correct nodes, and in the correct order. |
| 551 |
|
| 552 |
To configure a resource named `mysql` in a floating peers |
| 553 |
configuration in a 2-node cluster, using the replication address |
| 554 |
`10.9.9.100`, configure Pacemaker with the following `crm` commands: |
| 555 |
|
| 556 |
---------------------------- |
| 557 |
crm configure |
| 558 |
crm(live)configure# primitive p_ip_float_left ocf:heartbeat:IPaddr2 \ |
| 559 |
params ip=10.9.9.100 |
| 560 |
crm(live)configure# primitive p_drbd_mysql ocf:linbit:drbd \ |
| 561 |
params drbd_resource=mysql |
| 562 |
crm(live)configure# ms ms_drbd_mysql drbd_mysql \ |
| 563 |
meta master-max="1" master-node-max="1" \ |
| 564 |
clone-max="1" clone-node-max="1" \ |
| 565 |
notify="true" target-role="Master" |
| 566 |
crm(live)configure# order drbd_after_left \ |
| 567 |
inf: p_ip_float_left ms_drbd_mysql |
| 568 |
crm(live)configure# colocation drbd_on_left \ |
| 569 |
inf: ms_drbd_mysql p_ip_float_left |
| 570 |
crm(live)configure# commit |
| 571 |
bye |
| 572 |
---------------------------- |
| 573 |
|
| 574 |
After adding this configuration to the CIB, Pacemaker will execute the |
| 575 |
following actions: |
| 576 |
|
| 577 |
. Bring up the IP address 10.9.9.100 (on either 'alice' or 'bob'). |
| 578 |
. Bring up the DRBD resource according to the IP address configured. |
| 579 |
. Promote the DRBD resource to the Primary role. |
| 580 |
|
| 581 |
Then, in order to create the matching configuration in the other |
| 582 |
cluster, configure _that_ Pacemaker instance with the following |
| 583 |
commands: |
| 584 |
|
| 585 |
---------------------------- |
| 586 |
crm configure |
| 587 |
crm(live)configure# primitive p_ip_float_right ocf:heartbeat:IPaddr2 \ |
| 588 |
params ip=10.9.10.101 |
| 589 |
crm(live)configure# primitive drbd_mysql ocf:linbit:drbd \ |
| 590 |
params drbd_resource=mysql |
| 591 |
crm(live)configure# ms ms_drbd_mysql drbd_mysql \ |
| 592 |
meta master-max="1" master-node-max="1" \ |
| 593 |
clone-max="1" clone-node-max="1" \ |
| 594 |
notify="true" target-role="Slave" |
| 595 |
crm(live)configure# order drbd_after_right \ |
| 596 |
inf: p_ip_float_right ms_drbd_mysql |
| 597 |
crm(live)configure# colocation drbd_on_right |
| 598 |
inf: ms_drbd_mysql p_ip_float_right |
| 599 |
crm(live)configure# commit |
| 600 |
bye |
| 601 |
---------------------------- |
| 602 |
|
| 603 |
After adding this configuration to the CIB, Pacemaker will execute the |
| 604 |
following actions: |
| 605 |
|
| 606 |
. Bring up the IP address 10.9.10.101 (on either 'charlie' or |
| 607 |
'daisy'). |
| 608 |
. Bring up the DRBD resource according to the IP address configured. |
| 609 |
. Leave the DRBD resource in the Secondary role (due to |
| 610 |
`target-role="Slave"`). |
| 611 |
|
| 612 |
|
| 613 |
[[s-pacemaker-floating-peers-site-fail-over]] |
| 614 |
==== Site fail-over |
| 615 |
|
| 616 |
In split-site configurations, it may be necessary to transfer services |
| 617 |
from one site to another. This may be a consequence of a scheduled |
| 618 |
transition, or of a disastrous event. In case the transition is a |
| 619 |
normal, anticipated event, the recommended course of action is this: |
| 620 |
|
| 621 |
* Connect to the cluster on the site about to relinquish resources, |
| 622 |
and change the affected DRBD resource's `target-role` attribute from |
| 623 |
`Master` to `Slave`. This will shut down any resources depending on |
| 624 |
the Primary role of the DRBD resource, demote it, and continue to |
| 625 |
run, ready to receive updates from a new Primary. |
| 626 |
|
| 627 |
* Connect to the cluster on the site about to take over resources, and |
| 628 |
change the affected DRBD resource's `target-role` attribute from |
| 629 |
`Slave` to `Master`. This will promote the DRBD resources, start any |
| 630 |
other Pacemaker resources depending on the Primary role of the DRBD |
| 631 |
resource, and replicate updates to the remote site. |
| 632 |
|
| 633 |
* To fail back, simply reverse the procedure. |
| 634 |
|
| 635 |
In the event that of a catastrophic outage on the active site, it can |
| 636 |
be expected that the site is off line and no longer replicated to the |
| 637 |
backup site. In such an event: |
| 638 |
|
| 639 |
* Connect to the cluster on the still-functioning site resources, and |
| 640 |
change the affected DRBD resource's `target-role` attribute from |
| 641 |
`Slave` to `Master`. This will promote the DRBD resources, and start |
| 642 |
any other Pacemaker resources depending on the Primary role of the |
| 643 |
DRBD resource. |
| 644 |
|
| 645 |
* When the original site is restored or rebuilt, you may connect the |
| 646 |
DRBD resources again, and subsequently fail back using the reverse |
| 647 |
procedure. |
| 648 |
|
| 649 |
[[s-pacemaker-drbd-attr]] |
| 650 |
=== Importing DRBD's promotion scores into the CIB |
| 651 |
|
| 652 |
IMPORTANT: Everything described in this section depends on the `drbd-attr` |
| 653 |
OCF resource agent. It is available since drbd-utils version 9.15.0. On |
| 654 |
Debian/Ubuntu systems this is part of the `drbd-utils` package. On RPM |
| 655 |
based Linux distributions you need to install the `drbd-pacemaker` package. |
| 656 |
|
| 657 |
Every DRBD resource exposes a _promotion score_ on each node where it is |
| 658 |
configured. It is a numeric value that might be 0 or positive. The value |
| 659 |
reflects how desirable it is to promote the resource to master on this |
| 660 |
particular node. A node that has an _UpToDate_ disk and two _UpToDate_ |
| 661 |
replicas has a higher score than a node with an _UpToDate_ disk and |
| 662 |
just one _UpToDate_ replica. |
| 663 |
|
| 664 |
During startup, the _promotion score_ is 0. E.g., before the DRBD device has |
| 665 |
its backing device attached, or, if quorum is enabled, before quorum is |
| 666 |
gained. A value of 0 indicates that a promotion request will fail, |
| 667 |
and is mapped to a pacemaker score that indicates _must not run here_. |
| 668 |
|
| 669 |
The `drbd-attr` OCF resource agent imports these promotion scores into node |
| 670 |
attributes of a Pacemaker cluster. It needs to be configured like this: |
| 671 |
|
| 672 |
[source,drbd] |
| 673 |
---------------------------- |
| 674 |
primitive drbd-attr ocf:linbit:drbd-attr |
| 675 |
clone drbd-attr-clone drbd-attr |
| 676 |
---------------------------- |
| 677 |
|
| 678 |
These are _transient_ attributes (have a _lifetime_ of _reboot_ in pacemaker speak). That means, after |
| 679 |
a _reboot_ of the node, or local restart of pacemaker, those attributes will not exist until an instance of `drbd-attr` |
| 680 |
is started on that node. |
| 681 |
|
| 682 |
You can inspect the generated attributes with `crm_mon -A -1`. |
| 683 |
|
| 684 |
These attributed can be used in constraints for services that depend on the |
| 685 |
DRBD devices, or, when managing DRBD with the `ocf:linbit:drbd` resource agent, |
| 686 |
for the _Master_ role of that DRBD instance. |
| 687 |
|
| 688 |
Here is an example location constraint for the example resource from <<s-pacemaker-drbd-background>> |
| 689 |
[source,drbd] |
| 690 |
---------------------------- |
| 691 |
location lo_fs_mysql fs_mysql \ |
| 692 |
rule -inf: not_defined drbd-promotion-score-mysql \ |
| 693 |
rule drbd-promotion-score-mysql: defined drbd-promotion-score-mysql |
| 694 |
---------------------------- |
| 695 |
This means, that as long as the attribute is not defined, the fs_mysql file |
| 696 |
system cannot be mounted here. When the attribute is defined, its value |
| 697 |
becomes the score of the location constraint. |
| 698 |
|
| 699 |
This can also be used to cause Pacemaker to migrate a service away when |
| 700 |
DRBD loses a local backing device. Because a failed backing block device |
| 701 |
causes the promotion score to drop, other nodes with working backing devices |
| 702 |
will expose higher promotion scores. |
| 703 |
|
| 704 |
The attributes are updated live, independent of the resource-agent's |
| 705 |
monitor operation, with a dampening delay of 5 seconds by default. |
| 706 |
|
| 707 |
The resource agent has these optional parameters, |
| 708 |
see also its man page `ocf_linbit_drbd-attr(7)`: |
| 709 |
|
| 710 |
* `dampening_delay` |
| 711 |
* `attr_name_prefix` |
| 712 |
* `record_event_details` |
| 713 |
|
| 714 |
// Keep the empty line before this comment, otherwise the next chapter is folded into this |