| 1 |
=== Upgrading DRBD |
| 2 |
|
| 3 |
Upgrading DRBD is a fairly simple process. This section will cover |
| 4 |
the process of upgrading from 8.4.x to 9.0.x in great detail; for within-9 |
| 5 |
upgrades it gets to be much easier, see the short version <<s-upgrade-within-9,below>>. |
| 6 |
|
| 7 |
==== Compatibility |
| 8 |
DRBD is _wire protocol_ compatible over minor versions. Its _wire protocol_ is |
| 9 |
independent of the host kernel version and the machines' CPU architectures. |
| 10 |
|
| 11 |
DRBD is protocol compatible within a major number. E.g. All version 8.x.y releases |
| 12 |
are protocol compatible. |
| 13 |
|
| 14 |
DRBD 9.0.x is compatible with all drbd-8.x.y releases. |
| 15 |
DRBD 9.1.x is compatible with all drbd-9.0.x releases but no longer with drbd-8.x.y. |
| 16 |
|
| 17 |
[[s-upgrade-overview]] |
| 18 |
==== General overview |
| 19 |
|
| 20 |
The general process for upgrading 8.4 to 9.0 is as follows: |
| 21 |
|
| 22 |
* Configure the <<s-updating-your-repo,new repositories>> (if using packages from LINBIT) |
| 23 |
* Make sure that the current situation <<s-upgrade-check,is okay>> |
| 24 |
* <<s-upgrade-pausing-the-cluster,Pause>> any cluster manager |
| 25 |
* Get <<s-Upgrading-the-packages,new versions>> installed |
| 26 |
* If wanting to move to more than 2 nodes, then you'll need to resize the lower-level storage to provide room for the additional meta-data; this topic is discussed in the <<ch-lvm,LVM Chapter>>. |
| 27 |
* Deconfigure resources, unload DRBD 8.4, and <<s-upgrade-reload-kernel-mod,load the v9 kernel module>> |
| 28 |
* <<s-upgrade-convert,Convert DRBD meta-data>> to format `v09`, perhaps changing the number of bitmaps in the same step |
| 29 |
* Take the DRBD <<s-upgrade-start-drbd,resources up>>, and be happy |
| 30 |
|
| 31 |
|
| 32 |
[[s-updating-your-repo]] |
| 33 |
==== Updating your repository |
| 34 |
|
| 35 |
Due to the number of changes between the 8.4 and 9.0 branches we |
| 36 |
have created separate repositories for each. Perform this repository |
| 37 |
update on both servers. |
| 38 |
|
| 39 |
[[s-RHEL-systems]] |
| 40 |
===== RHEL/CentOS systems |
| 41 |
|
| 42 |
ifndef::de-brand[] |
| 43 |
Edit your `/etc/yum.repos.d/linbit.repo` file to reflect the following |
| 44 |
changes. |
| 45 |
|
| 46 |
---------------------------- |
| 47 |
[drbd-9.0] |
| 48 |
name=DRBD 9.0 |
| 49 |
baseurl=http://packages.linbit.com/<hash>/yum/rhel7/drbd-9.0/<arch> |
| 50 |
gpgcheck=0 |
| 51 |
---------------------------- |
| 52 |
|
| 53 |
NOTE: You will have to populate the <hash> and <arch> variables. The |
| 54 |
<hash> is provided by LINBIT support services. |
| 55 |
endif::de-brand[] |
| 56 |
ifdef::de-brand[] |
| 57 |
Edit your `*.repo` file to reflect the necessary changes. |
| 58 |
endif::de-brand[] |
| 59 |
|
| 60 |
[[s-Debian-Systems]] |
| 61 |
===== Debian/Ubuntu systems |
| 62 |
|
| 63 |
ifndef::de-brand[] |
| 64 |
Edit `/etc/apt/sources.list` (or a file in `/etc/apt/sources.d/`) to reflect the |
| 65 |
following changes. |
| 66 |
|
| 67 |
---------------------------- |
| 68 |
deb http://packages.linbit.com/<hash>/ bullseye drbd-9.0 |
| 69 |
---------------------------- |
| 70 |
|
| 71 |
In case you're not using the `bullseye` release, but some other, you'll need to |
| 72 |
change that line. |
| 73 |
|
| 74 |
NOTE: You will have to populate the `<hash>` variable. The |
| 75 |
`<hash>` is provided by LINBIT support services. |
| 76 |
|
| 77 |
|
| 78 |
Next you will want to add the DRBD signing key to your trusted keys. |
| 79 |
|
| 80 |
---------------------------- |
| 81 |
# gpg --keyserver subkeys.pgp.net --recv-keys 0x282B6E23 |
| 82 |
# gpg --export -a 282B6E23 | apt-key add - |
| 83 |
---------------------------- |
| 84 |
endif::de-brand[] |
| 85 |
ifdef::de-brand[] |
| 86 |
Edit `/etc/apt/sources.list` (or a file in `/etc/apt/sources.d/`) to reflect the |
| 87 |
the necessary changes. |
| 88 |
endif::de-brand[] |
| 89 |
|
| 90 |
Lastly perform an `apt update` so Debian recognizes the updated repository. |
| 91 |
|
| 92 |
---------------------------- |
| 93 |
# apt update |
| 94 |
---------------------------- |
| 95 |
|
| 96 |
[[s-upgrade-check]] |
| 97 |
==== Checking the DRBD state |
| 98 |
|
| 99 |
Before you begin make sure your resources are in sync. The output of |
| 100 |
`cat /proc/drbd` (which is only available _before_ 9.0) should show _UpToDate/UpToDate_. |
| 101 |
|
| 102 |
---------------------------- |
| 103 |
bob# cat /proc/drbd |
| 104 |
|
| 105 |
version: 8.4.9-1 (api:1/proto:86-101) |
| 106 |
GIT-hash: e081fb0570183db40caa29b26cb8ee907e9a7db3 build by linbit@buildsystem, 2016-11-18 14:49:21 |
| 107 |
|
| 108 |
0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r----- |
| 109 |
ns:0 nr:211852 dw:211852 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0 |
| 110 |
---------------------------- |
| 111 |
|
| 112 |
|
| 113 |
[[s-upgrade-pausing-the-cluster]] |
| 114 |
==== Pausing the services |
| 115 |
|
| 116 |
Now that you know the resources are in sync, start by upgrading the |
| 117 |
secondary node. |
| 118 |
This can be done manually or according to your cluster manager's documentation. |
| 119 |
ifndef::drbd-only[] |
| 120 |
Both processes are covered |
| 121 |
below. If you're running Pacemaker do not use the manual method. |
| 122 |
endif::drbd-only[] |
| 123 |
|
| 124 |
* Manual Method |
| 125 |
---------------------------- |
| 126 |
bob# systemctl stop [email protected] |
| 127 |
---------------------------- |
| 128 |
|
| 129 |
ifndef::drbd-only[] |
| 130 |
* Pacemaker |
| 131 |
|
| 132 |
Put the secondary node into standby mode. In this example 'bob' is secondary. |
| 133 |
|
| 134 |
---------------------------- |
| 135 |
bob# crm node standby bob |
| 136 |
---------------------------- |
| 137 |
|
| 138 |
NOTE: You can watch the status of your cluster using `crm_mon -rf` or watch |
| 139 |
`cat /proc/drbd` until it shows _Unconfigured_ for your resources. |
| 140 |
endif::drbd-only[] |
| 141 |
|
| 142 |
|
| 143 |
[[s-Upgrading-the-packages]] |
| 144 |
==== Upgrading the packages |
| 145 |
|
| 146 |
Now update your packages with either yum or apt. |
| 147 |
|
| 148 |
---------------------------- |
| 149 |
bob# yum upgrade |
| 150 |
---------------------------- |
| 151 |
|
| 152 |
---------------------------- |
| 153 |
bob# apt upgrade |
| 154 |
---------------------------- |
| 155 |
|
| 156 |
Once the upgrade is finished will now have the latest DRBD 9.0 kernel |
| 157 |
module and drbd-utils installed on your secondary node, 'bob'. |
| 158 |
|
| 159 |
But the kernel module is not active yet. |
| 160 |
|
| 161 |
[[s-upgrade-reload-kernel-mod]] |
| 162 |
==== Loading the new Kernel module |
| 163 |
|
| 164 |
By now the DRBD module should not be in use anymore, so we unload it via |
| 165 |
|
| 166 |
------------- |
| 167 |
bob# rmmod drbd_transport_tcp; rmmod drbd |
| 168 |
------------- |
| 169 |
|
| 170 |
If there's a message like `ERROR: Module drbd is in use`, then not all |
| 171 |
resources have been correctly stopped yet. + |
| 172 |
Retry <<s-upgrading-drbd>>, and/or run the command `drbdadm down all` to find |
| 173 |
out which resources are still active. |
| 174 |
|
| 175 |
Typical issues that might prevent unloading are these: |
| 176 |
|
| 177 |
* NFS export on a DRBD-backed filesystem (see `exportfs -v` output) |
| 178 |
* Filesystem still mounted - check `grep drbd /proc/mounts` |
| 179 |
* Loopback device active (`losetup -l`) |
| 180 |
* Device mapper using DRBD, directly or indirectly (`dmsetup ls --tree`) |
| 181 |
* LVM with a DRBD-PV (`pvs`) |
| 182 |
|
| 183 |
Please note that this list isn't complete - these are just the most common |
| 184 |
examples. |
| 185 |
|
| 186 |
Now we can load the new DRBD module: |
| 187 |
|
| 188 |
------- |
| 189 |
bob# modprobe drbd |
| 190 |
------- |
| 191 |
|
| 192 |
Now you should check the contents of `/proc/drbd` and verify that the correct |
| 193 |
(new) version is loaded; if the installed packages is for the wrong kernel |
| 194 |
version, the `modprobe` would be successful, but you'd be left with the old |
| 195 |
version being active again. |
| 196 |
|
| 197 |
The output of `cat /proc/drbd` should now show 9.0.x and look similar |
| 198 |
to this. |
| 199 |
|
| 200 |
---------------------------- |
| 201 |
version: 9.0.0 (api:2/proto:86-110) |
| 202 |
GIT-hash: 768965a7f158d966bd3bd4ff1014af7b3d9ff10c build by root@bob, 2015-09-03 13:58:02 |
| 203 |
Transports (api:10): tcp (1.0.0) |
| 204 |
---------------------------- |
| 205 |
|
| 206 |
NOTE: On the primary node, alice, 'cat /proc/drbd' will still show the |
| 207 |
prior version, until you upgrade it. |
| 208 |
|
| 209 |
//////////////////////// |
| 210 |
At this point the cluster is running two different versions of DRBD. While this |
| 211 |
is not recommended to be used for longer time spans, it is inevitable for the (short) upgrade period. |
| 212 |
|
| 213 |
Stop |
| 214 |
any service using DRBD and then DRBD on the primary node, alice, and promote |
| 215 |
'bob'. Again this can be done either manually or via the Pacemaker shell. |
| 216 |
|
| 217 |
* Manually |
| 218 |
---------------------------- |
| 219 |
alice # umount /dev/drbd/by-res/r0 |
| 220 |
alice # /etc/init.d/drbd stop |
| 221 |
bob # drbdadm primary r0 |
| 222 |
bob # mount /dev/drbd/by-res/r0/0 /mnt/drbd |
| 223 |
---------------------------- |
| 224 |
Please note that the mount command now references '/0' which defines |
| 225 |
the volume number of a resource. See <<s-recent-changes-volumes>> for |
| 226 |
more information on the new volumes feature. |
| 227 |
|
| 228 |
* Pacemaker |
| 229 |
---------------------------- |
| 230 |
# crm node standby alice |
| 231 |
---------------------------- |
| 232 |
|
| 233 |
WARNING: This will interrupt running services by stopping them and |
| 234 |
migrating them to the secondary server, bob. |
| 235 |
|
| 236 |
At this point you can safely upgrade DRBD by using yum or apt. |
| 237 |
|
| 238 |
---------------------------- |
| 239 |
alice# yum upgrade |
| 240 |
---------------------------- |
| 241 |
|
| 242 |
---------------------------- |
| 243 |
alice# apt upgrade |
| 244 |
---------------------------- |
| 245 |
|
| 246 |
Once the upgrade is complete you will now have the latest version |
| 247 |
of DRBD on alice and can start DRBD. |
| 248 |
|
| 249 |
* Manually |
| 250 |
---------------------------- |
| 251 |
alice# /etc/init.d/drbd start |
| 252 |
---------------------------- |
| 253 |
|
| 254 |
* Pacemaker |
| 255 |
---------------------------- |
| 256 |
alice# crm node online alice |
| 257 |
---------------------------- |
| 258 |
|
| 259 |
NOTE: Services will still be located on bob and will remain there |
| 260 |
until you migrate them back. |
| 261 |
|
| 262 |
Both servers should now show the latest version of DRBD in a connected |
| 263 |
state. |
| 264 |
|
| 265 |
---------------------------- |
| 266 |
# cat /proc/drbd |
| 267 |
version: 9.0.0 (api:2/proto:86-110) |
| 268 |
GIT-hash: 768965a7f158d966bd3bd4ff1014af7b3d9ff10c build by root@bob, 2015-09-03 13:58:02 |
| 269 |
Transports (api:10): tcp (1.0.0) |
| 270 |
|
| 271 |
# drbdsetup status |
| 272 |
r0 role:Secondary |
| 273 |
disk:UpToDate |
| 274 |
bob role:Secondary |
| 275 |
peer-disk:UpToDate |
| 276 |
---------------------------- |
| 277 |
|
| 278 |
//////////////////////// |
| 279 |
|
| 280 |
[[s-migrating_your_configuration_files]] |
| 281 |
==== Migrating your configuration files |
| 282 |
|
| 283 |
DRBD 9.0 is backward compatible with the 8.4 configuration files; |
| 284 |
however, some |
| 285 |
syntax has changed. See <<s-recent-changes-config>> for |
| 286 |
a full list of changes. In the meantime you can port your old |
| 287 |
configs fairly easily by using 'drbdadm dump all' command. This |
| 288 |
will output both a new global config followed by the |
| 289 |
new resource config files. Take this output and make changes |
| 290 |
accordingly. |
| 291 |
|
| 292 |
[[s-upgrade-convert]] |
| 293 |
|
| 294 |
==== Changing the meta-data |
| 295 |
|
| 296 |
Now you need to convert the on-disk metadata to the new version; this is really |
| 297 |
easy, it's just running one command and |
| 298 |
acknowledging two questions. |
| 299 |
|
| 300 |
|
| 301 |
If you want to change the number of nodes, you should already have increased |
| 302 |
the size of the lower level device, so that there is enough space to store the |
| 303 |
additional bitmaps; in that case, you'd run the command below with an |
| 304 |
additional argument `--max-peers=__<N>__`. When determining the number of |
| 305 |
(possible) peers please take setups like the <<s-drbd-client>> into account. |
| 306 |
|
| 307 |
Upgrading the DRBD metadata is as easy as running one command, and |
| 308 |
acknowledging the two questions: |
| 309 |
|
| 310 |
----------------- |
| 311 |
# drbdadm create-md <resource> |
| 312 |
You want me to create a v09 style flexible-size internal meta data block. |
| 313 |
There appears to be a v08 flexible-size internal meta data block |
| 314 |
already in place on <disk> at byte offset <offset> |
| 315 |
|
| 316 |
Valid v08 meta-data found, convert to v09? |
| 317 |
[need to type 'yes' to confirm] yes |
| 318 |
|
| 319 |
md_offset <offsets...> |
| 320 |
al_offset <offsets...> |
| 321 |
bm_offset <offsets...> |
| 322 |
|
| 323 |
Found some data |
| 324 |
|
| 325 |
==> This might destroy existing data! <== |
| 326 |
|
| 327 |
Do you want to proceed? |
| 328 |
[need to type 'yes' to confirm] yes |
| 329 |
|
| 330 |
Writing meta data... |
| 331 |
New drbd meta data block successfully created. |
| 332 |
success |
| 333 |
----------------- |
| 334 |
|
| 335 |
Of course, you can pass `all` for the resource names, too; and if you feel |
| 336 |
really lucky, you can avoid the questions via a command line like this here, |
| 337 |
too. (Yes, the order is important.) |
| 338 |
|
| 339 |
-------- |
| 340 |
drbdadm -v --max-peers=<N> -- --force create-md <resources> |
| 341 |
-------- |
| 342 |
|
| 343 |
|
| 344 |
[[s-upgrade-start-drbd]] |
| 345 |
==== Starting DRBD again |
| 346 |
|
| 347 |
Now, the only thing left to do is to get the DRBD devices up and running again - a simple `drbdadm up all` should do the trick. |
| 348 |
|
| 349 |
Now, depending on whether you've got a cluster manager or keep track of your |
| 350 |
resources manually, there are two different ways again. If you are using a cluster manager follow its documentation. |
| 351 |
|
| 352 |
* Manually |
| 353 |
|
| 354 |
---------------------------- |
| 355 |
bob# systemctl start [email protected] |
| 356 |
---------------------------- |
| 357 |
|
| 358 |
ifndef::drbd-only[] |
| 359 |
* Pacemaker |
| 360 |
---------------------------- |
| 361 |
# crm node online bob |
| 362 |
---------------------------- |
| 363 |
endif::drbd-only[] |
| 364 |
|
| 365 |
This should make DRBD connect to the other node, and the resynchronization |
| 366 |
process will start. |
| 367 |
|
| 368 |
When the two nodes are _UpToDate_ on all resources again, you can move your |
| 369 |
applications to the already upgraded node (here 'bob'), and then follow the |
| 370 |
same steps on the cluster node still running 8.4. |
| 371 |
|
| 372 |
|
| 373 |
[[s-upgrade-within-9]] |
| 374 |
==== From DRBD 9 to DRBD 9 |
| 375 |
|
| 376 |
If you are already running 9.0, it is sufficient to |
| 377 |
<<s-Upgrading-the-packages,install new package versions>>, make the cluster |
| 378 |
node <<s-upgrade-pausing-the-cluster,_standby_>>, |
| 379 |
<<s-upgrade-reload-kernel-mod,unload/reload>> the kernel module, |
| 380 |
<<s-upgrade-start-drbd,start the resources>>, and make the cluster node |
| 381 |
_online_ againfootnote:[At least that's the state at the time of writing - that's how it has been in the past, and we want to keep it that easy. But who |
| 382 |
knows? Who can tell? ;)]. |
| 383 |
|
| 384 |
These individual steps have been detailed above, so we won't repeat them here. |