Debian Codegraph @

UG9/en/upgrading.adoc

raw
1 === Upgrading DRBD
2
3 Upgrading DRBD is a fairly simple process. This section will cover
4 the process of upgrading from 8.4.x to 9.0.x in great detail; for within-9
5 upgrades it gets to be much easier, see the short version <<s-upgrade-within-9,below>>.
6
7 ==== Compatibility
8 DRBD is _wire protocol_ compatible over minor versions. Its _wire protocol_ is
9 independent of the host kernel version and the machines' CPU architectures.
10
11 DRBD is protocol compatible within a major number. E.g. All version 8.x.y releases
12 are protocol compatible.
13
14 DRBD 9.0.x is compatible with all drbd-8.x.y releases.
15 DRBD 9.1.x is compatible with all drbd-9.0.x releases but no longer with drbd-8.x.y.
16
17 [[s-upgrade-overview]]
18 ==== General overview
19
20 The general process for upgrading 8.4 to 9.0 is as follows:
21
22 * Configure the <<s-updating-your-repo,new repositories>> (if using packages from LINBIT)
23 * Make sure that the current situation <<s-upgrade-check,is okay>>
24 * <<s-upgrade-pausing-the-cluster,Pause>> any cluster manager
25 * Get <<s-Upgrading-the-packages,new versions>> installed
26 * If wanting to move to more than 2 nodes, then you'll need to resize the lower-level storage to provide room for the additional meta-data; this topic is discussed in the <<ch-lvm,LVM Chapter>>.
27 * Deconfigure resources, unload DRBD 8.4, and <<s-upgrade-reload-kernel-mod,load the v9 kernel module>>
28 * <<s-upgrade-convert,Convert DRBD meta-data>> to format `v09`, perhaps changing the number of bitmaps in the same step
29 * Take the DRBD <<s-upgrade-start-drbd,resources up>>, and be happy
30
31
32 [[s-updating-your-repo]]
33 ==== Updating your repository
34
35 Due to the number of changes between the 8.4 and 9.0 branches we
36 have created separate repositories for each. Perform this repository
37 update on both servers.
38
39 [[s-RHEL-systems]]
40 ===== RHEL/CentOS systems
41
42 ifndef::de-brand[]
43 Edit your `/etc/yum.repos.d/linbit.repo` file to reflect the following
44 changes.
45
46 ----------------------------
47 [drbd-9.0]
48 name=DRBD 9.0
49 baseurl=http://packages.linbit.com/<hash>/yum/rhel7/drbd-9.0/<arch>
50 gpgcheck=0
51 ----------------------------
52
53 NOTE: You will have to populate the <hash> and <arch> variables. The
54 <hash> is provided by LINBIT support services.
55 endif::de-brand[]
56 ifdef::de-brand[]
57 Edit your `*.repo` file to reflect the necessary changes.
58 endif::de-brand[]
59
60 [[s-Debian-Systems]]
61 ===== Debian/Ubuntu systems
62
63 ifndef::de-brand[]
64 Edit `/etc/apt/sources.list` (or a file in `/etc/apt/sources.d/`) to reflect the
65 following changes.
66
67 ----------------------------
68 deb http://packages.linbit.com/<hash>/ bullseye drbd-9.0
69 ----------------------------
70
71 In case you're not using the `bullseye` release, but some other, you'll need to
72 change that line.
73
74 NOTE: You will have to populate the `<hash>` variable. The
75 `<hash>` is provided by LINBIT support services.
76
77
78 Next you will want to add the DRBD signing key to your trusted keys.
79
80 ----------------------------
81 # gpg --keyserver subkeys.pgp.net --recv-keys 0x282B6E23
82 # gpg --export -a 282B6E23 | apt-key add -
83 ----------------------------
84 endif::de-brand[]
85 ifdef::de-brand[]
86 Edit `/etc/apt/sources.list` (or a file in `/etc/apt/sources.d/`) to reflect the
87 the necessary changes.
88 endif::de-brand[]
89
90 Lastly perform an `apt update` so Debian recognizes the updated repository.
91
92 ----------------------------
93 # apt update
94 ----------------------------
95
96 [[s-upgrade-check]]
97 ==== Checking the DRBD state
98
99 Before you begin make sure your resources are in sync. The output of
100 `cat /proc/drbd` (which is only available _before_ 9.0) should show _UpToDate/UpToDate_.
101
102 ----------------------------
103 bob# cat /proc/drbd
104
105 version: 8.4.9-1 (api:1/proto:86-101)
106 GIT-hash: e081fb0570183db40caa29b26cb8ee907e9a7db3 build by linbit@buildsystem, 2016-11-18 14:49:21
107
108 0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
109 ns:0 nr:211852 dw:211852 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
110 ----------------------------
111
112
113 [[s-upgrade-pausing-the-cluster]]
114 ==== Pausing the services
115
116 Now that you know the resources are in sync, start by upgrading the
117 secondary node.
118 This can be done manually or according to your cluster manager's documentation.
119 ifndef::drbd-only[]
120 Both processes are covered
121 below. If you're running Pacemaker do not use the manual method.
122 endif::drbd-only[]
123
124 * Manual Method
125 ----------------------------
126 bob# systemctl stop [email protected]
127 ----------------------------
128
129 ifndef::drbd-only[]
130 * Pacemaker
131
132 Put the secondary node into standby mode. In this example 'bob' is secondary.
133
134 ----------------------------
135 bob# crm node standby bob
136 ----------------------------
137
138 NOTE: You can watch the status of your cluster using `crm_mon -rf` or watch
139 `cat /proc/drbd` until it shows _Unconfigured_ for your resources.
140 endif::drbd-only[]
141
142
143 [[s-Upgrading-the-packages]]
144 ==== Upgrading the packages
145
146 Now update your packages with either yum or apt.
147
148 ----------------------------
149 bob# yum upgrade
150 ----------------------------
151
152 ----------------------------
153 bob# apt upgrade
154 ----------------------------
155
156 Once the upgrade is finished will now have the latest DRBD 9.0 kernel
157 module and drbd-utils installed on your secondary node, 'bob'.
158
159 But the kernel module is not active yet.
160
161 [[s-upgrade-reload-kernel-mod]]
162 ==== Loading the new Kernel module
163
164 By now the DRBD module should not be in use anymore, so we unload it via
165
166 -------------
167 bob# rmmod drbd_transport_tcp; rmmod drbd
168 -------------
169
170 If there's a message like `ERROR: Module drbd is in use`, then not all
171 resources have been correctly stopped yet. +
172 Retry <<s-upgrading-drbd>>, and/or run the command `drbdadm down all` to find
173 out which resources are still active.
174
175 Typical issues that might prevent unloading are these:
176
177 * NFS export on a DRBD-backed filesystem (see `exportfs -v` output)
178 * Filesystem still mounted - check `grep drbd /proc/mounts`
179 * Loopback device active (`losetup -l`)
180 * Device mapper using DRBD, directly or indirectly (`dmsetup ls --tree`)
181 * LVM with a DRBD-PV (`pvs`)
182
183 Please note that this list isn't complete - these are just the most common
184 examples.
185
186 Now we can load the new DRBD module:
187
188 -------
189 bob# modprobe drbd
190 -------
191
192 Now you should check the contents of `/proc/drbd` and verify that the correct
193 (new) version is loaded; if the installed packages is for the wrong kernel
194 version, the `modprobe` would be successful, but you'd be left with the old
195 version being active again.
196
197 The output of `cat /proc/drbd` should now show 9.0.x and look similar
198 to this.
199
200 ----------------------------
201 version: 9.0.0 (api:2/proto:86-110)
202 GIT-hash: 768965a7f158d966bd3bd4ff1014af7b3d9ff10c build by root@bob, 2015-09-03 13:58:02
203 Transports (api:10): tcp (1.0.0)
204 ----------------------------
205
206 NOTE: On the primary node, alice, 'cat /proc/drbd' will still show the
207 prior version, until you upgrade it.
208
209 ////////////////////////
210 At this point the cluster is running two different versions of DRBD. While this
211 is not recommended to be used for longer time spans, it is inevitable for the (short) upgrade period.
212
213 Stop
214 any service using DRBD and then DRBD on the primary node, alice, and promote
215 'bob'. Again this can be done either manually or via the Pacemaker shell.
216
217 * Manually
218 ----------------------------
219 alice # umount /dev/drbd/by-res/r0
220 alice # /etc/init.d/drbd stop
221 bob # drbdadm primary r0
222 bob # mount /dev/drbd/by-res/r0/0 /mnt/drbd
223 ----------------------------
224 Please note that the mount command now references '/0' which defines
225 the volume number of a resource. See <<s-recent-changes-volumes>> for
226 more information on the new volumes feature.
227
228 * Pacemaker
229 ----------------------------
230 # crm node standby alice
231 ----------------------------
232
233 WARNING: This will interrupt running services by stopping them and
234 migrating them to the secondary server, bob.
235
236 At this point you can safely upgrade DRBD by using yum or apt.
237
238 ----------------------------
239 alice# yum upgrade
240 ----------------------------
241
242 ----------------------------
243 alice# apt upgrade
244 ----------------------------
245
246 Once the upgrade is complete you will now have the latest version
247 of DRBD on alice and can start DRBD.
248
249 * Manually
250 ----------------------------
251 alice# /etc/init.d/drbd start
252 ----------------------------
253
254 * Pacemaker
255 ----------------------------
256 alice# crm node online alice
257 ----------------------------
258
259 NOTE: Services will still be located on bob and will remain there
260 until you migrate them back.
261
262 Both servers should now show the latest version of DRBD in a connected
263 state.
264
265 ----------------------------
266 # cat /proc/drbd
267 version: 9.0.0 (api:2/proto:86-110)
268 GIT-hash: 768965a7f158d966bd3bd4ff1014af7b3d9ff10c build by root@bob, 2015-09-03 13:58:02
269 Transports (api:10): tcp (1.0.0)
270
271 # drbdsetup status
272 r0 role:Secondary
273 disk:UpToDate
274 bob role:Secondary
275 peer-disk:UpToDate
276 ----------------------------
277
278 ////////////////////////
279
280 [[s-migrating_your_configuration_files]]
281 ==== Migrating your configuration files
282
283 DRBD 9.0 is backward compatible with the 8.4 configuration files;
284 however, some
285 syntax has changed. See <<s-recent-changes-config>> for
286 a full list of changes. In the meantime you can port your old
287 configs fairly easily by using 'drbdadm dump all' command. This
288 will output both a new global config followed by the
289 new resource config files. Take this output and make changes
290 accordingly.
291
292 [[s-upgrade-convert]]
293
294 ==== Changing the meta-data
295
296 Now you need to convert the on-disk metadata to the new version; this is really
297 easy, it's just running one command and
298 acknowledging two questions.
299
300
301 If you want to change the number of nodes, you should already have increased
302 the size of the lower level device, so that there is enough space to store the
303 additional bitmaps; in that case, you'd run the command below with an
304 additional argument `--max-peers=__<N>__`. When determining the number of
305 (possible) peers please take setups like the <<s-drbd-client>> into account.
306
307 Upgrading the DRBD metadata is as easy as running one command, and
308 acknowledging the two questions:
309
310 -----------------
311 # drbdadm create-md <resource>
312 You want me to create a v09 style flexible-size internal meta data block.
313 There appears to be a v08 flexible-size internal meta data block
314 already in place on <disk> at byte offset <offset>
315
316 Valid v08 meta-data found, convert to v09?
317 [need to type 'yes' to confirm] yes
318
319 md_offset <offsets...>
320 al_offset <offsets...>
321 bm_offset <offsets...>
322
323 Found some data
324
325 ==> This might destroy existing data! <==
326
327 Do you want to proceed?
328 [need to type 'yes' to confirm] yes
329
330 Writing meta data...
331 New drbd meta data block successfully created.
332 success
333 -----------------
334
335 Of course, you can pass `all` for the resource names, too; and if you feel
336 really lucky, you can avoid the questions via a command line like this here,
337 too. (Yes, the order is important.)
338
339 --------
340 drbdadm -v --max-peers=<N> -- --force create-md <resources>
341 --------
342
343
344 [[s-upgrade-start-drbd]]
345 ==== Starting DRBD again
346
347 Now, the only thing left to do is to get the DRBD devices up and running again - a simple `drbdadm up all` should do the trick.
348
349 Now, depending on whether you've got a cluster manager or keep track of your
350 resources manually, there are two different ways again. If you are using a cluster manager follow its documentation.
351
352 * Manually
353
354 ----------------------------
355 bob# systemctl start [email protected]
356 ----------------------------
357
358 ifndef::drbd-only[]
359 * Pacemaker
360 ----------------------------
361 # crm node online bob
362 ----------------------------
363 endif::drbd-only[]
364
365 This should make DRBD connect to the other node, and the resynchronization
366 process will start.
367
368 When the two nodes are _UpToDate_ on all resources again, you can move your
369 applications to the already upgraded node (here 'bob'), and then follow the
370 same steps on the cluster node still running 8.4.
371
372
373 [[s-upgrade-within-9]]
374 ==== From DRBD 9 to DRBD 9
375
376 If you are already running 9.0, it is sufficient to
377 <<s-Upgrading-the-packages,install new package versions>>, make the cluster
378 node <<s-upgrade-pausing-the-cluster,_standby_>>,
379 <<s-upgrade-reload-kernel-mod,unload/reload>> the kernel module,
380 <<s-upgrade-start-drbd,start the resources>>, and make the cluster node
381 _online_ againfootnote:[At least that's the state at the time of writing - that's how it has been in the past, and we want to keep it that easy. But who
382 knows? Who can tell? ;)].
383
384 These individual steps have been detailed above, so we won't repeat them here.