Debian Codegraph @

UG9/en/lvm.adoc

raw
1 [[ch-lvm]]
2 == Using LVM with DRBD
3
4 indexterm:[LVM]indexterm:[Logical Volume Management]This chapter deals
5 with managing DRBD in conjunction with LVM2. In particular, it covers
6
7 * using LVM Logical Volumes as backing devices for DRBD;
8
9 * using DRBD devices as Physical Volumes for LVM;
10
11 * combining these to concepts to implement a layered LVM approach
12 using DRBD.
13
14 If you happen to be unfamiliar with these terms to begin with,
15 <<s-lvm-primer>> may serve as your LVM starting point -- although you
16 are always encouraged, of course, to familiarize yourself with LVM in
17 some more detail than this section provides.
18
19 [[s-lvm-primer]]
20 === LVM primer
21
22 LVM2 is an implementation of logical volume management in the context
23 of the Linux device mapper framework. It has practically nothing in
24 common, other than the name and acronym, with the original LVM
25 implementation. The old implementation (now retroactively named
26 "LVM1") is considered obsolete; it is not covered in this section.
27
28 When working with LVM, it is important to understand its most basic
29 concepts:
30
31
32 .Physical Volume (PV)
33 indexterm:[LVM]indexterm:[Physical Volume (LVM)]A PV is an underlying
34 block device exclusively managed by LVM. PVs can either be entire hard
35 disks or individual partitions. It is common practice to create a
36 partition table on the hard disk where one partition is dedicated to
37 the use by the Linux LVM.
38
39 NOTE: The partition type "Linux LVM" (signature `0x8E`) can be used to
40 identify partitions for exclusive use by LVM. This, however, is not
41 required -- LVM recognizes PVs by way of a signature written to the
42 device upon PV initialization.
43
44 .Volume Group (VG)
45 indexterm:[LVM]indexterm:[Volume Group (LVM)]A VG is the basic
46 administrative unit of the LVM. A VG may include one or more several
47 PVs. Every VG has a unique name. A VG may be extended during runtime
48 by adding additional PVs, or by enlarging an existing PV.
49
50 .Logical Volume (LV)
51 indexterm:[LVM]indexterm:[Logical Volume (LVM)]LVs may be created
52 during runtime within VGs and are available to the other parts of the
53 kernel as regular block devices. As such, they may be used to hold a
54 file system, or for any other purpose block devices may be used
55 for. LVs may be resized while they are online, and they may also be
56 moved from one PV to another (as long as the PVs are part of the same
57 VG).
58
59 .Snapshot Logical Volume (SLV)
60 indexterm:[snapshots (LVM)]indexterm:[LVM]Snapshots are temporary
61 point-in-time copies of LVs. Creating snapshots is an operation that
62 completes almost instantly, even if the original LV (the _origin
63 volume_) has a size of several hundred GiByte. Usually, a snapshot
64 requires significantly less space than the original LV.
65
66 [[f-lvm-overview]]
67 .LVM overview
68 image::images/lvm.svg[]
69
70
71 [[s-lvm-lv-as-drbd-backing-dev]]
72 === Using a Logical Volume as a DRBD backing device
73
74 indexterm:[LVM]indexterm:[Logical Volume (LVM)]Since an existing
75 Logical Volume is simply a block device in Linux terms, you may of
76 course use it as a DRBD backing device. To use LV's in this manner,
77 you simply create them, and then initialize them for DRBD as you
78 normally would.
79
80 This example assumes that a Volume Group named `foo` already exists on
81 both nodes of on your LVM-enabled system, and that you wish to create
82 a DRBD resource named `r0` using a Logical Volume in that Volume
83 Group.
84
85 First, you create the Logical Volume:
86 indexterm:[LVM]indexterm:[lvcreate (LVM command)]
87 ----------------------------
88 # lvcreate --name bar --size 10G foo
89 Logical volume "bar" created
90 ----------------------------
91
92 Of course, you must complete this command on both nodes of your DRBD
93 cluster. After this, you should have a block device named
94 `/dev/foo/bar` on either node.
95
96 Then, you can simply enter the newly-created volumes in your resource
97 configuration:
98
99 [source,drbd]
100 ----------------------------
101 resource r0 {
102 ...
103 on alice {
104 device /dev/drbd0;
105 disk /dev/foo/bar;
106 ...
107 }
108 on bob {
109 device /dev/drbd0;
110 disk /dev/foo/bar;
111 ...
112 }
113 }
114 ----------------------------
115
116 Now you can <<s-first-time-up,continue to bring your resource up>>,
117 just as you would if you were using non-LVM block devices.
118
119 [[s-lvm-snapshots]]
120 === Using automated LVM snapshots during DRBD synchronization
121
122 While DRBD is synchronizing, the __SyncTarget__'s state is
123 _Inconsistent_ until the synchronization completes. If in this
124 situation the _SyncSource_ happens to fail (beyond repair), this puts
125 you in an unfortunate position: the node with good data is dead, and
126 the surviving node has bad (inconsistent) data.
127
128 When serving DRBD off an LVM Logical Volume, you can mitigate this
129 problem by creating an automated snapshot when synchronization starts,
130 and automatically removing that same snapshot once synchronization has
131 completed successfully.
132
133 In order to enable automated snapshotting during resynchronization,
134 add the following lines to your resource configuration:
135
136 .Automating snapshots before DRBD synchronization
137 ----------------------------
138 resource r0 {
139 handlers {
140 before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh";
141 after-resync-target "/usr/lib/drbd/unsnapshot-resync-target-lvm.sh";
142 }
143 }
144 ----------------------------
145
146 The two scripts parse the `$DRBD_RESOURCE` environment variable which
147 DRBD automatically passes to any `handler` it invokes. The
148 `snapshot-resync-target-lvm.sh` script then creates an LVM snapshot for
149 any volume the resource contains, then synchronization
150 kicks off. In case the script fails, the synchronization _does not
151 commence_.
152
153 Once synchronization completes, the `unsnapshot-resync-target-lvm.sh`
154 script removes the snapshot, which is then no longer needed. In case
155 unsnapshotting fails, the snapshot continues to linger around.
156
157 IMPORTANT: You should review dangling snapshots as soon as
158 possible. A full snapshot causes both the snapshot itself _and its
159 origin volume_ to fail.
160
161 If at any time your _SyncSource_ does fail beyond repair and you
162 decide to revert to your latest snapshot on the peer, you may do so by
163 issuing the `lvconvert -M` command.
164
165 [[s-lvm-drbd-as-pv]]
166 === Configuring a DRBD resource as a Physical Volume
167
168 indexterm:[LVM]indexterm:[Physical Volume (LVM)]In order to prepare a
169 DRBD resource for use as a Physical Volume, it is necessary to create
170 a PV signature on the DRBD device. In order to do so, issue one of the
171 following commands on the node where the resource is currently in the
172 primary role: indexterm:[LVM]indexterm:[pvcreate (LVM command)]
173
174 ----------------------------
175 # pvcreate /dev/drbdX
176 ----------------------------
177
178 or
179
180 ----------------------------
181 # pvcreate /dev/drbd/by-res/<resource>/0
182 ----------------------------
183
184 NOTE: This example assumes a single-volume resource.
185
186 Now, it is necessary to include this device in the list of devices LVM
187 scans for PV signatures. In order to do this, you must edit the LVM
188 configuration file, normally named
189 indexterm:[LVM]`/etc/lvm/lvm.conf`. Find the line in the
190 `devices` section that contains the `filter` keyword and edit it
191 accordingly. If _all_ your PVs are to be stored on DRBD devices, the
192 following is an appropriate `filter` option:
193 indexterm:[LVM]indexterm:[filter expression (LVM)]
194
195 [source,drbd]
196 ----------------------------
197 filter = [ "a|drbd.*|", "r|.*|" ]
198 ----------------------------
199
200 This filter expression accepts PV signatures found on any DRBD
201 devices, while rejecting (ignoring) all others.
202
203 NOTE: By default, LVM scans all block devices found in `/dev` for PV
204 signatures. This is equivalent to `filter = [ "a|.*|" ]`.
205
206 If you want to use stacked resources as LVM PVs, then you will need a
207 more explicit filter configuration. You need to make sure that LVM
208 detects PV signatures on stacked resources, while ignoring them on the
209 corresponding lower-level resources and backing devices. This example
210 assumes that your lower-level DRBD resources use device minors 0
211 through 9, whereas your stacked resources are using device minors from
212 10 upwards:
213
214 [source,drbd]
215 ----------------------------
216 filter = [ "a|drbd1[0-9]|", "r|.*|" ]
217 ----------------------------
218
219 This filter expression accepts PV signatures found only on the DRBD
220 devices `/dev/drbd10` through `/dev/drbd19`, while rejecting
221 (ignoring) all others.
222
223 After modifying the `lvm.conf` file, you must run the
224 indexterm:[LVM]indexterm:[vgscan (LVM command)]`vgscan` command so LVM
225 discards its configuration cache and re-scans devices for PV
226 signatures.
227
228 You may of course use a different `filter` configuration to match your
229 particular system configuration. What is important to remember,
230 however, is that you need to
231
232 * Accept (include) the DRBD devices you wish to use as PVs;
233 * Reject (exclude) the corresponding lower-level devices, so as to
234 avoid LVM finding duplicate PV signatures.
235
236 In addition, you should disable the LVM cache by setting:
237
238 [source,drbd]
239 ----------------------------
240 write_cache_state = 0
241 ----------------------------
242
243 After disabling the LVM cache, make sure you remove any stale cache
244 entries by deleting `/etc/lvm/cache/.cache`.
245
246 You must repeat the above steps on the peer nodes, too.
247
248 IMPORTANT: If your system has its root filesystem on LVM, Volume
249 Groups will be activated from your initial ramdisk (initrd) during
250 boot. In doing so, the LVM tools will evaluate an `lvm.conf` file
251 included in the initrd image. Thus, after you make any changes to your
252 `lvm.conf`, you should be certain to update your initrd with the
253 utility appropriate for your distribution (`mkinitrd`,
254 `update-initramfs` etc.).
255
256 When you have configured your new PV, you may proceed to add it to a
257 Volume Group, or create a new Volume Group from it. The DRBD resource
258 must, of course, be in the primary role while doing
259 so. indexterm:[LVM]indexterm:[vgcreate (LVM command)]
260
261 ----------------------------
262 # vgcreate <name> /dev/drbdX
263 ----------------------------
264
265 NOTE: While it is possible to mix DRBD and non-DRBD Physical Volumes
266 within the same Volume Group, doing so is not recommended and unlikely
267 to be of any practical value.
268
269 When you have created your VG, you may start carving Logical Volumes
270 out of it, using the indexterm:[LVM]indexterm:[lvcreate (LVM
271 command)]`lvcreate` command (as with a non-DRBD-backed Volume Group).
272
273 [[s-lvm-add-pv]]
274 === Adding a new DRBD volume to an existing Volume Group
275
276 Occasionally, you may want to add new DRBD-backed Physical Volumes to
277 a Volume Group. Whenever you do so, a new volume should be added to an
278 existing resource configuration. This preserves the replication stream
279 and ensures write fidelity across all PVs in the VG.
280
281 ifndef::drbd-only[]
282 IMPORTANT: if your LVM volume group is managed by Pacemaker as
283 explained in <<s-lvm-pacemaker>>, it is _imperative_ to place the
284 cluster in maintenance mode prior to making changes to the DRBD
285 configuration.
286 endif::drbd-only[]
287
288 Extend your resource configuration to include an additional volume, as
289 in the following example:
290
291 -------------------------------------
292 resource r0 {
293 volume 0 {
294 device /dev/drbd1;
295 disk /dev/sda7;
296 meta-disk internal;
297 }
298 volume 1 {
299 device /dev/drbd2;
300 disk /dev/sda8;
301 meta-disk internal;
302 }
303 on alice {
304 address 10.1.1.31:7789;
305 }
306 on bob {
307 address 10.1.1.32:7789;
308 }
309 }
310 -------------------------------------
311
312 Make sure your DRBD configuration is identical across nodes, then
313 issue:
314
315 -------------------------------------
316 # drbdadm adjust r0
317 -------------------------------------
318
319 This will implicitly call `drbdsetup new-minor r0 1` to enable the new volume `1` in the resource `r0`. Once the new
320 volume has been added to the replication stream, you may initialize
321 and add it to the volume group:
322
323 -------------------------------------
324 # pvcreate /dev/drbd/by-res/<resource>/1
325 # vgextend <name> /dev/drbd/by-res/<resource>/1
326 -------------------------------------
327
328 This will add the new PV `/dev/drbd/by-res/<resource>/1` to the
329 `<name>` VG, preserving write fidelity across the entire VG.
330
331
332 [[s-nested-lvm]]
333 === Nested LVM configuration with DRBD
334
335 It is possible, if slightly advanced, to both use
336 indexterm:[LVM]indexterm:[Logical Volume (LVM)]Logical Volumes as
337 backing devices for DRBD _and_ at the same time use a DRBD device
338 itself as a indexterm:[LVM]indexterm:[Physical Volume (LVM)]Physical
339 Volume. To provide an example, consider the following configuration:
340
341 * We have two partitions, named `/dev/sda1`, and `/dev/sdb1`, which we
342 intend to use as Physical Volumes.
343
344 * Both of these PVs are to become part of a Volume Group named
345 `local`.
346
347 * We want to create a 10-GiB Logical Volume in this VG, to be named `r0`.
348
349 * This LV will become the local backing device for our DRBD resource,
350 also named `r0`, which corresponds to the device `/dev/drbd0`.
351
352 * This device will be the sole PV for another Volume Group, named
353 `replicated`.
354
355 * This VG is to contain two more logical volumes named `foo`(4 GiB)
356 and `bar`(6 GiB).
357
358 In order to enable this configuration, follow these steps:
359
360 * Set an appropriate `filter` option in your `/etc/lvm/lvm.conf`:
361 +
362 --
363 indexterm:[LVM]indexterm:[filter expression (LVM)]
364 [source,drbd]
365 ----------------------------
366 filter = ["a|sd.*|", "a|drbd.*|", "r|.*|"]
367 ----------------------------
368
369 This filter expression accepts PV signatures found on any SCSI and
370 DRBD devices, while rejecting (ignoring) all others.
371
372 After modifying the `lvm.conf` file, you must run the
373 indexterm:[LVM]indexterm:[vgscan (LVM command)]`vgscan` command so LVM
374 discards its configuration cache and re-scans devices for PV
375 signatures.
376 --
377
378
379 * Disable the LVM cache by setting:
380 +
381 --
382 [source,drbd]
383 ----------------------------
384 write_cache_state = 0
385 ----------------------------
386
387 After disabling the LVM cache, make sure you remove any stale cache
388 entries by deleting `/etc/lvm/cache/.cache`.
389 --
390
391 * Now, you may initialize your two SCSI partitions as PVs:
392 indexterm:[LVM]indexterm:[pvcreate (LVM command)]
393 +
394 --
395 ----------------------------
396 # pvcreate /dev/sda1
397 Physical volume "/dev/sda1" successfully created
398 # pvcreate /dev/sdb1
399 Physical volume "/dev/sdb1" successfully created
400 ----------------------------
401 --
402
403
404 * The next step is creating your low-level VG named `local`,
405 consisting of the two PVs you just initialized:
406 indexterm:[LVM]indexterm:[vgcreate (LVM command)]
407 +
408 ----------------------------
409 # vgcreate local /dev/sda1 /dev/sda2
410 Volume group "local" successfully created
411 ----------------------------
412
413 * Now you may create your Logical Volume to be used as DRBD's backing
414 device: indexterm:[LVM]indexterm:[lvcreate (LVM command)]
415 +
416 ----------------------------
417 # lvcreate --name r0 --size 10G local
418 Logical volume "r0" created
419 ----------------------------
420
421 * Repeat all steps, up to this point, on the peer node.
422
423 * Then, edit your `/etc/drbd.conf` to create a new resource named `r0`:
424 indexterm:[drbd.conf]
425 +
426 --
427 [source,drbd]
428 ----------------------------
429 resource r0 {
430 device /dev/drbd0;
431 disk /dev/local/r0;
432 meta-disk internal;
433 on <host> { address <address>:<port>; }
434 on <host> { address <address>:<port>; }
435 }
436 ----------------------------
437
438 After you have created your new resource configuration, be sure to
439 copy your `drbd.conf` contents to the peer node.
440 --
441
442 * After this, initialize your resource as described in
443 <<s-first-time-up>>(on both nodes).
444
445 * Then, promote your resource (on one node): indexterm:[drbdadm, primary]
446 +
447 ----------------------------
448 # drbdadm primary r0
449 ----------------------------
450
451 * Now, on the node where you just promoted your resource, initialize
452 your DRBD device as a new Physical Volume:
453 +
454 --
455 indexterm:[LVM]indexterm:[pvcreate (LVM command)]
456
457 ----------------------------
458 # pvcreate /dev/drbd0
459 Physical volume "/dev/drbd0" successfully created
460 ----------------------------
461 --
462
463 * Create your VG named `replicated`, using the PV you just
464 initialized, on the same node: indexterm:[LVM]indexterm:[vgcreate
465 (LVM command)]
466 +
467 --
468 ----------------------------
469 # vgcreate replicated /dev/drbd0
470 Volume group "replicated" successfully created
471 ----------------------------
472 --
473
474 * Finally, create your new Logical Volumes within this newly-created VG via indexterm:[LVM]indexterm:[lvcreate (LVM command)]
475 +
476 --
477 ----------------------------
478 # lvcreate --name foo --size 4G replicated
479 Logical volume "foo" created
480 # lvcreate --name bar --size 6G replicated
481 Logical volume "bar" created
482 ----------------------------
483 --
484
485 The Logical Volumes `foo` and `bar` will now be available as
486 `/dev/replicated/foo` and `/dev/replicated/bar` on the local node.
487
488 [[s-switching_the_vg_to_the_other_node]]
489 ==== Switching the VG to the other node ====
490
491 To make them available on the other node, first issue the following
492 sequence of commands on the primary node:
493 indexterm:[LVM]indexterm:[vgchange (LVM command)]
494
495 ----------------------------
496 # vgchange -a n replicated
497 0 logical volume(s) in volume group "replicated" now active
498 # drbdadm secondary r0
499 ----------------------------
500
501
502 Then, issue these commands on the other (still secondary) node:
503 indexterm:[drbdadm, primary]indexterm:[LVM]indexterm:[vgchange (LVM command)]
504
505 ----------------------------
506 # drbdadm primary r0
507 # vgchange -a y replicated
508 2 logical volume(s) in volume group "replicated" now active
509 ----------------------------
510
511 After this, the block devices `/dev/replicated/foo` and
512 `/dev/replicated/bar` will be available on the other (now primary) node.
513
514 ifndef::drbd-only[]
515 [[s-lvm-pacemaker]]
516
517 === Highly available LVM with Pacemaker
518
519 The process of transferring volume groups between peers and making the
520 corresponding logical volumes available can be automated. The
521 Pacemaker LVM resource agent is designed for exactly that purpose.
522
523 In order to put an existing, DRBD-backed volume group under Pacemaker
524 management, run the following commands in the `crm` shell:
525
526 .Pacemaker configuration for DRBD-backed LVM Volume Group
527 ----------------------------
528 primitive p_drbd_r0 ocf:linbit:drbd \
529 params drbd_resource="r0" \
530 op monitor interval="29s" role="Master" \
531 op monitor interval="31s" role="Slave"
532 ms ms_drbd_r0 p_drbd_r0 \
533 meta master-max="1" master-node-max="1" \
534 clone-max="2" clone-node-max="1" \
535 notify="true"
536 primitive p_lvm_r0 ocf:heartbeat:LVM \
537 params volgrpname="r0"
538 colocation c_lvm_on_drbd inf: p_lvm_r0 ms_drbd_r0:Master
539 order o_drbd_before_lvm inf: ms_drbd_r0:promote p_lvm_r0:start
540 commit
541 ----------------------------
542
543 After you have committed this configuration, Pacemaker will
544 automatically make the `r0` volume group available on whichever node
545 currently has the Primary (Master) role for the DRBD resource.
546 endif::drbd-only[]