Debian Codegraph @

UG9/en/gfs.adoc

raw
1 [[ch-gfs]]
2 == Using GFS with DRBD
3
4 indexterm:[GFS]indexterm:[Global File System]This chapter outlines the
5 steps necessary to set up a DRBD resource as a block device holding a
6 shared Global File System (GFS). It covers both GFS and GFS2.
7
8 In order to use GFS on top of DRBD, you must configure DRBD in
9 indexterm:[dual-primary mode]<<s-dual-primary-mode,dual-primary
10 mode>>.
11
12 [IMPORTANT]
13 ===============================
14 All cluster file systems _require_ fencing - not only via the DRBD
15 resource, but STONITH! A faulty member _must_ be killed.
16
17 You'll want these settings:
18
19 net {
20 fencing resource-and-stonith;
21 }
22 handlers {
23 # Make sure the other node is confirmed
24 # dead after this!
25 outdate-peer "/sbin/kill-other-node.sh";
26 }
27
28 There must be _no_ volatile caches!
29 Please see https://fedorahosted.org/cluster/wiki/DRBD_Cookbook for some more information.
30 ===============================
31
32
33 [[s-gfs-primer]]
34 === GFS primer
35
36 The Red Hat Global File System (GFS) is Red Hat's implementation of a
37 concurrent-access shared storage file system. As any such filesystem,
38 GFS allows multiple nodes to access the same storage device, in
39 read/write fashion, simultaneously without risking data corruption. It
40 does so by using a Distributed Lock Manager (DLM) which manages
41 concurrent access from cluster members.
42
43 GFS was designed, from the outset, for use with conventional shared
44 storage devices. Regardless, it is perfectly possible to use DRBD, in
45 dual-primary mode, as a replicated storage device for
46 GFS. Applications may benefit from reduced read/write latency due to
47 the fact that DRBD normally reads from and writes to local storage, as
48 opposed to the SAN devices GFS is normally configured to run
49 from. Also, of course, DRBD adds an additional physical copy to every
50 GFS filesystem, thus adding redundancy to the concept.
51
52 GFS makes use of a cluster-aware variant of LVM, indexterm:[LVM]termed
53 Cluster Logical Volume Manager or indexterm:[CLVM]CLVM. As such, some parallelism
54 exists between using DRBD as the data storage for GFS, and using
55 <<s-lvm-drbd-as-pv,DRBD as a Physical Volume for conventional LVM>>.
56
57 GFS file systems are usually tightly integrated with Red Hat's own
58 cluster management framework, the indexterm:[Red Hat Cluster
59 Suite]<<ch-rhcs,Red Hat Cluster>>. This chapter explains
60 the use of DRBD in conjunction with GFS in the Red Hat Cluster context.
61
62 GFS, CLVM, and Red Hat Cluster are available in Red Hat
63 Enterprise Linux (RHEL) and distributions derived from it, such as
64 indexterm:[CentOS]CentOS. Packages built from the same sources are
65 also available in indexterm:[Debian GNU/Linux]Debian GNU/Linux. This
66 chapter assumes running GFS on a Red Hat Enterprise Linux system.
67
68 [[s-gfs-create-resource]]
69 === Creating a DRBD resource suitable for GFS
70
71 Since GFS is a shared cluster file system expecting concurrent
72 read/write storage access from all cluster nodes, any DRBD resource to
73 be used for storing a GFS filesystem must be configured in
74 <<s-dual-primary-mode,dual-primary mode>>. Also, it is recommended to
75 use some of DRBD's
76 <<s-automatic-split-brain-recovery-configuration,features for
77 automatic recovery from split brain>>. To
78 do all this, include the following lines in the resource
79 configuration: indexterm:[drbd.conf]
80
81 [source,drbd]
82 ----------------------------
83 resource <resource> {
84 net {
85 allow-two-primaries yes;
86 after-sb-0pri discard-zero-changes;
87 after-sb-1pri discard-secondary;
88 after-sb-2pri disconnect;
89 ...
90 }
91 ...
92 }
93 ----------------------------
94
95 [WARNING]
96 ===============================
97 By configuring auto-recovery policies, you are configuring effectively configuring automatic data-loss! Be sure you understand the implications.
98 ===============================
99
100
101 Once you have added these options to <<ch-configure,your
102 freshly-configured resource>>, you may <<s-first-time-up,initialize
103 your resource as you normally would>>. Since the
104 indexterm:[drbd.conf]`allow-two-primaries` option is set to `yes` for
105 this resource, you will be able to <<s-switch-resource-roles,promote
106 the resource>>to the primary role on two nodes.
107
108 [[s-gfs-configure-lvm]]
109 === Configuring LVM to recognize the DRBD resource
110
111 GFS uses CLVM, the cluster-aware version of LVM, to manage block
112 devices to be used by GFS. In order to use CLVM with DRBD, ensure that
113 your LVM configuration
114
115 * uses clustered locking. To do this, set the following option in
116 `/etc/lvm/lvm.conf`:
117 +
118 [source,drbd]
119 ----------------------------
120 locking_type = 3
121 ----------------------------
122
123 * scans your DRBD devices to recognize DRBD-based Physical Volumes
124 (PVs). This applies as to conventional (non-clustered) LVM; see
125 <<s-lvm-drbd-as-pv>> for details.
126
127 [[s-gfs-enable]]
128 === Configuring your cluster to support GFS
129
130 After you have created your new DRBD resource and
131 <<s-rhcs-config,completed your initial cluster configuration>>, you
132 must enable and start the following system services on both nodes of
133 your GFS cluster:
134
135 * `cman` (this also starts `ccsd` and `fenced`),
136
137 * `clvmd`.
138
139
140
141 [[s-gfs-create]]
142 === Creating a GFS filesystem
143
144 In order to create a GFS filesystem on your dual-primary DRBD
145 resource, you must first initialize it as a <<s-lvm-primer,Logical
146 Volume for LVM>>.
147
148 Contrary to conventional, non-cluster-aware LVM configurations, the
149 following steps must be completed on only one node due to the
150 cluster-aware nature of CLVM: indexterm:[LVM]indexterm:[pvcreate (LVM
151 command)]indexterm:[LVM]indexterm:[vgcreate (LVM
152 command)]indexterm:[LVM]indexterm:[lvcreate (LVM command)]
153
154 ----------------------------
155 # pvcreate /dev/drbd/by-res/<resource>/0
156 Physical volume "/dev/drbd<num>" successfully created
157 # vgcreate <vg-name> /dev/drbd/by-res/<resource>/0
158 Volume group "<vg-name>" successfully created
159 # lvcreate --size <size> --name <lv-name> <vg-name>
160 Logical volume "<lv-name>" created
161 ----------------------------
162
163 NOTE: This example assumes a single-volume resource.
164
165 CLVM will immediately notify the peer node of these changes;
166 indexterm:[LVM]indexterm:[lvdisplay (LVM
167 command)]indexterm:[LVM]indexterm:[lvs (LVM command)] issuing `lvs`
168 (or `lvdisplay`) on the peer node will list the newly created logical
169 volume.
170
171 indexterm:[GFS]Now, you may proceed by creating the actual filesystem:
172 ----------------------------
173 # mkfs -t gfs -p lock_dlm -j 2 /dev/<vg-name>/<lv-name>
174 ----------------------------
175
176 Or, for a GFS2 filesystem:
177
178 ----------------------------
179 # mkfs -t gfs2 -p lock_dlm -j 2 -t <cluster>:<name>
180 /dev/<vg-name>/<lv-name>
181 ----------------------------
182
183 The `-j` option in this command refers to the number of journals to
184 keep for GFS. This must be identical to the number of nodes with concurrent _Primary_ role in the GFS
185 cluster; since DRBD does not support more than two _Primary_ nodes until DRBD 9.1, the value to
186 set here is always 2.
187
188 The `-t` option, applicable only for GFS2 filesystems, defines the lock
189 table name. This follows the format _<cluster>:<name>_, where _<cluster>_
190 must match your cluster name as defined in
191 `/etc/cluster/cluster.conf`. Thus, only members of that cluster will
192 be permitted to use the filesystem. By contrast, _<name>_ is an
193 arbitrary file system name unique in the cluster.
194
195 [[s-gfs-use]]
196 === Using your GFS filesystem
197
198 After you have created your filesystem, you may add it to
199 `/etc/fstab`:
200
201 [source,fstab]
202 ----------------------------
203 /dev/<vg-name>/<lv-name> <mountpoint> gfs defaults 0 0
204 ----------------------------
205
206 For a GFS2 filesystem, simply change the filesystem type:
207
208 [source,fstab]
209 ----------------------------
210 /dev/<vg-name>/<lv-name> <mountpoint> gfs2 defaults 0 0
211 ----------------------------
212
213 Do not forget to make this change on both (or, with DRBD 9.1, all) cluster nodes.
214
215 After this, you may mount your new filesystem by starting the
216 `gfs` service (on both nodes): indexterm:[GFS]
217
218 ----------------------------
219 # service gfs start
220 ----------------------------
221
222 From then onwards, as long as you have DRBD configured to start
223 automatically on system startup, before the RHCS services and the
224 `gfs` service, you will be able to use this GFS file system as you
225 would use one that is configured on traditional shared storage.