| 1 |
[[ch-gfs]] |
| 2 |
== Using GFS with DRBD |
| 3 |
|
| 4 |
indexterm:[GFS]indexterm:[Global File System]This chapter outlines the |
| 5 |
steps necessary to set up a DRBD resource as a block device holding a |
| 6 |
shared Global File System (GFS). It covers both GFS and GFS2. |
| 7 |
|
| 8 |
In order to use GFS on top of DRBD, you must configure DRBD in |
| 9 |
indexterm:[dual-primary mode]<<s-dual-primary-mode,dual-primary |
| 10 |
mode>>. |
| 11 |
|
| 12 |
[IMPORTANT] |
| 13 |
=============================== |
| 14 |
All cluster file systems _require_ fencing - not only via the DRBD |
| 15 |
resource, but STONITH! A faulty member _must_ be killed. |
| 16 |
|
| 17 |
You'll want these settings: |
| 18 |
|
| 19 |
net { |
| 20 |
fencing resource-and-stonith; |
| 21 |
} |
| 22 |
handlers { |
| 23 |
# Make sure the other node is confirmed |
| 24 |
# dead after this! |
| 25 |
outdate-peer "/sbin/kill-other-node.sh"; |
| 26 |
} |
| 27 |
|
| 28 |
There must be _no_ volatile caches! |
| 29 |
Please see https://fedorahosted.org/cluster/wiki/DRBD_Cookbook for some more information. |
| 30 |
=============================== |
| 31 |
|
| 32 |
|
| 33 |
[[s-gfs-primer]] |
| 34 |
=== GFS primer |
| 35 |
|
| 36 |
The Red Hat Global File System (GFS) is Red Hat's implementation of a |
| 37 |
concurrent-access shared storage file system. As any such filesystem, |
| 38 |
GFS allows multiple nodes to access the same storage device, in |
| 39 |
read/write fashion, simultaneously without risking data corruption. It |
| 40 |
does so by using a Distributed Lock Manager (DLM) which manages |
| 41 |
concurrent access from cluster members. |
| 42 |
|
| 43 |
GFS was designed, from the outset, for use with conventional shared |
| 44 |
storage devices. Regardless, it is perfectly possible to use DRBD, in |
| 45 |
dual-primary mode, as a replicated storage device for |
| 46 |
GFS. Applications may benefit from reduced read/write latency due to |
| 47 |
the fact that DRBD normally reads from and writes to local storage, as |
| 48 |
opposed to the SAN devices GFS is normally configured to run |
| 49 |
from. Also, of course, DRBD adds an additional physical copy to every |
| 50 |
GFS filesystem, thus adding redundancy to the concept. |
| 51 |
|
| 52 |
GFS makes use of a cluster-aware variant of LVM, indexterm:[LVM]termed |
| 53 |
Cluster Logical Volume Manager or indexterm:[CLVM]CLVM. As such, some parallelism |
| 54 |
exists between using DRBD as the data storage for GFS, and using |
| 55 |
<<s-lvm-drbd-as-pv,DRBD as a Physical Volume for conventional LVM>>. |
| 56 |
|
| 57 |
GFS file systems are usually tightly integrated with Red Hat's own |
| 58 |
cluster management framework, the indexterm:[Red Hat Cluster |
| 59 |
Suite]<<ch-rhcs,Red Hat Cluster>>. This chapter explains |
| 60 |
the use of DRBD in conjunction with GFS in the Red Hat Cluster context. |
| 61 |
|
| 62 |
GFS, CLVM, and Red Hat Cluster are available in Red Hat |
| 63 |
Enterprise Linux (RHEL) and distributions derived from it, such as |
| 64 |
indexterm:[CentOS]CentOS. Packages built from the same sources are |
| 65 |
also available in indexterm:[Debian GNU/Linux]Debian GNU/Linux. This |
| 66 |
chapter assumes running GFS on a Red Hat Enterprise Linux system. |
| 67 |
|
| 68 |
[[s-gfs-create-resource]] |
| 69 |
=== Creating a DRBD resource suitable for GFS |
| 70 |
|
| 71 |
Since GFS is a shared cluster file system expecting concurrent |
| 72 |
read/write storage access from all cluster nodes, any DRBD resource to |
| 73 |
be used for storing a GFS filesystem must be configured in |
| 74 |
<<s-dual-primary-mode,dual-primary mode>>. Also, it is recommended to |
| 75 |
use some of DRBD's |
| 76 |
<<s-automatic-split-brain-recovery-configuration,features for |
| 77 |
automatic recovery from split brain>>. To |
| 78 |
do all this, include the following lines in the resource |
| 79 |
configuration: indexterm:[drbd.conf] |
| 80 |
|
| 81 |
[source,drbd] |
| 82 |
---------------------------- |
| 83 |
resource <resource> { |
| 84 |
net { |
| 85 |
allow-two-primaries yes; |
| 86 |
after-sb-0pri discard-zero-changes; |
| 87 |
after-sb-1pri discard-secondary; |
| 88 |
after-sb-2pri disconnect; |
| 89 |
... |
| 90 |
} |
| 91 |
... |
| 92 |
} |
| 93 |
---------------------------- |
| 94 |
|
| 95 |
[WARNING] |
| 96 |
=============================== |
| 97 |
By configuring auto-recovery policies, you are configuring effectively configuring automatic data-loss! Be sure you understand the implications. |
| 98 |
=============================== |
| 99 |
|
| 100 |
|
| 101 |
Once you have added these options to <<ch-configure,your |
| 102 |
freshly-configured resource>>, you may <<s-first-time-up,initialize |
| 103 |
your resource as you normally would>>. Since the |
| 104 |
indexterm:[drbd.conf]`allow-two-primaries` option is set to `yes` for |
| 105 |
this resource, you will be able to <<s-switch-resource-roles,promote |
| 106 |
the resource>>to the primary role on two nodes. |
| 107 |
|
| 108 |
[[s-gfs-configure-lvm]] |
| 109 |
=== Configuring LVM to recognize the DRBD resource |
| 110 |
|
| 111 |
GFS uses CLVM, the cluster-aware version of LVM, to manage block |
| 112 |
devices to be used by GFS. In order to use CLVM with DRBD, ensure that |
| 113 |
your LVM configuration |
| 114 |
|
| 115 |
* uses clustered locking. To do this, set the following option in |
| 116 |
`/etc/lvm/lvm.conf`: |
| 117 |
+ |
| 118 |
[source,drbd] |
| 119 |
---------------------------- |
| 120 |
locking_type = 3 |
| 121 |
---------------------------- |
| 122 |
|
| 123 |
* scans your DRBD devices to recognize DRBD-based Physical Volumes |
| 124 |
(PVs). This applies as to conventional (non-clustered) LVM; see |
| 125 |
<<s-lvm-drbd-as-pv>> for details. |
| 126 |
|
| 127 |
[[s-gfs-enable]] |
| 128 |
=== Configuring your cluster to support GFS |
| 129 |
|
| 130 |
After you have created your new DRBD resource and |
| 131 |
<<s-rhcs-config,completed your initial cluster configuration>>, you |
| 132 |
must enable and start the following system services on both nodes of |
| 133 |
your GFS cluster: |
| 134 |
|
| 135 |
* `cman` (this also starts `ccsd` and `fenced`), |
| 136 |
|
| 137 |
* `clvmd`. |
| 138 |
|
| 139 |
|
| 140 |
|
| 141 |
[[s-gfs-create]] |
| 142 |
=== Creating a GFS filesystem |
| 143 |
|
| 144 |
In order to create a GFS filesystem on your dual-primary DRBD |
| 145 |
resource, you must first initialize it as a <<s-lvm-primer,Logical |
| 146 |
Volume for LVM>>. |
| 147 |
|
| 148 |
Contrary to conventional, non-cluster-aware LVM configurations, the |
| 149 |
following steps must be completed on only one node due to the |
| 150 |
cluster-aware nature of CLVM: indexterm:[LVM]indexterm:[pvcreate (LVM |
| 151 |
command)]indexterm:[LVM]indexterm:[vgcreate (LVM |
| 152 |
command)]indexterm:[LVM]indexterm:[lvcreate (LVM command)] |
| 153 |
|
| 154 |
---------------------------- |
| 155 |
# pvcreate /dev/drbd/by-res/<resource>/0 |
| 156 |
Physical volume "/dev/drbd<num>" successfully created |
| 157 |
# vgcreate <vg-name> /dev/drbd/by-res/<resource>/0 |
| 158 |
Volume group "<vg-name>" successfully created |
| 159 |
# lvcreate --size <size> --name <lv-name> <vg-name> |
| 160 |
Logical volume "<lv-name>" created |
| 161 |
---------------------------- |
| 162 |
|
| 163 |
NOTE: This example assumes a single-volume resource. |
| 164 |
|
| 165 |
CLVM will immediately notify the peer node of these changes; |
| 166 |
indexterm:[LVM]indexterm:[lvdisplay (LVM |
| 167 |
command)]indexterm:[LVM]indexterm:[lvs (LVM command)] issuing `lvs` |
| 168 |
(or `lvdisplay`) on the peer node will list the newly created logical |
| 169 |
volume. |
| 170 |
|
| 171 |
indexterm:[GFS]Now, you may proceed by creating the actual filesystem: |
| 172 |
---------------------------- |
| 173 |
# mkfs -t gfs -p lock_dlm -j 2 /dev/<vg-name>/<lv-name> |
| 174 |
---------------------------- |
| 175 |
|
| 176 |
Or, for a GFS2 filesystem: |
| 177 |
|
| 178 |
---------------------------- |
| 179 |
# mkfs -t gfs2 -p lock_dlm -j 2 -t <cluster>:<name> |
| 180 |
/dev/<vg-name>/<lv-name> |
| 181 |
---------------------------- |
| 182 |
|
| 183 |
The `-j` option in this command refers to the number of journals to |
| 184 |
keep for GFS. This must be identical to the number of nodes with concurrent _Primary_ role in the GFS |
| 185 |
cluster; since DRBD does not support more than two _Primary_ nodes until DRBD 9.1, the value to |
| 186 |
set here is always 2. |
| 187 |
|
| 188 |
The `-t` option, applicable only for GFS2 filesystems, defines the lock |
| 189 |
table name. This follows the format _<cluster>:<name>_, where _<cluster>_ |
| 190 |
must match your cluster name as defined in |
| 191 |
`/etc/cluster/cluster.conf`. Thus, only members of that cluster will |
| 192 |
be permitted to use the filesystem. By contrast, _<name>_ is an |
| 193 |
arbitrary file system name unique in the cluster. |
| 194 |
|
| 195 |
[[s-gfs-use]] |
| 196 |
=== Using your GFS filesystem |
| 197 |
|
| 198 |
After you have created your filesystem, you may add it to |
| 199 |
`/etc/fstab`: |
| 200 |
|
| 201 |
[source,fstab] |
| 202 |
---------------------------- |
| 203 |
/dev/<vg-name>/<lv-name> <mountpoint> gfs defaults 0 0 |
| 204 |
---------------------------- |
| 205 |
|
| 206 |
For a GFS2 filesystem, simply change the filesystem type: |
| 207 |
|
| 208 |
[source,fstab] |
| 209 |
---------------------------- |
| 210 |
/dev/<vg-name>/<lv-name> <mountpoint> gfs2 defaults 0 0 |
| 211 |
---------------------------- |
| 212 |
|
| 213 |
Do not forget to make this change on both (or, with DRBD 9.1, all) cluster nodes. |
| 214 |
|
| 215 |
After this, you may mount your new filesystem by starting the |
| 216 |
`gfs` service (on both nodes): indexterm:[GFS] |
| 217 |
|
| 218 |
---------------------------- |
| 219 |
# service gfs start |
| 220 |
---------------------------- |
| 221 |
|
| 222 |
From then onwards, as long as you have DRBD configured to start |
| 223 |
automatically on system startup, before the RHCS services and the |
| 224 |
`gfs` service, you will be able to use this GFS file system as you |
| 225 |
would use one that is configured on traditional shared storage. |