[[ch-gfs]] == Using GFS with DRBD indexterm:[GFS]indexterm:[Global File System]This chapter outlines the steps necessary to set up a DRBD resource as a block device holding a shared Global File System (GFS). It covers both GFS and GFS2. In order to use GFS on top of DRBD, you must configure DRBD in indexterm:[dual-primary mode]<>. [IMPORTANT] =============================== All cluster file systems _require_ fencing - not only via the DRBD resource, but STONITH! A faulty member _must_ be killed. You'll want these settings: net { fencing resource-and-stonith; } handlers { # Make sure the other node is confirmed # dead after this! outdate-peer "/sbin/kill-other-node.sh"; } There must be _no_ volatile caches! Please see https://fedorahosted.org/cluster/wiki/DRBD_Cookbook for some more information. =============================== [[s-gfs-primer]] === GFS primer The Red Hat Global File System (GFS) is Red Hat's implementation of a concurrent-access shared storage file system. As any such filesystem, GFS allows multiple nodes to access the same storage device, in read/write fashion, simultaneously without risking data corruption. It does so by using a Distributed Lock Manager (DLM) which manages concurrent access from cluster members. GFS was designed, from the outset, for use with conventional shared storage devices. Regardless, it is perfectly possible to use DRBD, in dual-primary mode, as a replicated storage device for GFS. Applications may benefit from reduced read/write latency due to the fact that DRBD normally reads from and writes to local storage, as opposed to the SAN devices GFS is normally configured to run from. Also, of course, DRBD adds an additional physical copy to every GFS filesystem, thus adding redundancy to the concept. GFS makes use of a cluster-aware variant of LVM, indexterm:[LVM]termed Cluster Logical Volume Manager or indexterm:[CLVM]CLVM. As such, some parallelism exists between using DRBD as the data storage for GFS, and using <>. GFS file systems are usually tightly integrated with Red Hat's own cluster management framework, the indexterm:[Red Hat Cluster Suite]<>. This chapter explains the use of DRBD in conjunction with GFS in the Red Hat Cluster context. GFS, CLVM, and Red Hat Cluster are available in Red Hat Enterprise Linux (RHEL) and distributions derived from it, such as indexterm:[CentOS]CentOS. Packages built from the same sources are also available in indexterm:[Debian GNU/Linux]Debian GNU/Linux. This chapter assumes running GFS on a Red Hat Enterprise Linux system. [[s-gfs-create-resource]] === Creating a DRBD resource suitable for GFS Since GFS is a shared cluster file system expecting concurrent read/write storage access from all cluster nodes, any DRBD resource to be used for storing a GFS filesystem must be configured in <>. Also, it is recommended to use some of DRBD's <>. To do all this, include the following lines in the resource configuration: indexterm:[drbd.conf] [source,drbd] ---------------------------- resource { net { allow-two-primaries yes; after-sb-0pri discard-zero-changes; after-sb-1pri discard-secondary; after-sb-2pri disconnect; ... } ... } ---------------------------- [WARNING] =============================== By configuring auto-recovery policies, you are configuring effectively configuring automatic data-loss! Be sure you understand the implications. =============================== Once you have added these options to <>, you may <>. Since the indexterm:[drbd.conf]`allow-two-primaries` option is set to `yes` for this resource, you will be able to <>to the primary role on two nodes. [[s-gfs-configure-lvm]] === Configuring LVM to recognize the DRBD resource GFS uses CLVM, the cluster-aware version of LVM, to manage block devices to be used by GFS. In order to use CLVM with DRBD, ensure that your LVM configuration * uses clustered locking. To do this, set the following option in `/etc/lvm/lvm.conf`: + [source,drbd] ---------------------------- locking_type = 3 ---------------------------- * scans your DRBD devices to recognize DRBD-based Physical Volumes (PVs). This applies as to conventional (non-clustered) LVM; see <> for details. [[s-gfs-enable]] === Configuring your cluster to support GFS After you have created your new DRBD resource and <>, you must enable and start the following system services on both nodes of your GFS cluster: * `cman` (this also starts `ccsd` and `fenced`), * `clvmd`. [[s-gfs-create]] === Creating a GFS filesystem In order to create a GFS filesystem on your dual-primary DRBD resource, you must first initialize it as a <>. Contrary to conventional, non-cluster-aware LVM configurations, the following steps must be completed on only one node due to the cluster-aware nature of CLVM: indexterm:[LVM]indexterm:[pvcreate (LVM command)]indexterm:[LVM]indexterm:[vgcreate (LVM command)]indexterm:[LVM]indexterm:[lvcreate (LVM command)] ---------------------------- # pvcreate /dev/drbd/by-res//0 Physical volume "/dev/drbd" successfully created # vgcreate /dev/drbd/by-res//0 Volume group "" successfully created # lvcreate --size --name Logical volume "" created ---------------------------- NOTE: This example assumes a single-volume resource. CLVM will immediately notify the peer node of these changes; indexterm:[LVM]indexterm:[lvdisplay (LVM command)]indexterm:[LVM]indexterm:[lvs (LVM command)] issuing `lvs` (or `lvdisplay`) on the peer node will list the newly created logical volume. indexterm:[GFS]Now, you may proceed by creating the actual filesystem: ---------------------------- # mkfs -t gfs -p lock_dlm -j 2 /dev// ---------------------------- Or, for a GFS2 filesystem: ---------------------------- # mkfs -t gfs2 -p lock_dlm -j 2 -t : /dev// ---------------------------- The `-j` option in this command refers to the number of journals to keep for GFS. This must be identical to the number of nodes with concurrent _Primary_ role in the GFS cluster; since DRBD does not support more than two _Primary_ nodes until DRBD 9.1, the value to set here is always 2. The `-t` option, applicable only for GFS2 filesystems, defines the lock table name. This follows the format _:_, where __ must match your cluster name as defined in `/etc/cluster/cluster.conf`. Thus, only members of that cluster will be permitted to use the filesystem. By contrast, __ is an arbitrary file system name unique in the cluster. [[s-gfs-use]] === Using your GFS filesystem After you have created your filesystem, you may add it to `/etc/fstab`: [source,fstab] ---------------------------- /dev// gfs defaults 0 0 ---------------------------- For a GFS2 filesystem, simply change the filesystem type: [source,fstab] ---------------------------- /dev// gfs2 defaults 0 0 ---------------------------- Do not forget to make this change on both (or, with DRBD 9.1, all) cluster nodes. After this, you may mount your new filesystem by starting the `gfs` service (on both nodes): indexterm:[GFS] ---------------------------- # service gfs start ---------------------------- From then onwards, as long as you have DRBD configured to start automatically on system startup, before the RHCS services and the `gfs` service, you will be able to use this GFS file system as you would use one that is configured on traditional shared storage.