| 1 |
[[ch-ocfs2]] |
| 2 |
== Using OCFS2 with DRBD |
| 3 |
|
| 4 |
indexterm:[OCFS2]indexterm:[Oracle Cluster File System]This chapter |
| 5 |
outlines the steps necessary to set up a DRBD resource as a block |
| 6 |
device holding a shared Oracle Cluster File System, version 2 (OCFS2). |
| 7 |
|
| 8 |
|
| 9 |
[IMPORTANT] |
| 10 |
=============================== |
| 11 |
All cluster file systems _require_ fencing - not only via the DRBD |
| 12 |
resource, but STONITH! A faulty member _must_ be killed. |
| 13 |
|
| 14 |
You'll want these settings: |
| 15 |
|
| 16 |
net { |
| 17 |
fencing resource-and-stonith; |
| 18 |
} |
| 19 |
handlers { |
| 20 |
# Make sure the other node is confirmed |
| 21 |
# dead after this! |
| 22 |
outdate-peer "/sbin/kill-other-node.sh"; |
| 23 |
} |
| 24 |
|
| 25 |
There must be _no_ volatile caches! |
| 26 |
You might take a few hints of the page at https://fedorahosted.org/cluster/wiki/DRBD_Cookbook, |
| 27 |
although that's about GFS2, not OCFS2. |
| 28 |
=============================== |
| 29 |
|
| 30 |
|
| 31 |
|
| 32 |
[[s-ocfs2-primer]] |
| 33 |
=== OCFS2 primer |
| 34 |
|
| 35 |
The Oracle Cluster File System, version 2 (OCFS2) is a concurrent |
| 36 |
access shared storage file system developed by Oracle |
| 37 |
Corporation. Unlike its predecessor OCFS, which was specifically |
| 38 |
designed and only suitable for Oracle database payloads, OCFS2 is a |
| 39 |
general-purpose filesystem that implements most POSIX semantics. The |
| 40 |
most common use case for OCFS2 is arguably Oracle Real Application |
| 41 |
Cluster (RAC), but OCFS2 may also be used for load-balanced NFS |
| 42 |
clusters, for example. |
| 43 |
|
| 44 |
Although originally designed for use with conventional shared storage |
| 45 |
devices, OCFS2 is equally well suited to be deployed on |
| 46 |
<<s-dual-primary-mode,dual-Primary DRBD>>. Applications reading from |
| 47 |
the filesystem may benefit from reduced read latency due to the fact |
| 48 |
that DRBD reads from and writes to local storage, as opposed to the |
| 49 |
SAN devices OCFS2 otherwise normally runs on. In addition, DRBD adds |
| 50 |
redundancy to OCFS2 by adding an additional copy to every filesystem |
| 51 |
image, as opposed to just a single filesystem image that is merely |
| 52 |
shared. |
| 53 |
|
| 54 |
Like other shared cluster file systems such as <<ch-gfs,GFS>>, OCFS2 |
| 55 |
allows multiple nodes to access the same storage device, in read/write |
| 56 |
mode, simultaneously without risking data corruption. It does so by |
| 57 |
using a Distributed Lock Manager (DLM) which manages concurrent access |
| 58 |
from cluster nodes. The DLM itself uses a virtual file system |
| 59 |
(`ocfs2_dlmfs`) which is separate from the actual OCFS2 file systems |
| 60 |
present on the system. |
| 61 |
|
| 62 |
OCFS2 may either use an intrinsic cluster communication layer to |
| 63 |
manage cluster membership and filesystem mount and unmount operation, |
| 64 |
or alternatively defer those tasks to the |
| 65 |
<<ch-pacemaker,Pacemaker>>cluster infrastructure. |
| 66 |
|
| 67 |
OCFS2 is available in SUSE Linux Enterprise Server (where it is the |
| 68 |
primarily supported shared cluster file system), CentOS, Debian |
| 69 |
GNU/Linux, and Ubuntu Server Edition. Oracle also provides packages |
| 70 |
for Red Hat Enterprise Linux (RHEL). This chapter assumes running |
| 71 |
OCFS2 on a SUSE Linux Enterprise Server system. |
| 72 |
|
| 73 |
[[s-ocfs2-create-resource]] |
| 74 |
=== Creating a DRBD resource suitable for OCFS2 |
| 75 |
|
| 76 |
Since OCFS2 is a shared cluster file system expecting concurrent |
| 77 |
read/write storage access from all cluster nodes, any DRBD resource to |
| 78 |
be used for storing a OCFS2 filesystem must be configured in |
| 79 |
<<s-dual-primary-mode,dual-primary mode>>. Also, it is recommended to |
| 80 |
use some of DRBD's |
| 81 |
<<s-automatic-split-brain-recovery-configuration,features for |
| 82 |
automatic recovery from split brain>>. To |
| 83 |
do all this, include the following lines in the resource |
| 84 |
configuration: indexterm:[drbd.conf] |
| 85 |
|
| 86 |
[source,drbd] |
| 87 |
---------------------------- |
| 88 |
resource <resource> { |
| 89 |
net { |
| 90 |
# allow-two-primaries yes; |
| 91 |
after-sb-0pri discard-zero-changes; |
| 92 |
after-sb-1pri discard-secondary; |
| 93 |
after-sb-2pri disconnect; |
| 94 |
... |
| 95 |
} |
| 96 |
... |
| 97 |
} |
| 98 |
---------------------------- |
| 99 |
|
| 100 |
[WARNING] |
| 101 |
=============================== |
| 102 |
By setting auto-recovery policies, you are effectively configuring automatic data-loss! Be sure you understand the implications. |
| 103 |
=============================== |
| 104 |
|
| 105 |
|
| 106 |
It is not recommended to set the `allow-two-primaries` option to `yes` |
| 107 |
upon initial configuration. You should do so after the initial |
| 108 |
resource synchronization has completed. |
| 109 |
|
| 110 |
Once you have added these options to <<ch-configure,your |
| 111 |
freshly-configured resource>>, you may <<s-first-time-up,initialize |
| 112 |
your resource as you normally would>>. After you set the |
| 113 |
indexterm:[drbd.conf]`allow-two-primaries` option to `yes` for this |
| 114 |
resource, you will be able to <<s-switch-resource-roles,promote the |
| 115 |
resource>>to the primary role on both nodes. |
| 116 |
|
| 117 |
NOTE: With DRBD 9.1 it will be possible to have more than two nodes in the |
| 118 |
_Primary_ role; with DRBD 9.0 you can only use two primaries, but more nodes |
| 119 |
can be in _Secondary_ role to provide more redundancy. |
| 120 |
|
| 121 |
|
| 122 |
[[s-ocfs2-create]] |
| 123 |
=== Creating an OCFS2 filesystem |
| 124 |
|
| 125 |
Now, use OCFS2's `mkfs` implementation to create the file system: |
| 126 |
---------------------------- |
| 127 |
# mkfs -t ocfs2 -N 2 -L ocfs2_drbd0 /dev/drbd0 |
| 128 |
mkfs.ocfs2 1.4.0 |
| 129 |
Filesystem label=ocfs2_drbd0 |
| 130 |
Block size=1024 (bits=10) |
| 131 |
Cluster size=4096 (bits=12) |
| 132 |
Volume size=205586432 (50192 clusters) (200768 blocks) |
| 133 |
7 cluster groups (tail covers 4112 clusters, rest cover 7680 clusters) |
| 134 |
Journal size=4194304 |
| 135 |
Initial number of node slots: 2 |
| 136 |
Creating bitmaps: done |
| 137 |
Initializing superblock: done |
| 138 |
Writing system files: done |
| 139 |
Writing superblock: done |
| 140 |
Writing backup superblock: 0 block(s) |
| 141 |
Formatting Journals: done |
| 142 |
Writing lost+found: done |
| 143 |
mkfs.ocfs2 successful |
| 144 |
---------------------------- |
| 145 |
|
| 146 |
This will create an OCFS2 file system with two node slots on |
| 147 |
`/dev/drbd0`, and set the filesystem label to `ocfs2_drbd0`. You may |
| 148 |
specify other options on `mkfs` invocation; please see the `mkfs.ocfs2` |
| 149 |
system manual page for details. |
| 150 |
|
| 151 |
[[s-ocfs2-pacemaker]] |
| 152 |
=== Pacemaker OCFS2 management |
| 153 |
|
| 154 |
[[s-ocfs2-pacemaker-drbd]] |
| 155 |
==== Adding a Dual-Primary DRBD resource to Pacemaker |
| 156 |
|
| 157 |
An existing <<s-ocfs2-create-resource,Dual-Primary DRBD resource>>may |
| 158 |
be added to Pacemaker resource management with the following |
| 159 |
`crm` configuration: |
| 160 |
|
| 161 |
[source,drbd] |
| 162 |
---------------------------- |
| 163 |
primitive p_drbd_ocfs2 ocf:linbit:drbd \ |
| 164 |
params drbd_resource="ocfs2" |
| 165 |
ms ms_drbd_ocfs2 p_drbd_ocfs2 \ |
| 166 |
meta master-max=2 clone-max=2 notify=true |
| 167 |
---------------------------- |
| 168 |
|
| 169 |
IMPORTANT: Note the `master-max=2` meta variable; it enables |
| 170 |
dual-Master mode for a Pacemaker master/slave set. This requires that |
| 171 |
`allow-two-primaries` is also set to `yes` in the DRBD |
| 172 |
configuration. Otherwise, Pacemaker will flag a configuration error |
| 173 |
during resource validation. |
| 174 |
|
| 175 |
[[s-ocfs2-pacemaker-mgmtdaemons]] |
| 176 |
==== Adding OCFS2 management capability to Pacemaker |
| 177 |
|
| 178 |
In order to manage OCFS2 and the kernel Distributed Lock Manager |
| 179 |
(DLM), Pacemaker uses a total of three different resource agents: |
| 180 |
|
| 181 |
* `ocf:pacemaker:controld` -- Pacemaker's interface to the DLM; |
| 182 |
|
| 183 |
* `ocf:ocfs2:o2cb` -- Pacemaker's interface to OCFS2 cluster |
| 184 |
management; |
| 185 |
|
| 186 |
* `ocf:heartbeat:Filesystem` -- the generic filesystem management |
| 187 |
resource agent which supports cluster file systems when configured |
| 188 |
as a Pacemaker clone. |
| 189 |
|
| 190 |
You may enable all nodes in a Pacemaker cluster for OCFS2 management |
| 191 |
by creating a _cloned group_ of resources, with the following |
| 192 |
`crm` configuration: |
| 193 |
|
| 194 |
[source,drbd] |
| 195 |
---------------------------- |
| 196 |
primitive p_controld ocf:pacemaker:controld |
| 197 |
primitive p_o2cb ocf:ocfs2:o2cb |
| 198 |
group g_ocfs2mgmt p_controld p_o2cb |
| 199 |
clone cl_ocfs2mgmt g_ocfs2mgmt meta interleave=true |
| 200 |
---------------------------- |
| 201 |
|
| 202 |
Once this configuration is committed, Pacemaker will start instances |
| 203 |
of the `controld` and `o2cb` resource types on all nodes in the cluster. |
| 204 |
|
| 205 |
[[s-ocfs2-pacemaker-fs]] |
| 206 |
==== Adding an OCFS2 filesystem to Pacemaker |
| 207 |
|
| 208 |
Pacemaker manages OCFS2 filesystems using the conventional |
| 209 |
`ocf:heartbeat:Filesystem` resource agent, albeit in clone mode. To |
| 210 |
put an OCFS2 filesystem under Pacemaker management, use the following |
| 211 |
`crm` configuration: |
| 212 |
|
| 213 |
[source,drbd] |
| 214 |
---------------------------- |
| 215 |
primitive p_fs_ocfs2 ocf:heartbeat:Filesystem \ |
| 216 |
params device="/dev/drbd/by-res/ocfs2/0" directory="/srv/ocfs2" \ |
| 217 |
fstype="ocfs2" options="rw,noatime" |
| 218 |
clone cl_fs_ocfs2 p_fs_ocfs2 |
| 219 |
---------------------------- |
| 220 |
|
| 221 |
NOTE: This example assumes a single-volume resource. |
| 222 |
|
| 223 |
[[s-ocfs2-pacemaker-constraints]] |
| 224 |
==== Adding required Pacemaker constraints to manage OCFS2 filesystems |
| 225 |
|
| 226 |
In order to tie all OCFS2-related resources and clones together, add |
| 227 |
the following constraints to your Pacemaker configuration: |
| 228 |
|
| 229 |
[source,drbd] |
| 230 |
---------------------------- |
| 231 |
order o_ocfs2 ms_drbd_ocfs2:promote cl_ocfs2mgmt:start cl_fs_ocfs2:start |
| 232 |
colocation c_ocfs2 cl_fs_ocfs2 cl_ocfs2mgmt ms_drbd_ocfs2:Master |
| 233 |
---------------------------- |
| 234 |
|
| 235 |
[[s-ocfs2-legacy]] |
| 236 |
=== Legacy OCFS2 management (without Pacemaker) |
| 237 |
|
| 238 |
IMPORTANT: The information presented in this section applies to legacy |
| 239 |
systems where OCFS2 DLM support is not available in Pacemaker. It is |
| 240 |
preserved here for reference purposes only. New installations should |
| 241 |
always use the <<s-ocfs2-pacemaker,Pacemaker>> approach. |
| 242 |
|
| 243 |
[[s-ocfs2-enable]] |
| 244 |
==== Configuring your cluster to support OCFS2 |
| 245 |
|
| 246 |
[[s-ocfs2-create-cluster-conf]] |
| 247 |
===== Creating the configuration file |
| 248 |
|
| 249 |
OCFS2 uses a central configuration file, `/etc/ocfs2/cluster.conf`. |
| 250 |
|
| 251 |
When creating your OCFS2 cluster, be sure to add both your hosts to |
| 252 |
the cluster configuration. The default port (7777) is usually an |
| 253 |
acceptable choice for cluster interconnect communications. If you |
| 254 |
choose any other port number, be sure to choose one that does not |
| 255 |
clash with an existing port used by DRBD (or any other configured |
| 256 |
TCP/IP). |
| 257 |
|
| 258 |
If you feel less than comfortable editing the `cluster.conf` file |
| 259 |
directly, you may also use the `ocfs2console` graphical configuration |
| 260 |
utility which is usually more convenient. Regardless of the approach |
| 261 |
you selected, your `/etc/ocfs2/cluster.conf` file contents should look |
| 262 |
roughly like this: |
| 263 |
|
| 264 |
[source,drbd] |
| 265 |
---------------------------- |
| 266 |
node: |
| 267 |
ip_port = 7777 |
| 268 |
ip_address = 10.1.1.31 |
| 269 |
number = 0 |
| 270 |
name = alice |
| 271 |
cluster = ocfs2 |
| 272 |
|
| 273 |
node: |
| 274 |
ip_port = 7777 |
| 275 |
ip_address = 10.1.1.32 |
| 276 |
number = 1 |
| 277 |
name = bob |
| 278 |
cluster = ocfs2 |
| 279 |
|
| 280 |
cluster: |
| 281 |
node_count = 2 |
| 282 |
name = ocfs2 |
| 283 |
---------------------------- |
| 284 |
|
| 285 |
|
| 286 |
When you have configured you cluster configuration, use `scp` to |
| 287 |
distribute the configuration to both nodes in the cluster. |
| 288 |
|
| 289 |
[[s-configure-o2cb-driver]] |
| 290 |
===== Configuring the O2CB driver |
| 291 |
|
| 292 |
[[s-suse_linux_enterprise_systems]] |
| 293 |
====== SUSE Linux Enterprise systems |
| 294 |
|
| 295 |
On SLES, you may utilize the `configure` option of the `o2cb` init |
| 296 |
script: |
| 297 |
|
| 298 |
---------------------------- |
| 299 |
# /etc/init.d/o2cb configure |
| 300 |
Configuring the O2CB driver. |
| 301 |
|
| 302 |
This will configure the on-boot properties of the O2CB driver. |
| 303 |
The following questions will determine whether the driver is loaded on |
| 304 |
boot. The current values will be shown in brackets ('[]'). Hitting |
| 305 |
<ENTER> without typing an answer will keep that current value. Ctrl-C |
| 306 |
will abort. |
| 307 |
|
| 308 |
Load O2CB driver on boot (y/n) [y]: |
| 309 |
Cluster to start on boot (Enter "none" to clear) [ocfs2]: |
| 310 |
Specify heartbeat dead threshold (>=7) [31]: |
| 311 |
Specify network idle timeout in ms (>=5000) [30000]: |
| 312 |
Specify network keepalive delay in ms (>=1000) [2000]: |
| 313 |
Specify network reconnect delay in ms (>=2000) [2000]: |
| 314 |
Use user-space driven heartbeat? (y/n) [n]: |
| 315 |
Writing O2CB configuration: OK |
| 316 |
Loading module "configfs": OK |
| 317 |
Mounting configfs filesystem at /sys/kernel/config: OK |
| 318 |
Loading module "ocfs2_nodemanager": OK |
| 319 |
Loading module "ocfs2_dlm": OK |
| 320 |
Loading module "ocfs2_dlmfs": OK |
| 321 |
Mounting ocfs2_dlmfs filesystem at /dlm: OK |
| 322 |
Starting O2CB cluster ocfs2: OK |
| 323 |
---------------------------- |
| 324 |
|
| 325 |
[[s-_debian_gnu_linux_systems]] |
| 326 |
====== .Debian GNU/Linux systems |
| 327 |
On Debian, the `configure` option to `/etc/init.d/o2cb` is not |
| 328 |
available. Instead, reconfigure the `ocfs2-tools` package to enable the |
| 329 |
driver: |
| 330 |
|
| 331 |
---------------------------- |
| 332 |
# dpkg-reconfigure -p medium -f readline ocfs2-tools |
| 333 |
Configuring ocfs2-tools |
| 334 |
Would you like to start an OCFS2 cluster (O2CB) at boot time? yes |
| 335 |
Name of the cluster to start at boot time: ocfs2 |
| 336 |
The O2CB heartbeat threshold sets up the maximum time in seconds that a node |
| 337 |
awaits for an I/O operation. After it, the node "fences" itself, and you will |
| 338 |
probably see a crash. |
| 339 |
|
| 340 |
It is calculated as the result of: (threshold - 1) x 2. |
| 341 |
|
| 342 |
Its default value is 31 (60 seconds). |
| 343 |
|
| 344 |
Raise it if you have slow disks and/or crashes with kernel messages like: |
| 345 |
|
| 346 |
o2hb_write_timeout: 164 ERROR: heartbeat write timeout to device XXXX after NNNN |
| 347 |
milliseconds |
| 348 |
O2CB Heartbeat threshold: `31` |
| 349 |
Loading filesystem "configfs": OK |
| 350 |
Mounting configfs filesystem at /sys/kernel/config: OK |
| 351 |
Loading stack plugin "o2cb": OK |
| 352 |
Loading filesystem "ocfs2_dlmfs": OK |
| 353 |
Mounting ocfs2_dlmfs filesystem at /dlm: OK |
| 354 |
Setting cluster stack "o2cb": OK |
| 355 |
Starting O2CB cluster ocfs2: OK |
| 356 |
---------------------------- |
| 357 |
|
| 358 |
[[s-ocfs2-use]] |
| 359 |
==== Using your OCFS2 filesystem |
| 360 |
|
| 361 |
When you have completed cluster configuration and created your file |
| 362 |
system, you may mount it as any other file system: |
| 363 |
---------------------------- |
| 364 |
# mount -t ocfs2 /dev/drbd0 /shared |
| 365 |
---------------------------- |
| 366 |
|
| 367 |
Your kernel log (accessible by issuing the command `dmesg`) should |
| 368 |
then contain a line similar to this one: |
| 369 |
|
| 370 |
[source,drbd] |
| 371 |
---------------------------- |
| 372 |
ocfs2: Mounting device (147,0) on (node 0, slot 0) with ordered data mode. |
| 373 |
---------------------------- |
| 374 |
|
| 375 |
From that point forward, you should be able to simultaneously mount |
| 376 |
your OCFS2 filesystem on both your nodes, in read/write mode. |