Debian Codegraph @

UG9/en/ocfs2.adoc

raw
1 [[ch-ocfs2]]
2 == Using OCFS2 with DRBD
3
4 indexterm:[OCFS2]indexterm:[Oracle Cluster File System]This chapter
5 outlines the steps necessary to set up a DRBD resource as a block
6 device holding a shared Oracle Cluster File System, version 2 (OCFS2).
7
8
9 [IMPORTANT]
10 ===============================
11 All cluster file systems _require_ fencing - not only via the DRBD
12 resource, but STONITH! A faulty member _must_ be killed.
13
14 You'll want these settings:
15
16 net {
17 fencing resource-and-stonith;
18 }
19 handlers {
20 # Make sure the other node is confirmed
21 # dead after this!
22 outdate-peer "/sbin/kill-other-node.sh";
23 }
24
25 There must be _no_ volatile caches!
26 You might take a few hints of the page at https://fedorahosted.org/cluster/wiki/DRBD_Cookbook,
27 although that's about GFS2, not OCFS2.
28 ===============================
29
30
31
32 [[s-ocfs2-primer]]
33 === OCFS2 primer
34
35 The Oracle Cluster File System, version 2 (OCFS2) is a concurrent
36 access shared storage file system developed by Oracle
37 Corporation. Unlike its predecessor OCFS, which was specifically
38 designed and only suitable for Oracle database payloads, OCFS2 is a
39 general-purpose filesystem that implements most POSIX semantics. The
40 most common use case for OCFS2 is arguably Oracle Real Application
41 Cluster (RAC), but OCFS2 may also be used for load-balanced NFS
42 clusters, for example.
43
44 Although originally designed for use with conventional shared storage
45 devices, OCFS2 is equally well suited to be deployed on
46 <<s-dual-primary-mode,dual-Primary DRBD>>. Applications reading from
47 the filesystem may benefit from reduced read latency due to the fact
48 that DRBD reads from and writes to local storage, as opposed to the
49 SAN devices OCFS2 otherwise normally runs on. In addition, DRBD adds
50 redundancy to OCFS2 by adding an additional copy to every filesystem
51 image, as opposed to just a single filesystem image that is merely
52 shared.
53
54 Like other shared cluster file systems such as <<ch-gfs,GFS>>, OCFS2
55 allows multiple nodes to access the same storage device, in read/write
56 mode, simultaneously without risking data corruption. It does so by
57 using a Distributed Lock Manager (DLM) which manages concurrent access
58 from cluster nodes. The DLM itself uses a virtual file system
59 (`ocfs2_dlmfs`) which is separate from the actual OCFS2 file systems
60 present on the system.
61
62 OCFS2 may either use an intrinsic cluster communication layer to
63 manage cluster membership and filesystem mount and unmount operation,
64 or alternatively defer those tasks to the
65 <<ch-pacemaker,Pacemaker>>cluster infrastructure.
66
67 OCFS2 is available in SUSE Linux Enterprise Server (where it is the
68 primarily supported shared cluster file system), CentOS, Debian
69 GNU/Linux, and Ubuntu Server Edition. Oracle also provides packages
70 for Red Hat Enterprise Linux (RHEL). This chapter assumes running
71 OCFS2 on a SUSE Linux Enterprise Server system.
72
73 [[s-ocfs2-create-resource]]
74 === Creating a DRBD resource suitable for OCFS2
75
76 Since OCFS2 is a shared cluster file system expecting concurrent
77 read/write storage access from all cluster nodes, any DRBD resource to
78 be used for storing a OCFS2 filesystem must be configured in
79 <<s-dual-primary-mode,dual-primary mode>>. Also, it is recommended to
80 use some of DRBD's
81 <<s-automatic-split-brain-recovery-configuration,features for
82 automatic recovery from split brain>>. To
83 do all this, include the following lines in the resource
84 configuration: indexterm:[drbd.conf]
85
86 [source,drbd]
87 ----------------------------
88 resource <resource> {
89 net {
90 # allow-two-primaries yes;
91 after-sb-0pri discard-zero-changes;
92 after-sb-1pri discard-secondary;
93 after-sb-2pri disconnect;
94 ...
95 }
96 ...
97 }
98 ----------------------------
99
100 [WARNING]
101 ===============================
102 By setting auto-recovery policies, you are effectively configuring automatic data-loss! Be sure you understand the implications.
103 ===============================
104
105
106 It is not recommended to set the `allow-two-primaries` option to `yes`
107 upon initial configuration. You should do so after the initial
108 resource synchronization has completed.
109
110 Once you have added these options to <<ch-configure,your
111 freshly-configured resource>>, you may <<s-first-time-up,initialize
112 your resource as you normally would>>. After you set the
113 indexterm:[drbd.conf]`allow-two-primaries` option to `yes` for this
114 resource, you will be able to <<s-switch-resource-roles,promote the
115 resource>>to the primary role on both nodes.
116
117 NOTE: With DRBD 9.1 it will be possible to have more than two nodes in the
118 _Primary_ role; with DRBD 9.0 you can only use two primaries, but more nodes
119 can be in _Secondary_ role to provide more redundancy.
120
121
122 [[s-ocfs2-create]]
123 === Creating an OCFS2 filesystem
124
125 Now, use OCFS2's `mkfs` implementation to create the file system:
126 ----------------------------
127 # mkfs -t ocfs2 -N 2 -L ocfs2_drbd0 /dev/drbd0
128 mkfs.ocfs2 1.4.0
129 Filesystem label=ocfs2_drbd0
130 Block size=1024 (bits=10)
131 Cluster size=4096 (bits=12)
132 Volume size=205586432 (50192 clusters) (200768 blocks)
133 7 cluster groups (tail covers 4112 clusters, rest cover 7680 clusters)
134 Journal size=4194304
135 Initial number of node slots: 2
136 Creating bitmaps: done
137 Initializing superblock: done
138 Writing system files: done
139 Writing superblock: done
140 Writing backup superblock: 0 block(s)
141 Formatting Journals: done
142 Writing lost+found: done
143 mkfs.ocfs2 successful
144 ----------------------------
145
146 This will create an OCFS2 file system with two node slots on
147 `/dev/drbd0`, and set the filesystem label to `ocfs2_drbd0`. You may
148 specify other options on `mkfs` invocation; please see the `mkfs.ocfs2`
149 system manual page for details.
150
151 [[s-ocfs2-pacemaker]]
152 === Pacemaker OCFS2 management
153
154 [[s-ocfs2-pacemaker-drbd]]
155 ==== Adding a Dual-Primary DRBD resource to Pacemaker
156
157 An existing <<s-ocfs2-create-resource,Dual-Primary DRBD resource>>may
158 be added to Pacemaker resource management with the following
159 `crm` configuration:
160
161 [source,drbd]
162 ----------------------------
163 primitive p_drbd_ocfs2 ocf:linbit:drbd \
164 params drbd_resource="ocfs2"
165 ms ms_drbd_ocfs2 p_drbd_ocfs2 \
166 meta master-max=2 clone-max=2 notify=true
167 ----------------------------
168
169 IMPORTANT: Note the `master-max=2` meta variable; it enables
170 dual-Master mode for a Pacemaker master/slave set. This requires that
171 `allow-two-primaries` is also set to `yes` in the DRBD
172 configuration. Otherwise, Pacemaker will flag a configuration error
173 during resource validation.
174
175 [[s-ocfs2-pacemaker-mgmtdaemons]]
176 ==== Adding OCFS2 management capability to Pacemaker
177
178 In order to manage OCFS2 and the kernel Distributed Lock Manager
179 (DLM), Pacemaker uses a total of three different resource agents:
180
181 * `ocf:pacemaker:controld` -- Pacemaker's interface to the DLM;
182
183 * `ocf:ocfs2:o2cb` -- Pacemaker's interface to OCFS2 cluster
184 management;
185
186 * `ocf:heartbeat:Filesystem` -- the generic filesystem management
187 resource agent which supports cluster file systems when configured
188 as a Pacemaker clone.
189
190 You may enable all nodes in a Pacemaker cluster for OCFS2 management
191 by creating a _cloned group_ of resources, with the following
192 `crm` configuration:
193
194 [source,drbd]
195 ----------------------------
196 primitive p_controld ocf:pacemaker:controld
197 primitive p_o2cb ocf:ocfs2:o2cb
198 group g_ocfs2mgmt p_controld p_o2cb
199 clone cl_ocfs2mgmt g_ocfs2mgmt meta interleave=true
200 ----------------------------
201
202 Once this configuration is committed, Pacemaker will start instances
203 of the `controld` and `o2cb` resource types on all nodes in the cluster.
204
205 [[s-ocfs2-pacemaker-fs]]
206 ==== Adding an OCFS2 filesystem to Pacemaker
207
208 Pacemaker manages OCFS2 filesystems using the conventional
209 `ocf:heartbeat:Filesystem` resource agent, albeit in clone mode. To
210 put an OCFS2 filesystem under Pacemaker management, use the following
211 `crm` configuration:
212
213 [source,drbd]
214 ----------------------------
215 primitive p_fs_ocfs2 ocf:heartbeat:Filesystem \
216 params device="/dev/drbd/by-res/ocfs2/0" directory="/srv/ocfs2" \
217 fstype="ocfs2" options="rw,noatime"
218 clone cl_fs_ocfs2 p_fs_ocfs2
219 ----------------------------
220
221 NOTE: This example assumes a single-volume resource.
222
223 [[s-ocfs2-pacemaker-constraints]]
224 ==== Adding required Pacemaker constraints to manage OCFS2 filesystems
225
226 In order to tie all OCFS2-related resources and clones together, add
227 the following constraints to your Pacemaker configuration:
228
229 [source,drbd]
230 ----------------------------
231 order o_ocfs2 ms_drbd_ocfs2:promote cl_ocfs2mgmt:start cl_fs_ocfs2:start
232 colocation c_ocfs2 cl_fs_ocfs2 cl_ocfs2mgmt ms_drbd_ocfs2:Master
233 ----------------------------
234
235 [[s-ocfs2-legacy]]
236 === Legacy OCFS2 management (without Pacemaker)
237
238 IMPORTANT: The information presented in this section applies to legacy
239 systems where OCFS2 DLM support is not available in Pacemaker. It is
240 preserved here for reference purposes only. New installations should
241 always use the <<s-ocfs2-pacemaker,Pacemaker>> approach.
242
243 [[s-ocfs2-enable]]
244 ==== Configuring your cluster to support OCFS2
245
246 [[s-ocfs2-create-cluster-conf]]
247 ===== Creating the configuration file
248
249 OCFS2 uses a central configuration file, `/etc/ocfs2/cluster.conf`.
250
251 When creating your OCFS2 cluster, be sure to add both your hosts to
252 the cluster configuration. The default port (7777) is usually an
253 acceptable choice for cluster interconnect communications. If you
254 choose any other port number, be sure to choose one that does not
255 clash with an existing port used by DRBD (or any other configured
256 TCP/IP).
257
258 If you feel less than comfortable editing the `cluster.conf` file
259 directly, you may also use the `ocfs2console` graphical configuration
260 utility which is usually more convenient. Regardless of the approach
261 you selected, your `/etc/ocfs2/cluster.conf` file contents should look
262 roughly like this:
263
264 [source,drbd]
265 ----------------------------
266 node:
267 ip_port = 7777
268 ip_address = 10.1.1.31
269 number = 0
270 name = alice
271 cluster = ocfs2
272
273 node:
274 ip_port = 7777
275 ip_address = 10.1.1.32
276 number = 1
277 name = bob
278 cluster = ocfs2
279
280 cluster:
281 node_count = 2
282 name = ocfs2
283 ----------------------------
284
285
286 When you have configured you cluster configuration, use `scp` to
287 distribute the configuration to both nodes in the cluster.
288
289 [[s-configure-o2cb-driver]]
290 ===== Configuring the O2CB driver
291
292 [[s-suse_linux_enterprise_systems]]
293 ====== SUSE Linux Enterprise systems
294
295 On SLES, you may utilize the `configure` option of the `o2cb` init
296 script:
297
298 ----------------------------
299 # /etc/init.d/o2cb configure
300 Configuring the O2CB driver.
301
302 This will configure the on-boot properties of the O2CB driver.
303 The following questions will determine whether the driver is loaded on
304 boot. The current values will be shown in brackets ('[]'). Hitting
305 <ENTER> without typing an answer will keep that current value. Ctrl-C
306 will abort.
307
308 Load O2CB driver on boot (y/n) [y]:
309 Cluster to start on boot (Enter "none" to clear) [ocfs2]:
310 Specify heartbeat dead threshold (>=7) [31]:
311 Specify network idle timeout in ms (>=5000) [30000]:
312 Specify network keepalive delay in ms (>=1000) [2000]:
313 Specify network reconnect delay in ms (>=2000) [2000]:
314 Use user-space driven heartbeat? (y/n) [n]:
315 Writing O2CB configuration: OK
316 Loading module "configfs": OK
317 Mounting configfs filesystem at /sys/kernel/config: OK
318 Loading module "ocfs2_nodemanager": OK
319 Loading module "ocfs2_dlm": OK
320 Loading module "ocfs2_dlmfs": OK
321 Mounting ocfs2_dlmfs filesystem at /dlm: OK
322 Starting O2CB cluster ocfs2: OK
323 ----------------------------
324
325 [[s-_debian_gnu_linux_systems]]
326 ====== .Debian GNU/Linux systems
327 On Debian, the `configure` option to `/etc/init.d/o2cb` is not
328 available. Instead, reconfigure the `ocfs2-tools` package to enable the
329 driver:
330
331 ----------------------------
332 # dpkg-reconfigure -p medium -f readline ocfs2-tools
333 Configuring ocfs2-tools
334 Would you like to start an OCFS2 cluster (O2CB) at boot time? yes
335 Name of the cluster to start at boot time: ocfs2
336 The O2CB heartbeat threshold sets up the maximum time in seconds that a node
337 awaits for an I/O operation. After it, the node "fences" itself, and you will
338 probably see a crash.
339
340 It is calculated as the result of: (threshold - 1) x 2.
341
342 Its default value is 31 (60 seconds).
343
344 Raise it if you have slow disks and/or crashes with kernel messages like:
345
346 o2hb_write_timeout: 164 ERROR: heartbeat write timeout to device XXXX after NNNN
347 milliseconds
348 O2CB Heartbeat threshold: `31`
349 Loading filesystem "configfs": OK
350 Mounting configfs filesystem at /sys/kernel/config: OK
351 Loading stack plugin "o2cb": OK
352 Loading filesystem "ocfs2_dlmfs": OK
353 Mounting ocfs2_dlmfs filesystem at /dlm: OK
354 Setting cluster stack "o2cb": OK
355 Starting O2CB cluster ocfs2: OK
356 ----------------------------
357
358 [[s-ocfs2-use]]
359 ==== Using your OCFS2 filesystem
360
361 When you have completed cluster configuration and created your file
362 system, you may mount it as any other file system:
363 ----------------------------
364 # mount -t ocfs2 /dev/drbd0 /shared
365 ----------------------------
366
367 Your kernel log (accessible by issuing the command `dmesg`) should
368 then contain a line similar to this one:
369
370 [source,drbd]
371 ----------------------------
372 ocfs2: Mounting device (147,0) on (node 0, slot 0) with ordered data mode.
373 ----------------------------
374
375 From that point forward, you should be able to simultaneously mount
376 your OCFS2 filesystem on both your nodes, in read/write mode.