Debian Codegraph @

doc/html/_sources/imap/reference/faqs/o-sharedfilesystem-gpfs.rst.txt

raw
1 Shared File Systems GPFS for high availability
2 ----------------------------------------------
3
4 Stephen L. Ulmer ([email protected]) actually has used Cyrus with GPFS, and
5 writes:
6
7 FOR THE LOVE OF GOD, RUN AWAY!
8
9 We had our Cyrus message store on GPFS for just about a year. I've been
10 a Unix systems administrator for almost 15 years; It was the worst
11 single judgement of my professional career. Period.
12
13 During the 18 months when we had GPFS deployed, my unit had TWO crit
14 sits and uncovered over 30 bugs in the GPFS software alone (not counting
15 stuff we found in VSD, AIX, et cetera). The situation ended with the
16 GPFS architect suggesting that we do something else. He's a great guy,
17 and he helped us many times, but the product just doesn't do what we
18 wanted.
19
20 GPFS is the successor to the Multi-Media Filesystem, which was used in
21 IBM's Videocharger product. It's excellent at streaming small numbers of
22 large files (like, say, movies). It's horrible when you get above a few
23 hundred-thousand files, as the systems can't possibly have enough memory
24 to keep track of the filesystem meta-data.
25
26 Our Cyrus installation has about 80K users, close to 1TB of disk, and
27 many millions of files. Just the number of files alone would be enough
28 to kill the idea of using GPFS.
29
30 Cyrus makes pretty extensive use of mmap().
31 While GPFS implements mmap(), the GPFS architect had some words about
32 the way certain operations are accomplished in Cyrus. I think there are
33 (or used to be) places where an mmap'd file is opened for write with
34 another file handle (or from another process). GPFS doesn't handle this
35 well. This technique works accidentally on non-clustered filesystems
36 because AIX (also) mmap's things for you behind your back (in addition
37 to whatever you do) and then serializes all of the access to regions of
38 those files. That's really the only reason why Cyrus works on JFS.
39
40 Also note that the other groups/developers within IBM (especially the
41 group that does the automounter) have their collective heads up their
42 ass with respect to supporting "after market" filesystems on AIX. After
43 two freakin' years of PMRs they still couldn't figure out how to make
44 autofs interact predictably with locally-mounted GPFSs. I constantly had
45 to employ work-arounds in my automounter maps.
46
47 If you just want failover, then use the HACMP product to deal with the
48 failover scenario. If you need to scale beyond one system image, try out
49 a Cyrus Murder. That's what we're using now, it works great.
50
51 Note that in the Murder scenario, you can still use HACMP to have one of
52 your back-ends take over for another if it fails. You just have to
53 carefully craft your cyrus.conf files to only bind to a single IP
54 address, so that you can run two separate instances of Cyrus on the same
55 machine during the failover.
56
57 I will be happy to discuss our particular situation, implementation
58 choices and details with you if you'd like to contact me out-of-band.
59
60
61
62 We're currently running our Murder on::
63
64 2 x p630 [backends]
65 4 x 1.4GHz Power4+ CPU
66 8GB Real Memory
67
68 4 x p615 [frontends]
69 2 x 1.2GHz Pwer4+ CPU
70 4GB Real Memory
71
72 The frontends are also the web servers for our Virtual Hosting cluster. We're running version 2.1.x of Cyrus. Now that 2.2.x is stable we'll upgrade, but you can imagine that it'll take some planning.
73
74 In short, we don't recommend it. If you want to do it, it may possibly work but you may also lose your email or have corrupted cyrus.* files. You can look at the mailing list archives for more information.
75
76 There are several critical things that must be supported for this to work:
77
78 * file locking
79 * mmap()
80 * writing to an mmap()'d file with the write() syscall
81
82 In general, this is bad news. Don't do it. Really. About the closest you can currently get is having a "warm spare" that takes over the IP of the affected backend, and have a SAN between the two systems.
83
84 :ref:`Cyrus Murder <murder>` may be a "good enough" solution in some environments, as a way to partition failures and spread load across many machines. Combined with a warm-spare approach this can be a good way to achieve highly available systems (though not ones which are fully redundant)