Debian Codegraph @

doc/html/_sources/imap/concepts/deployment/storage.rst.txt

raw
1 :tocdepth: 2
2
3 .. |product| replace:: Cyrus IMAP
4
5 .. _imap-deployment-storage:
6
7 ======================
8 Storage Considerations
9 ======================
10
11 Storage considerations are a complex matter, as the various options
12 provide or restrict one's ability to adjust the necessary parameters as
13 the need arises. It is foremost a challenge to clearly articulate and
14 prioritize the criteria for storage, and map the theory on to a
15 practical implementation design.
16
17 This article intends to provide information and outline details, and
18 sometimes opinions and recommendations, but it is not a guide to
19 providing you with the storage solution that you want or require.
20
21 Generally, the most important considerations for storage include;
22
23 :ref:`imap-deployment-storage-redundancy`,
24
25 because nothing is as humiliating as losing all your data.
26
27 :ref:`imap-deployment-storage-availability`,
28
29 because nothing is more stressful than none of your data being
30 available.
31
32 :ref:`imap-deployment-storage-performance`,
33
34 because nothing is as annoying as waiting, followed by some more
35 waiting.
36
37 :ref:`imap-deployment-storage-scalability`,
38
39 because ``-ENOSPC`` is good only when it applies to your stomach.
40
41 :ref:`imap-deployment-storage-capacity`,
42
43 because your data must be available, backed up and archived.
44
45 :ref:`imap-deployment-storage-cost`,
46
47 because you can't buy a beer or feed a family with an empty wallet.
48
49 Storage is not a part of |product|, in that |product| does not ship
50 a particular storage solution as part of the product, and it has no
51 particular requirements for storage either.
52
53 As such, your SAN, NAS, local disk, local array of disks or network
54 share or even the flash drive of a Raspberry Pi could be used, although
55 the following considerations are important:
56
57 * The Cyrus IMAP spool is I/O intensive (large volumes of data are read
58 and get written).
59
60 * The Cyrus IMAP spool consists of many small files.
61
62 As such, we recommend you take into account;
63
64 * The available bandwidth between the IMAP server and the storage
65 provider, if at all on the network,
66
67 * The (network) protocol overhead, if any, should file-level read
68 and/or write locking be required or implied.
69
70 * Atomic file operations.
71
72 * Parallel access (such as shared mailboxes or multi-client
73 attendance).
74
75 General Notes on Storage
76 ========================
77
78 The aforementioned considerations
79 :ref:`imap-deployment-storage-redundancy`,
80 :ref:`imap-deployment-storage-availability`,
81 :ref:`imap-deployment-storage-performance`,
82 :ref:`imap-deployment-storage-scalability`,
83 :ref:`imap-deployment-storage-capacity` and
84 :ref:`imap-deployment-storage-cost`
85 are not all of them equally important -- not to all organizations, and
86 not to all requirements when the priorities are set out against the
87 implied cost of the supposed ideal solution.
88
89 They are also not mutually exclusive in that, for example, redundancy
90 may partly address some of the availability concerns -- depending on the
91 exact nature of the final deployment of course, and backup/recovery
92 capabilities in turn may partly address redundancy requirements. Neither
93 necessarily directly addresses availability concerns, however.
94
95 What is deemed acceptable is another culprit -- more often then not,
96 operational cost, familiarity of staff with a particular storage
97 solution, or flexibility of a storage solution (or lack thereof) may get
98 in the way of an otherwise appropriate storage solution.
99
100 We believe that provided a sufficient amount of accurate information,
101 however, you are able to make an informed choice, and that an informed
102 choice is always better than an ill-informed one.
103
104 .. _imap-deployment-storage-redundancy:
105
106 Redundancy
107 ==========
108
109 Storage redundancy is achieved through replication of data. It is
110 important to understand that, as a matter of design principle,
111 redundancy does not in and by itself provide increased availability.
112
113 How redundancy could increase availability depends on the exact
114 implementation, and the various options for practical implementation
115 each have their own set of implications for cases of failure and the
116 need to, under certain circumstances, failover and/or recover.
117
118 How redundancy is achieved in an "acceptable" manner is another subject
119 open to interpretation; it is sometimes deemed acceptable to create
120 backups daily, and therefore potentially accept the loss of up to one
121 day's worth of information from live spools -- which may or may not be
122 recoverable through different means. More commonly however is to not
123 settle for anything less than real-time replication of data.
124
125 While storage ultimately amounts to disks, it is important to understand
126 that a number of (virtual) devices, channels, links and interfaces exist
127 between an application operating data on disk [#]_, and the physical
128 sectors and blocks or cells of storage on that disk. In a way, this
129 number of layers can be compared with the `OSI model for networking`_ --
130 but it is not the same at all.
131
132 This section addresses the most commonly used levels at which
133 replication can be applied.
134
135 Storage Volume Level Replication
136 --------------------------------
137
138 When using the term :term:`storage volume level replication` we mean to
139 indicate the replication of :term:`disk volumes` as a whole. A
140 simplistic replication scenario of a data disk between two nodes could
141 look as follows:
142
143 .. graphviz::
144
145 digraph drbd {
146 rankdir = LR;
147 splines = true;
148 overlab = prism;
149
150 edge [color=gray50, fontname=Calibri, fontsize=11];
151 node [style=filled, shape=record, fontname=Calibri, fontsize=11];
152
153 subgraph cluster_master {
154 label = "Master";
155
156 color = "#BBFFBB";
157 fontname = Calibri;
158 rankdir = TB;
159 style = filled;
160
161 "OS Disk 0" [label="OS Disk",color="green"];
162 "Data Disk 0" [label="Data Disk",color="green"];
163 }
164
165 subgraph cluster_slave {
166 label = "Slave";
167
168 color = "#FFBBBB";
169 fontname = Calibri;
170 rankdir = TB;
171 style = filled;
172
173 "OS Disk 1" [label="OS Disk",color="green"];
174 "Data Disk 1" [label="Data Disk",color="red"];
175 }
176
177 "Data Disk 0" -> "Data Disk 1" [label="One-Way Replication"];
178 }
179
180 For a fully detailed picture of the internal structures, please see the
181 `DRBD`_ website, the canonical experts on this level of replication.
182
183 Normally storage-level replication occurs in such
184 fashion that it can be compared with a distributed version of a RAID-1
185 array. This incurs limitations that need to be evaluated carefully.
186
187 In a hardware RAID-1 array, storage is physically constrained to a
188 single node, and pairs of replicated disks are treated as one. In a
189 software RAID-1 array, it is the operating system's software RAID driver
190 that can (must) address the individual disks, but makes the array appear
191 as a single disk to all higher-level software. Here too, the disks are
192 physically constrained to one physical node.
193
194 In both cases, a *single point of control* exists with full and
195 exclusive access to the physical disk device(s), namely the interface
196 for *all higher-level software* to interact with the storage.
197
198 This is the underlying cause of the storage-level replication conundrum.
199
200 To illustrate the conundrum, we use a software RAID-1 array. The
201 individual disk volumes that make up the RAID-1 array are not hidden
202 from the rest of the operating system, but more importantly, direct
203 access to the underlying device is not prohibited. With an example pair
204 ``sda2`` and ``sdb2``, nothing prevents you from executing ``mkfs.ext4``
205 on ``/dev/sdb2`` thereby corrupting the array -- other than perhaps not
206 having the necessary administrative privileges.
207
208 To further illustrate, position one disk in the RAID-1 array on the
209 other side of a network (such as is a `DRBD`_ topology, as illustrated).
210 Since now two nodes participate in nurturing the mirrored volume, two
211 points of control exist -- each node controls the access to its local
212 disk device(s).
213
214 Participating nodes are **required** to successfully coordinate their
215 I/O with one another, which on the level of entire storage volumes is a
216 very impractical effort with high latency and enormous overhead, should
217 more than one node be allowed to access the replicated device [#]_.
218
219 It is therefore understood that, using storage level replication;
220
221 * Only one side of the mirrored volume can be active (master), and the
222 other side must remain passive (slave),
223
224 * The active and passive nodes therefore have a cluster solution
225 implemented to manage application's failover and management of the
226 change in replication topology (a slave becomes the I/O master, the
227 former master becomes the replication slave, and other slaves, if
228 any, learn about the new master to replicate from),
229
230 * Failover implementations include fencing, the STONITH principle,
231 ensuring no two nodes in parallel perform I/O on the same volume at
232 any given time.
233
234 .. WARNING::
235
236 Storage volume level replication does not protect against filesystem
237 or payload corruption -- the replication happily mirrors the
238 "faulty" bits as it is completely agnostic to the bits' meaning and
239 relevance.
240
241 .. WARNING::
242
243 For the reasons outlined in this section, storage volume level
244 replication has only a limited number of |product| deployment
245 scenarios for which it would be recommended -- such as *Disaster
246 Recovery Failover*.
247
248 .. _imap-deployment-storage-integrated-storage-protocol-level-replication:
249
250 Integrated Storage Protocol Level Replication
251 ---------------------------------------------
252
253 Integrated storage protocol level replication is a different approach to
254 making storage volumes redundant, applying the replication on a
255 different level.
256
257 Integrated storage protocol level replication isn't necessarily limited
258 to replication for the purposes of redundancy only, as it may already
259 include parallel access controls, distribution across multiple storage
260 nodes (each providing a part of the total storage volume available),
261 enabling the use of cheap commodity hardware to provide the individual
262 parts (called "bricks") that make up the larger volume.
263
264 Additional features may include the use of a geographically oriented set
265 of parameters for the calculation and assignment of replicated chunks of
266 data (ie. "brick replication topology").
267
268 .. graphviz::
269
270 digraph {
271 rankdir = TB;
272 splines = true;
273 overlab = prism;
274
275 edge [color=gray50, fontname=Calibri, fontsize=11];
276 node [style=filled, shape=record, fontname=Calibri, fontsize=11];
277
278 "Storage Client #1" -> "Storage Access Point" [dir=back,color=green];
279 "Storage Client #2" -> "Storage Access Point" [dir=back,color=green];
280 "Storage Client #3" -> "Storage Access Point" [dir=back,color=green];
281 "Storage Client #4" -> "Storage Access Point" [dir=back,color=green];
282
283 subgraph cluster_storage {
284 color = green;
285 label = "Distributed and/or Replicated Volume Manager w/ Integrated Distributed (File-) Locking";
286
287 "Storage Access Point" [shape=point,color=green];
288
289 "Brick #1" [color=green];
290 "Brick #2" [color=green];
291 "Brick #3" [color=green];
292 "Brick #4" [color=green];
293
294 "Storage Access Point" -> "Brick #1" [color=green];
295 "Storage Access Point" -> "Brick #2" [color=green];
296 "Storage Access Point" -> "Brick #3" [color=green];
297 "Storage Access Point" -> "Brick #4" [color=green];
298 }
299 }
300
301 Current implementations of this type of technology include `GlusterFS`_
302 and `Ceph`_. Put way too simplistically, both technologies apply very
303 smart ways of storing individual objects, sometimes with additional
304 facilities for certain object types. How they work exactly is far beyond
305 the scope of this document.
306
307 Both technologies however are considered more efficient for fewer,
308 larger objects, than they are for more, smaller objects. Both storage
309 solutions also tend to be more efficient at addressing individual
310 objects directly, rather than hierarchies of objects (for listing).
311
312 This is meant to indicate that while both solutions scale up to millions
313 of objects, they facilitate a particular **I/O pattern** better than the
314 I/O pattern typically associated with a large volume of messages in IMAP
315 spools. More frequent and very short-lived I/O against individual
316 objects in a filesystem mounted directly causes a significant amount of
317 overhead in negotiating the access to the objects across the storage
318 cluster [2]_.
319
320 Both technologies are perfectly suitable for large clusters with
321 relatively small filesystems (see `Filesystems: Smaller is Better`_)
322 if they are mounted directly from the storage clients. They are
323 particularly feasible if not too many parallel write operations to
324 individual objects (files) are likely to occur (think, for example, of
325 web application servers and (asset-)caching proxies).
326
327 Alternatively, fewer larger objects could be stored -- such as disk
328 images for a virtualization environment. The I/O patterns internal to
329 the virtual machine would remain the same, but the I/O pattern of the
330 storage client (the hypervisor) is the equivalent of a single
331 lock-and-open when the virtual machine starts.
332
333 It is therefore understood that, especially in deployments of a larger
334 scale, one should not mount a GlusterFS or CephFS filesystem directly
335 from within an IMAP server, as an individual IMAP mail spool consists of
336 many very small objects each individually addressed frequently, and in
337 short-lived I/O operations, and consider the use of these distributed
338 filesystems for a different level of object storage, such as disk images
339 for a virtualization environment:
340
341 .. graphviz::
342
343 digraph {
344 rankdir = TB;
345 splines = true;
346 overlab = prism;
347
348 edge [color=gray50, fontname=Calibri, fontsize=11];
349 node [style=filled, shape=record, fontname=Calibri, fontsize=11];
350
351 subgraph cluster_guests {
352 label = "Guest Nodes";
353
354 "Guest #1";
355 "Guest #2";
356 "Guest #3";
357 }
358
359 subgraph cluster_hypervisors {
360 label = "Virtualization Platform";
361
362 "Hypervisor #1";
363 "Hypervisor #2";
364 }
365
366 subgraph cluster_storage {
367 color = green;
368 label = "Distributed and/or Replicated Volume
369 Manager w/ Integrated Distributed (File-) Locking";
370
371 subgraph cluster_replbricks1 {
372 label = "Replicated Bricks";
373
374 "Brick #1" [color=green];
375 "Brick #3" [color=green];
376 }
377
378 subgraph cluster_replbricks2 {
379 label = "Replicated Bricks";
380
381 "Brick #2" [color=green];
382 "Brick #4" [color=green];
383 }
384
385 }
386
387 "Guest #1" -> "Hypervisor #1" [dir=both,color=green];
388 "Guest #2" -> "Hypervisor #1" [dir=both,color=green];
389 "Guest #3" -> "Hypervisor #2" [dir=both,color=green];
390
391 "Hypervisor #1" -> "Brick #4" [dir=both,label="Guest #1"];
392 "Hypervisor #1" -> "Brick #3" [dir=both,label="Guest #2"];
393 "Hypervisor #2" -> "Brick #3" [dir=both,label="Guest #3"];
394 }
395
396 In this illustration, *Hypervisor #1* and *Hypervisor #2* are storage
397 clients, and replicated bricks hold the disk images of each guest.
398
399 Each hypervisor can, in parallel, perform I/O against each individual
400 disk image, allowing (for example) both *Hypervisor #1* and
401 *Hypervisor #2* to run guests with disk images for which *Brick #3* has
402 been selected as the authoritative copy.
403
404 .. _deployment-application-replication:
405
406 Application Level Replication
407 -----------------------------
408
409 Yet another means to provide redundancy of data is to use application-
410 level replication where available.
411
412 Famous examples include database server replication, where one or more
413 MySQL masters are used for write operations, and one or more MySQL
414 slaves are used for read operations, and LDAP replication.
415
416 Cyrus IMAP can also replicate its mail spools to other systems, such
417 that multiple backends hold the payload served to your users.
418
419 Shared Storage (Generic)
420 ------------------------
421
422 Contrary to popular belief, all shared storage -- NFS, iSCSI and FC
423 alike -- are **not** storage devices. They are *network protocols* for
424 which the application just so happens to be storage -- with perhaps the
425 exception to the rule being Fiber-Channel not strictly cohering to the
426 `OSI model for networking`_, although its own 5-layer model equates.
427
428 iSCSI and Fiber-Channel LUNs however are *mapped* to storage devices by
429 your favorite operating system's drivers for each technology, or
430 possibly by a hardware device (an :term:`HBA`, or in iSCSI, an
431 *initiator*).
432
433 As such, use of these network protocols for which the purpose just so
434 happens to be storage does **not** provide redundancy.
435
436 It is imperative this is understood and equally well applied in planning
437 for storage infrastructure, or that your storage appliance vendor or
438 consultancy partner is trusted in their judgement.
439
440 Shared Storage (NFS)
441 --------------------
442
443 Use of the Networked File System (NFS) in and by itself does **not**
444 provide redundancy, although the underlying storage volume might be
445 replicated.
446
447 For a variety of reasons, the use of `NFS is considered harmful`_ and is
448 therefore, and for other reasons, most definitely not recommended for
449 |product| IMAP spool storage, or any other storage related to
450 functional components of |product| itself -- IMAP, LDAP, SQL, etc.
451
452 Most individual concerns can be addressed separately, and some should or
453 must already be resolved to address other potentially problematic areas
454 of a given infrastructure, regardless of the use of NFS.
455
456 A couple of concerns however only have *workarounds*, not solutions --
457 such as disabling locking -- and a number of concerns have no solution
458 at all.
459
460 One penalty to address is the inability for NFS mounted volumes to cache
461 I/O, known as in-memory buffer caching.
462
463 A technology called `FS Cache`_ can facilitate eliminating round-trip-
464 incurred network-latency, but is still a filesystem-backed solution
465 (for which filesystem the local kernel applies buffer caching), requires
466 yet another daemon, and introduces yet another layer of synchronicity to
467 be maintained -- aside from `other limitations <https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/fscachelimitnfs.html>`_.
468
469 An NFS-backed storage volume can still be used for fewer, larger files,
470 such as guest disk images.
471
472 Shared Storage (iSCSI or FC LUNs)
473 ---------------------------------
474
475 Both iSCSI LUNs and Fiber-Channel LUNs facilitate attaching a networked
476 block storage device as if it were a local disk (creating devices
477 similar to ``/dev/sd{a,b,c,d}`` etc.).
478
479 Since such a LUN is available over a "network" infrastructure, it may be
480 shared between multiple nodes but when it is, nodes need to coordinate
481 their I/O on some other level.
482
483 With an example case of hypervisors, either `Cluster LVM`_ [#]_ or
484 `GFS`_ [#]_ could be used to protect against corruption of the LUN.
485
486 ..
487 Shared Storage (Disk Device)
488 ----------------------------
489
490 .. include:: needs-work.txt
491
492
493 .. _imap-deployment-storage-availability:
494
495 Availability
496 ============
497
498 Availability of storage too can be achieved via multiple routes. In one
499 of the aforementioned technologies, replicated bricks both available
500 real-time and online, in a parallel read-write capacity, provided high-
501 availability through redundancy (see
502 :ref:`imap-deployment-storage-integrated-storage-protocol-level-replication`).
503
504 An existing chunk of storage you might have is likely backed by a level
505 of RAID, with redundancy through mirroring individual disk volumes
506 and/or the inline calculation of parity, and perhaps also some spare
507 disks to replace those that are kicked or fall out of line.
508
509 Further features might include battery-backed I/O controllers, redundant
510 power supplies connected to different power groups, a further UPS and
511 a diesel generator (you start up once a month, right?).
512
513 The availability features of a data center are beyond the scope of this
514 document, but when we speak of availability with regards to storage, we
515 intend to speak of immediate, instant, online availability with
516 automated failover (such as the RAID array) -- and more prominently,
517 without interruption.
518
519 Multipath
520 ---------
521
522 Multipath is an enhancement technique in which multiple paths that are
523 available to the storage can be balanced, shaped and failed over
524 automatically. Imagine the following networking diagram:
525
526 .. graphviz::
527
528 digraph {
529 rankdir = TB;
530 splines = true;
531 overlab = prism;
532
533 edge [color=gray50, fontname=Calibri, fontsize=11];
534 node [style=filled, shape=record, fontname=Calibri, fontsize=11];
535
536 "Node";
537
538 "Switch #1"; "Switch #2";
539
540 "Canister #1"; "Canister #2";
541
542 "iSCSI Target #1", "iSCSI Target #2";
543
544 "Node" -> "Switch #1" [dir=none]
545 "Node" -> "Switch #2" [dir=none];
546
547 "Switch #1" -> "Canister #1" [dir=none];
548 "Switch #1" -> "Canister #2" [dir=none];
549
550 "Switch #2" -> "Canister #1" [dir=none];
551 "Switch #2" -> "Canister #2" [dir=none];
552
553 "Canister #1" -> "iSCSI Target #1" [dir=none];
554 "Canister #1" -> "iSCSI Target #2" [dir=none];
555
556 "Canister #2" -> "iSCSI Target #1" [dir=none];
557 "Canister #2" -> "iSCSI Target #2" [dir=none];
558 }
559
560 The *null* situation is depicted in the previous wiring diagram. When
561 multipath kicks in, primary vs. secondary paths will be chosen for each
562 individual target (that is unique). However, the system maintains a list
563 of potential paths, and continuously monitors all paths for their
564 viability.
565
566 In the example, for *Node* attaching to *iSCSI Target #1* results in up
567 to 4 paths to *iSCSI Target #1* -- *4* paths, not *8*, because the
568 networking of *Switch #1* and *Switch #2* is not considered a path with
569 iSCSI -- *two nodes* and *two send targets each*.
570
571 Multipath chooses one path to the available storage:
572
573 .. graphviz::
574
575 digraph {
576 rankdir = TB;
577 splines = true;
578 overlab = prism;
579
580 edge [color=gray50, fontname=Calibri, fontsize=11];
581 node [style=filled, shape=record, fontname=Calibri, fontsize=11];
582
583 "Node";
584
585 "Switch #1" [color=green];
586 "Switch #2";
587
588 "Canister #1";
589 "Canister #2" [color=green];
590
591 "iSCSI Target #1" [color=green];
592 "iSCSI Target #2";
593
594 "Node" -> "Switch #1" [dir=none,color=green]
595 "Node" -> "Switch #2" [dir=none];
596
597 "Switch #1" -> "Canister #1" [dir=none];
598 "Switch #1" -> "Canister #2" [dir=none,color=green];
599
600 "Switch #2" -> "Canister #1" [dir=none];
601 "Switch #2" -> "Canister #2" [dir=none];
602
603 "Canister #1" -> "iSCSI Target #1" [dir=none];
604 "Canister #1" -> "iSCSI Target #2" [dir=none];
605
606 "Canister #2" -> "iSCSI Target #1" [dir=none,color=green];
607 "Canister #2" -> "iSCSI Target #2" [dir=none];
608 }
609
610 Should one port, bridge, controller, switch or cable fail, then the I/O
611 can fall back on to any of the remaining available paths.
612
613 As per the example, this might mean the following (with *Canister #2*
614 failing):
615
616 .. graphviz::
617
618 digraph {
619 rankdir = TB;
620 splines = true;
621 overlab = prism;
622
623 edge [color=gray50, fontname=Calibri, fontsize=11];
624 node [style=filled, shape=record, fontname=Calibri, fontsize=11];
625
626 "Node";
627
628 "Switch #1" [color=green];
629 "Switch #2";
630
631 "Canister #1" [color=green];
632 "Canister #2" [color=red];
633
634 "iSCSI Target #1" [color=green];
635 "iSCSI Target #2";
636
637 "Node" -> "Switch #1" [dir=none,color=green]
638 "Node" -> "Switch #2" [dir=none];
639
640 "Switch #1" -> "Canister #1" [dir=none,color=green];
641 "Switch #1" -> "Canister #2" [dir=none,color=red];
642
643 "Switch #2" -> "Canister #1" [dir=none];
644 "Switch #2" -> "Canister #2" [dir=none];
645
646 "Canister #1" -> "iSCSI Target #1" [dir=none,color=green];
647 "Canister #1" -> "iSCSI Target #2" [dir=none];
648
649 "Canister #2" -> "iSCSI Target #1" [dir=none,color=red];
650 "Canister #2" -> "iSCSI Target #2" [dir=none];
651 }
652
653 .. _imap-deployment-storage-performance:
654
655 Performance
656 ===========
657
658 Storage Tiering
659 ---------------
660
661 Storage tiering includes the combination of different types of storage
662 or storage volumes with different performance expectations within the
663 infrastructure, so that a larger volume of slower, cheaper storage can
664 be used for items that are not used that much, and/or are not that
665 important for day-to-day operations, while a smaller volume of faster,
666 more expensive storage can be used for items that are frequently
667 accessed and have significant importance to everyday use.
668
669 The |product| administrator guide has a section on using
670 :ref:`admin-tweaking-cyrus-imapd-storage-tiering` to tweak Cyrus IMAP
671 performance, to illustrate various opportunities to make optimal use of
672 your storage.
673
674 As a general rule of thumb, you could divide
675 :term:`operating system disks` and :term:`payload disks`; the operating
676 system disk could hold your base installation of a node, including
677 everything in the root (``/``) filesystem, while your payload disk(s)
678 hold the files and directories that contain the system's service(s)
679 payload (such as ``/var/lib/mysql/``, ``/var/spool/cyrus/``,
680 ``/var/lib/imap/``, ``/var/lib/dirsrv/``, etc.).
681
682 Distributing what is and what is not frequently used may be a cumbersome
683 task for administrators. Some storage vendor's appliances offer
684 automated storage tiering, where some disks in the appliance are SSDs,
685 while others are SATA or SAS HDDs, and the appliance itself tiers the
686 storage.
687
688 A similar solution is available to Linux nodes, through `dm-cache`_,
689 provided they run a recent kernel.
690
691 Disk Buffering
692 --------------
693
694 Reading from a disk is considered very, very slow when compared to
695 accessing a node's (real) memory. While dependent on the particular I/O
696 pattern of an application, it is not uncommon at all for an application
697 to read the same part of a disk volume several times during a relatively
698 short period of time.
699
700 In |product|, for example, while a user is logged in, a mail
701 folder's :file:`cyrus.index` is read more frequently than it is
702 written to -- such as when refreshing the folder view, when opening a
703 message in the folder, when replying to a message, etc.
704
705 It is important to appreciate the impact of `memory-based buffer cache`_
706 for this type of I/O on the overall performance of the environment.
707
708 Should no (local) memory-based buffer cache be available, because for
709 example you are using a network filesystem (NFS, GlusterFS, etc.), then
710 it is extremely important to appreciate the consequences in terms of the
711 performance expectations.
712
713 Readahead
714 ---------
715
716 Reading ahead is a feature in which -- in a future-predicting,
717 anticipatory fashion -- a chunk of storage is read in addition to the
718 chunk of storage actually being requested.
719
720 A common application of read-ahead is to record all files accessed
721 during the boot process of a node, such that later boot sequences can
722 read files from disk, and insert them in to the
723 `memory-based buffer cache`_ ahead of software actually issuing the call
724 to read the file. The file's contents can now be reproduced from the
725 faster (real) memory rather then from the slow disk.
726
727 Readahead generally does not matter for small files, unless read
728 operations work on a collective of aggregate message files. It does
729 however matter for attached devices on infrastructural components such
730 as hypervisors, where entire block devices (for the guest) are the files
731 or block devices being read.
732
733 The ideal setting for readahead depends on a variety of factors and can
734 usually only be established by monitoring an environment and tweaking
735 the setting (followed by some more monitoring).
736
737 ..
738 Writeback
739 ---------
740
741 `Linux Page Cache`_
742
743 .. _imap-deployment-storage-scalability:
744
745 Scalability
746 ===========
747
748 When originally planning for storage capacity, a few things are to be
749 taken in to account. We'll point these out and address them later in
750 this section.
751
752 Generically speaking, when storage capacity is planned for initially,
753 a certain period of time is used to establish how much storage might be
754 required (for that duration).
755
756 However, let's suppose regulatory provisions dictate a period of 10
757 years of business communications need to be preserved. How does one
758 accurately predict the volume of communications over the next 10 years?
759
760 Let's suppose your organization is in flux, expanding or contracting as
761 businesses do at times, or budget cuts and unexpected extra tasks to
762 your organization might incur. Or when the organization takes over or
763 otherwise incorporates another.
764
765 Today's storage coming with a certain price-tag, and tomorrow's with a
766 different one, it can be an interesting exercise to plan for storage to
767 grow organically as needed, rather than make large investments to provide
768 capacity that may only be used years from today, or not be used at all,
769 or turn out to still not be sufficient.
770
771 One may also consider planning for the future expansion of the storage
772 solution chosen today, possibly including significant changes in
773 requirements (larger mailboxes).
774
775 Data Retention
776 --------------
777
778 |product| by default does not delete IMAP spool contents from the
779 filesystem for a period of 69 days.
780
781 This means that when a 100 users each have 1 GB of quota, the actual
782 data footprint might grow way beyond 100 GB on disk.
783
784 Depending on the nature of how you use |product|, a reasonable
785 expectation can be formulated and used for calculating the likely amount
786 of disk space used in addition to the content that continues to count
787 towards quota.
788
789 For example, if a large amount of message traffic ends up in a shared
790 folder that many users read messages from and respond to (such as might
791 be the case for an [email protected] email address), then around triple
792 the amount of traffic per month will continue to be stored on disk, plus
793 what you decide is still current and not deleted by users (the "live
794 size").
795
796 Shared Folders
797 --------------
798
799 Shared folders (primarily those to which mail is delivered) do not, by
800 default, have any quota on them. They are also not purged by default. As
801 such, they could grow infinitely (until disks run out of space).
802
803 A busy mailing list used for human communications, such as
804 [email protected], can be expected to grow to as much as 1
805 GB of data foot print on disk over a period of 3 years -- at a message
806 rate of less than ~100 a day and without purging.
807
808 A mailing list with automated messages generated from applications, such
809 as [email protected], which is notified of all ticket changes for KDE's
810 upstream Bugzilla, can be expected to grow to up to 3.5 GB over the same
811 period -- at a message rate of ~300 per day and without purging.
812
813 User's Groupware Folders
814 ------------------------
815
816 Users tend not to clean up their calendars, removing old appointments
817 that have no bearing on today's views/operations any longer. They do
818 count towards a user's quota.
819
820 .. _imap-deployment-storage-capacity:
821
822 Capacity
823 ========
824
825 Regardless of the volume of storage in total, this section relates to
826 the volume of storage allocated to any one singular specific purpose in
827 |product|, and capacity planning in light of that (not the layer
828 providing the storage).
829
830 Archiving and e-Discovery notwithstanding, the largest chunks of data
831 volume you are going to find in |product| are the live IMAP
832 spools.
833
834 Let each individual IMAP spool be considered a volume, or part of a
835 volume if you feel inclined to share volumes across Cyrus IMAP backend
836 instances. Regardless, you need a filesystem **somewhere** (even if the
837 bricks building the volume are distributed) -- the recommended
838 restrictions you should put forth to the individual chunks of storage
839 lay therein.
840
841 Saturating the argument to make a point, imagine, if you will, a million
842 users with one gigabyte of data each. Just the original, minimal data
843 footprint is now around and about one petabyte.
844
845 Performing a filesystem check (:command:`fsck.ext4` comes to mind) on a
846 single one petabyte volume will be prohibitively expensive simply
847 considering the duration of the command to complete execution, let alone
848 successful execution, for your **million** users will not have access to
849 their data while the command has not finished -- again, let alone it
850 finished successfully.
851
852 **Solely therefore** would you require a second copy of the groupware
853 payload, now establishing a minimal data footprint to **two** petabyte.
854
855 .. NOTE::
856
857 Also note that the two replicas of one petabyte would, if the
858 replication occurs at the storage volume level, run the risk of
859 corrupting both replicas' filesystems.
860
861 Your requirements for data redundancy aside, filesystem checks needing
862 to be performed at least regularly [#]_, if not for the level of
863 likelihood they need to happen because actual recovery is required,
864 should be restricted to a volume of data that enables the check to
865 complete in a timely fashion, and possibly (when no data redundancy is
866 implemented) even within a timeframe you feel comfortable you can hold
867 off your users/customers while they have no access to their data.
868
869 For filesystem checks to need to happen regularly, is not to say that
870 such filesystem checks require the node to be taken offline. Should you
871 use Logical Volume Management (LVM) for example, and not allocate 100%
872 of the volume group to the logical volume that holds the IMAP spool,
873 than intermediate filesystem checks can be executed on a snapshot of
874 said logical volume instead, and while the node remains online, to give
875 you a generic impression of the filesystem's health. Given this
876 information, you can schedule a service window should you feel the need
877 to check the actual filesystem.
878
879 A good article on filesystems, the corruption of data and their causes
880 and mitigation strategies has been written up by `LWN`_,
881 `The 2006 Linux Filesystem Workshop`_. This article also explains what
882 it is a filesystem check actually does, and why it is usually configured
883 to be ran after either a certain amount of delay or number of mounts.
884
885 ..
886 Using Bricks to Build a Larger Volume
887 -------------------------------------
888
889 500 bricks of 4 TB each would build a two petabyte storage volume with
890 enough space for redundant storage, where individual bricks can be taken
891 offline, its filesystem can be checked, and the brick can be brought
892 back online, without interrupting data availability.
893
894 Distributing Payload
895 --------------------
896
897 250 systems of 4 TB each would amount to one petabyte of total storage
898 volume,
899
900 .. _imap-deployment-storage-cost:
901
902 Cost
903 ====
904
905 When cost is of no concern, multiple vendors of storage solutions will
906 tell you precisely what you need to hear -- I think we've all been
907 there.
908
909 When cost is a concern, however, cheaper disks are often slower, fail
910 faster, and sometimes also do not provide the
911 :ref:`imap-deployment-storage-capacity` desired.
912
913 On the other hand, stuffing many consumer-grade SATA III disks in to
914 some commodity hardware likely raises run-time costs -- energy.
915
916 However, a chassis of a storage solution usually comes at a higher
917 price point, and therefore expands capacity with relatively large
918 chunks, which may not be what you require at that moment.
919
920 .. rubric:: Footnotes
921
922 .. [#]
923
924 Applications may also operate on data not stored on disk at all,
925 which is another common avenue potentially resulting in loss of data
926 -- or *corruption*, which is merely a type of data-loss.
927
928 .. [#]
929
930 With read operations, the other node(s) must be blocked from
931 writing, and with write operations, the other node(s) must be
932 blocked from reading and writing.
933
934 .. [#]
935
936 When using ClusterLVM, you would use logical volumes as disks for
937 your guests.
938
939 .. [#]
940
941 When using GFS, you would mount the GFS filesystem partition on each
942 hypervisor and use disk image files.
943
944 .. [#]
945
946 Execute filesystem checks regularly to increase your level of
947 confidence, that should emergency repairs need to be performed, you
948 are able to recover swiftly.
949
950 The :term:`MTBF` of a stable filesystem has most often been subject
951 to the failure of the underlying disk, with the filesystem unable to
952 recover (in time) from the underlying disk failing (partly).
953
954
955 .. _DRBD: http://www.drbd.org/
956 .. _OSI model for networking: http://en.wikipedia.org/wiki/OSI_model
957 .. _LWN: http://lwn.net
958 .. _The 2006 Linux Filesystem Workshop: http://lwn.net/Articles/190222/
959 .. _GlusterFS: http://www.glusterfs.org
960 .. _Ceph: http://ceph.com
961 .. _NFS is considered harmful: http://www.time-travellers.org/shane/papers/NFS_considered_harmful.html
962 .. _Filesystems\: Smaller is Better: https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Global_File_System_2/ch-considerations.html#s2-fssize-gfs2
963
964 .. _Linux Page Cache: http://www.westnet.com/~gsmith/content/linux-pdflush.htm
965 .. _FS Cache: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/ch-fscache.html
966 .. _Dovecot Oy: http://www.dovecot.fi
967 .. _memory-based buffer cache: http://www.tldp.org/LDP/sag/html/buffer-cache.html
968 .. _GFS: http://en.wikipedia.org/wiki/GFS2
969 .. _Cluster LVM: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Volume_Manager_Administration/LVM_Cluster_Overview.html
970 .. _dm-cache: http://en.wikipedia.org/wiki/Dm-cache
971 .. _Kolab Systems AG: https://kolabsys.com