Debian Codegraph @

doc/html/_sources/imap/concepts/features/mailbox-distribution.rst.txt

raw
1 :tocdepth: 2
2
3 .. _imap-features-mailbox-distribution:
4
5 ====================
6 Mailbox Distribution
7 ====================
8
9 Prior to Cyrus IMAP version 2.5.0, when creating a mailbox, should no
10 target partition have been specified, the mailbox is either created on:
11
12 * the configured ``defaultserver``, or
13
14 * the server that is found to have the most free disk space, and
15
16 * the configured ``defaultpartition``, or
17
18 * the most fitting partition if ``defaultpartition`` is not
19 configured.
20
21 New configuration options are available since Cyrus IMAP 2.5.0, that
22 allow more weighted and better balanced backend and partition
23 selection.
24
25 Partition Selection Mode
26 ========================
27
28 Among the partitions, how the most fitting one is selected depends on
29 the configured selection mode, using the ``partition_select_mode``
30 setting in :cyrusman:`imapd.conf(5)`.
31
32 .. rubric:: Available Selection Modes
33
34 .. sidebar:: Example Scenario for Selection Modes
35
36 Suppose that the partitions configured are:
37
38 +-----------+---------+--------+-----+
39 | Partition | Total | Free | % |
40 +===========+=========+========+=====+
41 | ``part1`` | 1000 GB | 400 GB | 40% |
42 +-----------+---------+--------+-----+
43 | ``part2`` | 1000 GB | 600 GB | 60% |
44 +-----------+---------+--------+-----+
45 | ``part3`` | 100 GB | 30 GB | 30% |
46 +-----------+---------+--------+-----+
47 | ``part4`` | 100 GB | 70 GB | 70% |
48 +-----------+---------+--------+-----+
49
50 * **random**
51
52 Choice is (pseudo-)random. Each partition has the same probability
53 of being selected.
54
55 * **freespace-most**
56
57 The partition which has the most absolute free space (counted in KiB
58 units) is selected.
59
60 In the example scenario, ``part2`` would be selected as most
61 fitting, since 600 GB of free space is the biggest of all
62 partitions.
63
64 * **freespace-percent-most**
65
66 The partition which has the most relative free space (counted in
67 percentiles) is selected.
68
69 In the example scenario, ``part4`` would be selected as most
70 fitting, since 70% of free space is the biggest of all partitions.
71
72 * **freespace-percent-weighted**
73
74 For each partition, the percentage of free space is its weight. Then
75 a weighted choice is performed to select one of those partitions.
76
77 As such, the more free space the partition has, the higher its
78 chances to be selected.
79
80 In the example scenario, the weight of each partition would be:
81
82 * 40 for ``part1``
83 * 60 for ``part2``
84 * 30 for ``part3``
85 * 70 for ``part4``
86
87 The sum of all weights being 200, the probability for each
88 partition to be selected as most fitting is:
89
90 * 20% for ``part1``
91 * 30% for ``part2``
92 * 15% for ``part3``
93 * 35% for ``part4``
94
95 Out of 20 (hypothetically empty) mailboxes to be created,
96 chances are:
97
98 * 4 are created on ``part1``
99 * 6 are created on ``part2``
100 * 3 are created on ``part3``
101 * 7 are created on ``part4``
102
103 .. NOTE::
104
105 In ``freespace-percent-weighted`` mode, partitions percentage
106 usages converge towards 100%. So if they have different usages,
107 those differences will stay and only really diminish upon
108 reaching 100% of usage.
109
110 You may also observe growing differences between partitions
111 usage when those partitions do not have the same total disk
112 space.
113
114 **freespace-percent-weighted-delta**
115
116 As for ``freespace-percent-weighted``, a weight is associated to
117 each partition. It is computed as follows:
118
119 .. math::
120
121 free - leastfree + 0.5
122
123 Then a weighted choice is performed to select one of those
124 partitions.
125
126 As such, considering the percentages of usage, the more the
127 partition is lagging behind the most used partition (which is the
128 one with the lowest **percentage** of free space), the higher its
129 chances are to be selected.
130
131 .. NOTE::
132
133 The added 0.5 in partitions weight causes the selection to get
134 smoother the more partitions get close to each other.
135
136 In the example scenario, the weight of each partition would be:
137
138 * For ``part1``: :math:`40 - 30 + 0.5 = 10.5`
139 * For ``part2``: :math:`60 - 30 + 0.5 = 30.5`
140 * For ``part3``: :math:`30 - 30 + 0.5 = 00.5`
141 * For ``part4``: :math:`70 - 30 + 0.5 = 40.5`
142
143 The sum of all weights amounting to 82, the probability for each
144 partition to be selected as most fitting would be:
145
146 * For ``part1``: 12.8%
147 * For ``part2``: 37.2%
148 * For ``part3``: 00.6%
149 * For ``part4``: 49.4%
150
151 .. NOTE::
152
153 In ``freespace-percent-weighted-delta`` mode, partitions
154 percentage usages converge towards the most used one, after
155 which partitions usages grow equally.
156
157 Special Cases
158 -------------
159
160 Q: What happens when two partitions are equal as most fitting?
161 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
162
163 Suppose you are using the ``freespace-most`` selection mode, that two
164 (or more) partitions have the same free size, and that this freespace
165 happens to be the biggest one of all configured partitions.
166
167 In that case, only one of those partitions will be selected. You may not
168 know in advance which one will be: it depends of the order in which
169 configured partitions are stored in memory (``hashtable``). In
170 particular, it may not be the first one that appears listed in your
171 :cyrusman:`imapd.conf(5)` configuration file.
172
173 Also note that since the selected partition will now have less free
174 space, it shall not be seen as most fitting next time.
175
176 Q: What happens when two partitions point to the same device?
177 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
178
179 Suppose you are using the ``freespace-most`` or
180 ``freespace-percent-most`` selection mode, and that two (or more)
181 partitions actually point to a different directory on the same
182 filesystem.
183
184 In that case, only one of those partitions will be checked, as if the
185 others were not configured. Cyrus IMAP uses the device id of the
186 partition hierarchy for this. You may not know in advance which one will
187 be: it depends of the order in which configured partitions are stored in
188 memory (``hashtable``). In particular, it may not be the first one that
189 appears listed in your :cyrusman:`imapd.conf(5)` configuration file.
190
191 Excluding Partitions
192 --------------------
193
194 **partition_select_exclude**
195
196 Partitions listed in the ``partition_select_exclude`` setting in
197 :cyrusman:`imapd.conf(5)` are permanently excluded from being
198 selected.
199
200 Listed partition's names are separated by spaces and/or commas. Only
201 configured partition names (and not paths) are expected.
202
203 With the following configuration in :cyrusman:`imapd.conf(5)`,
204 ``parta`` and ``partb`` would be permanently excluded from being
205 selected:
206
207 .. parsed-literal::
208
209 partition-part1: /path/to/part1
210 partition-part2: /path/to/part2
211 partition-part3: /path/to/part3
212 partition-part4: /path/to/part4
213 partition-parta: /path/to/parta
214 partition-partb: /path/to/partb
215
216 partition_mode_exclude: parta partb
217
218 **partition_select_soft_usage_limit**
219
220 When using a selection mode other than ``random``, partitions are
221 automatically excluded if their usage percentage is beyond the
222 ``partition_select_soft_usage_limit`` integer setting.
223
224 If all partitions are beyond the configured value, this feature is
225 automatically deactivated. A partition is thus selected as if the
226 setting was not configured.
227
228 In the example scenario, setting ``partition_mode_soft_usage_limit``
229 to ``50`` would exclude partitions ``part1`` and ``part3`` since
230 their disk usages are 60% and 70% respectively.
231
232 But setting the option to ``20`` would have no immediate effect on
233 the selection of a partition, since the usage of all partitions is
234 beyond 20%.
235
236 .. IMPORTANT::
237
238 **Exclusion is not absolute**
239
240 Partitions are only excluded when creating a new user mailbox
241 according to the configured selection mode.
242
243 If you explicitly specify an excluded partition, the mailbox will
244 be created on that partition.
245
246 Mailboxes previously created on such partitions are of course still
247 accessible, and subfolders are by default still created on the same
248 partition as the parent folder.
249
250 Partition's Usage Information Reset
251 -----------------------------------
252
253 By default, partition's usage information is retrieved only once -- when
254 the service first initializes.
255
256 If you tend to use the same service instance for a long lapse of time
257 and performs a large amount of mailbox creations, it may be useful to
258 configure the ``partition_select_usage_reinit`` setting so that
259 each partition's usage information is refreshed after the configured
260 number of creation requests.
261
262 .. NOTE::
263
264 This only concerns selection modes other than `random``.
265
266 Backend Selection Mode (Cyrus IMAP Murder)
267 ==========================================
268
269 Upon creating a user mailbox, if the target server is not given as
270 extra parameter, the mailbox is created on either;
271
272 * the configured ``defaultserver`` backend
273
274 * the most fitting backend (partition) among the servers listed in the
275 ``serverlist`` setting, if ``defaultserver`` is not configured.
276
277 .. rubric:: Related Settings
278
279 * ``serverlist_select_mode``
280 * ``serverlist_select_usage_reinit``
281 * ``serverlist_select_soft_usage_limit``
282
283 Available Selection Modes on Frontend
284 -------------------------------------
285
286 **random**
287
288 The selection is (pseudo-)random. Each backend has the same
289 probability of being selected.
290
291 .. sidebar:: Example Scenario for Selection Modes
292
293 Suppose that the partitions configured are:
294
295 +---------+-----------+---------+---------+-----+
296 | Backend | Partition | Total | Free | % |
297 +=========+===========+=========+=========+=====+
298 | ``be1`` | | 2000 GB | 1000 GB | 50% |
299 +---------+-----------+---------+---------+-----+
300 | | ``part1`` | 1000 GB | 500 GB | 50% |
301 +---------+-----------+---------+---------+-----+
302 | | ``part2`` | 1000 GB | 500 GB | 50% |
303 +---------+-----------+---------+---------+-----+
304 | ``be2`` | | 2000 GB | 900 GB | 45% |
305 +---------+-----------+---------+---------+-----+
306 | | ``part1`` | 1000 GB | 200 GB | 20% |
307 +---------+-----------+---------+---------+-----+
308 | | ``part2`` | 1000 GB | 700 GB | 70% |
309 +---------+-----------+---------+---------+-----+
310 | ``be3`` | | 200 GB | 110 GB | 55% |
311 +---------+-----------+---------+---------+-----+
312 | | ``part1`` | 100 GB | 30 GB | 30% |
313 +---------+-----------+---------+---------+-----+
314 | | ``part2`` | 100 GB | 80 GB | 80% |
315 +---------+-----------+---------+---------+-----+
316
317 **freespace-most**
318
319 The backend which has the most absolute free space (counted in KiB
320 units) is selected.
321
322 .. NOTE::
323
324 The considered free space is the sum of the free space on all
325 available partitions on the backend.
326
327 In the example scenario, ``be1`` would be selected as most fitting,
328 since 1000 GB of free space is the most of all backends.
329
330 **freespace-percent-most**
331
332 On each backend, only the partition with the most percentage of free
333 space is considered. The selected backend is the one whose partition
334 has the highest percentage of free space.
335
336 In the example scenario, ``be3`` would be selected as most fitting,
337 since it has a partition with 80% of free space which is the
338 highest of all backends.
339
340 **freespace-percent-weighted**
341
342 On each backend, only the partition with the most percentage of free
343 space is considered: this is the backend weight. Then a weighted
344 choice is performed to select one of the backends.
345
346 In the example scenario, the weight of each backend would be:
347
348 * 50 for ``be1``
349 * 70 for ``be2``
350 * 80 for ``be3``
351
352 The sum of all weights being 200, the probability for each backend
353 to be selected as most fitting would be:
354
355 * 25% for ``be1``
356 * 35% for ``be2``
357 * 40% for ``be3``
358
359 **freespace-percent-weighted-delta**
360
361 On each backend, only the partition with the most percentage of free
362 space is considered.
363
364 Like with ``freespace-percent-weighted``, a weight is associated to
365 each backend. It is computed as follows:
366
367 .. math::
368
369 free - leastfree + 0.5
370
371 Then a weighted choice is performed to select one of the backends.
372
373 In the example scenario, the weight of each backend would be:
374
375 * For ``be1``: :math:`50 - 50 + 0.5 = 0.5`
376 * For ``be2``: :math:`70 - 50 + 0.5 = 20.5`
377 * For ``be3``: :math:`80 - 50 + 0.5 = 30.5`
378
379 Then the probability for each backend to be selected as most fitting would be:
380
381 * 1.0% for ``be1``
382 * 39.8% for ``be2``
383 * 59.2% for ``be3``
384
385 Excluding Backends
386 ------------------
387
388 When using a selection mode other than ``random``, backends are
389 automatically excluded if their considered usage percentage is beyond
390 the ``serverlist_select_soft_usage_limit`` integer setting.
391
392 Backend's Usage Information Reset
393 ---------------------------------
394
395 By default backends usage data are retrieved only once upon service
396 initialization.
397
398 .. NOTE::
399
400 This only concerns selection modes other than ``random``.
401
402 If you tend to use the same service instance for a long lapse of time
403 and performs a large amounts of mailbox creations, it may be useful to
404 configure the ``serverlist_select_usage_reinit`` so that the backend's
405 disk usage information is refreshed after the configured number of
406 creation requests.
407
408 Back to :ref:`imap-features`