Debian Codegraph @

doc/html/_sources/imap/concepts/deployment/mailbox_creation_distribution.rst.txt

raw
1 Mailbox Creation Distribution
2 =============================
3
4 By default, when creating a mailbox in Cyrus IMAP:
5
6 * the backend with the most free disk space is selected on the Murder frontend
7 * the partition with the most free disk space is selected on the backend
8
9 This may not be the most appropriate backend or partition to create the new mailbox on, and Cyrus IMAP therefor allows for a variety of additional modes of calculating and selecting the most appropriate backend and partition. The exact mode for the selection is controlled with the ``imapd.conf`` setting ``serverlist_select_mode`` on the frontend and ``partition_select_mode`` on the backend.
10
11 Alternatively, a default backend can be configured with the ``defaultserver`` setting on a frontend, and a default partition can be configured with the ``defaultpartition`` on a backend.
12
13
14 Prior to Cyrus IMAP version 2.5, when creating a mailbox, should no
15 target partition have been specified, the mailbox is either created on:
16
17 * the configured ``defaultserver``, or
18
19 * the server that is found to have the most free disk space, and
20
21 * the configured ``defaultpartition``, or
22
23 * the most fitting partition if ``defaultpartition`` is not
24 configured.
25
26 New configuration options are available since Cyrus IMAP 2.5, that
27 allow more weighted and better balanced backend and partition
28 selection.
29
30 Selection Mode
31 --------------
32
33 Among the partitions, how the most fitting one is selected depends on the configured selection mode: ``partition_mode`` setting in ``/etc/imapd.conf``
34
35 Available Selection Modes
36 """""""""""""""""""""""""
37
38 .. todo::
39
40 See if this is the same as the section in the administrator guide - if so, just link there?
41
42 * random
43
44 Choice is (pseudo-)random. Each partition has the same probability of being selected.
45
46 * freespace-most
47
48 The partition which has the most free space (counted in KiB units) is selected.
49
50 .. note::
51 Example of ``freespace-most`` selection on backend
52
53 Suppose that the configured partitions are:
54
55 * ``part1`` which has a 1000GiB capacity, 400GiB being free (that is, 40% free space)
56 * ``part2`` which has a 1000GiB capacity, 600GiB being free (that is, 60% free space)
57 * ``part3`` which has a 100GiB capacity, 30GiB being free (that is, 30% free space)
58 * ``part4`` which has a 100GiB capacity, 70GiB being free (that is, 70% free space)
59
60 In that case ``part2`` will be selected as most fitting, since 600GiB of free space is the biggest of all partitions.
61
62 * freespace-percent-most
63
64 The partition which has the most percentage of free space is selected.
65
66 .. note::
67 Example of ``freespace-percent-most`` selection on backend
68
69 In the same conditions, ``part4`` would be selected as most fitting, since 70% of free space is the biggest of all partitions.
70
71 * freespace-percent-weighted
72
73 For each partition, the percentage of free space is its weight. Then a weighted choice is performed to select one of those partitions.
74
75 As such, the more free space the partition has, the more chances it has to be selected.
76
77 .. note::
78 Example of ``freespace-percent-weighted`` selection on backend
79
80 In the same conditions, the weight of each partition would be:
81
82 * 40 for ``part1``
83 * 60 for ``part2``
84 * 30 for ``part3``
85 * 70 for ``part4``
86
87 The sum of each weight being 200, the probability for each partition to be selected as most fitting would be:
88
89 * 20% for ``part1``
90 * 30% for ``part2``
91 * 15% for ``part3``
92 * 35% for ``part4``
93
94 Usage convergence
95 """""""""""""""""
96
97 In ``freespace-percent-weighted`` mode, partitions percentage usages converge towards 100%. So if they have different usages, those differences will stay and only really diminish upon reaching 100% of usage.
98
99 You may also observe growing differences between partitions usage when those partitions do not have the same total disk space.
100
101 * freespace-percent-weighted-delta
102
103 As for ``freespace-percent-weighted``, a weight is associated to each partition. It is computed as follows: (*percentage of freespace of partition*) - (*lowest percentage of freespace of all partitions*) + 0.5
104
105 Then a weighted choice is performed to select one of those partitions.
106
107 As such, considering the percentages of usage, the more the partition is lagging behind the most used partition (which is the one with the lowest percentage of free space), the more chances it has to be selected.
108
109 Computed weight
110 """""""""""""""
111
112 The added 0.5 in partitions weight is so that selection gets smoother when all partitions get close to each other.
113
114 .. note ::
115 **Example of ``freespace-percent-weighted-delta`` Selection on Backend**
116
117 In the same conditions, the weight of each partition would be:
118
119 * 40 - 30 + 0.5 = 10.5 for ``part1``
120 * 60 - 30 + 0.5 = 30.5 for ``part2``
121 * 30 - 30 + 0.5 = 0.5 for ``part3``
122 * 70 - 30 + 0.5 = 40.5 for ``part4``
123
124 Then the probability for each partition to be selected as most fitting would be:
125
126 * 12.8% for ``part1``
127 * 37.2% for ``part2``
128 * 0.6% for ``part3``
129 * 49.4% for ``part4``
130
131 **Usage convergence**
132
133 In ``freespace-percent-weighted-delta`` mode, partitions percentage usages converge towards the most used one. And then partitions usages grow equally.
134
135 Special cases
136 -------------
137
138 What happens when two partitions are equal as most fitting?
139 """""""""""""""""""""""""""""""""""""""""""""""""""""""""""
140
141 Suppose you are using the ``freespace-most`` selection mode, that two (or more) partitions have the same free size, and that this freespace happens to be the biggest one of all configured partitions.
142
143 In that case, only one of those partitions will be selected. You may not know in advance which one will be: it depends of the order in which configured partitions are stored in memory (``hashtable``). In particular, it may not be the first one that appears listed in your ``/etc/imapd.conf`` configuration file.
144
145 Also note that since the selected partition will now have less free space, it shall not be seen as most fitting next time.
146
147 What happens when two partitions point to the same device?
148 """"""""""""""""""""""""""""""""""""""""""""""""""""""""""
149
150 Suppose you are using the ``freespace-most`` or ``freespace-percent-most`` selection mode, and that two (or more) partitions actually point to the same device (that is the device id is the same).
151
152 In that case, only one of those partitions will be checked, as if the others were not configured. You may not know in advance which one will be: it depends of the order in which configured partitions are stored in memory (``hashtable``). In particular, it may not be the first one that appears listed in your ``/etc/imapd.conf`` configuration file.
153
154 Partitions Exclusion
155 --------------------
156
157 Partitions listed in the ``partition_mode_exclude`` setting are permanently excluded from being selected.
158
159 * partition_mode_exclude
160
161 Listed partitions names are separated by space or comma. Only configured partition names (and not paths) are expected.
162
163 .. note::
164 Example of permanent partition exclusion
165
166 With the following configuration in `/etc/imapd.conf`, ``parta`` and ``partb`` would be permanently excluded from being selected:
167
168 partition-part1: /path/to/part1
169 partition-part2: /path/to/part2
170 partition-part3: /path/to/part3
171 partition-part4: /path/to/part4
172 partition-parta: /path/to/parta
173 partition-partb: /path/to/partb
174
175 partition_mode_exclude: parta partb
176
177
178 When using a selection mode other than ``random``, partitions are automatically excluded if their usage percentage is beyond the ``partition_mode_soft_usage_limit`` integer setting.
179
180
181 * partition_mode_soft_usage_limit
182
183 If all partitions are beyond the configured value, this feature is automatically deactivated. A partition is thus selected as if the setting was not set.
184
185 .. note::
186 Example of partition exclusion using ``partition_mode_soft_usage_limit``
187
188 In the same conditions than <xref linkend="exam-Deployment_Guide-Available_Selection_Modes_on_Backend-Example_of_freespace_most_Selection_on_Backend" />, setting ``partition_mode_soft_usage_limit`` to ``50`` would exclude partitions ``part1`` and ``part3`` since their disk usage is respectively 60% and 70%.
189
190 But setting the option to ``20`` would have no effect, since the usage of all partitions is beyond 20%.
191
192 .. warning::
193
194 **Exclusion is not absolute**
195
196 Partitions are only excluded when creating a new user mailbox according to the configured selection mode.
197
198 If you explicitly specify an excluded partition, the mailbox will be created on that partition.
199
200 Mailboxes previously created on such partitions are of course still accessible, and subfolders are by default still created on the same partition as the parent folder.
201
202
203 Partitions Usage Data Reset
204 ---------------------------
205
206 By default partitions usage data are retrieved only once upon service initialization. This only concerns selection modes other than ``random``.
207
208 If you tend to use the same service instance for a long lapse of time and performs a large amount of mailboxes creation, it may be useful to configure the ``partition_mode_usage_reinit`` so that partitions usage data are refreshed after the configured number of creation requests.
209
210
211 Mailbox Creation Distribution Through ``murder frontend``
212 ---------------------------------------------------------
213
214 Upon creating a user mailbox, if the target server is not given as extra parameter, the mailbox is either created on
215
216 * the configured ``defaultserver`` backend
217 * the most fitting backend among the servers listed in the ``serverlist`` setting, if ``defaultserver`` is not configured
218
219
220 Selection Mode
221 """"""""""""""
222
223 Among the backends, how the most fitting one is selected depends on the configured selection mode: ``serverlist_mode`` setting in ``/etc/imapd.conf``.
224
225 The principle is similar to the mailbox creation distribution on backend (see <xref linkend="sect-Deployment_Guide-Mailbox_Creation_Distribution-On_backend" />).
226
227
228 Available Selection Modes on Frontend
229 """""""""""""""""""""""""""""""""""""
230
231 * random
232
233 Choice is (pseudo-)random. Each backend has the same probability of being selected.
234
235 * freespace-most
236
237 The backend which has the most free space (counted in KiB units) is selected. The considered free space is the sum of all available partitions free space on the backend.
238
239 .. note::
240 Example of ``freespace-most`` Selection on Frontend
241
242 Suppose that the configured backends are:
243
244 * ``backend1`` which has a 2000GiB capacity, 1000GiB being free (that is, 50% free space), composed of 2 partitions
245 * ``part1`` which has a 1000GiB capacity, 500GiB being free (that is, 50% free space)
246 * ``part2`` which has a 1000GiB capacity, 500GiB being free (that is, 50% free space)
247
248 * ``backend2`` which has a 2000GiB capacity, 900GiB being free (that is, 45% free space), composed of 2 partitions
249 * ``part1`` which has a 1000GiB capacity, 200GiB being free (that is, 20% free space)
250 * ``part2`` which has a 1000GiB capacity, 700GiB being free (that is, 70% free space)
251
252 * ``backend3`` which has a 200GiB capacity, 110GiB being free (that is, 55% free space), composed of 2 partitions
253 * ``part1`` which has a 100GiB capacity, 30GiB being free (that is, 30% free space)
254 * ``part2`` which has a 100GiB capacity, 80GiB being free (that is, 80% free space)
255
256 * In that case ``backend1`` will be selected as most fitting, since 1000GiB of free space is the biggest of all backends.
257
258 * freespace-percent-most
259
260 On each backend, the partition with the most percentage of free space is considered. The selected backend is the one whose partition has the most percentage of free space.
261
262 .. note::
263 Example of ``freespace-percent-most`` Selection on Frontend
264
265 In the same conditions than <xref linkend="exam-Deployment_Guide-Available_Selection_Modes_on_Frontend-Example_of_freespace_most_Selection_on_Frontend" /> ``backend3`` would be selected as most fitting, since it has a partition with 80% of free space which is the biggest of all backends.
266
267
268 * freespace-percent-weighted
269
270 On each backend, the partition with the most percentage of free space is considered: it is the backend weight. Then a weighted choice is performed to select one of the backends.
271
272 .. note::
273 Example of ``freespace-percent-weighted`` Selection on Frontend
274
275 In the same conditions than <xref linkend="exam-Deployment_Guide-Available_Selection_Modes_on_Frontend-Example_of_freespace_most_Selection_on_Frontend" />, the weight of each backend would be:
276
277 * 50 for ``backend1``
278 * 70 for ``backend2``
279 * 80 for ``backend3``
280
281 The sum of each weight being 200, the probability for each backend to be selected as most fitting would be:
282
283 * 25% for ``backend1``
284 * 35% for ``backend2``
285 * 40% for ``backend3``
286
287 * freespace-percent-weighted-delta
288
289 On each backend, the partition with the most percentage of free space is considered. As for ``freespace-percent-weighted``, a weight is associated to each backend. It is computed as follows: (*percentage of freespace on backend*) - (*lowest percentage of freespace of all backends*) + 0.5
290
291 Then a weighted choice is performed to select one of the backends.
292
293 .. note::
294 **Example of ``freespace-percent-weighted-delta`` Selection on Frontend**
295
296 In the same conditions than <xref linkend="exam-Deployment_Guide-Available_Selection_Modes_on_Frontend-Example_of_freespace_most_Selection_on_Frontend" />, the weight of each backend would be:
297
298 * 50 - 50 + 0.5 = 0.5 for ``backend1``
299 * 70 - 50 + 0.5 = 20.5 for ``backend2``
300 * 80 - 50 + 0.5 = 30.5 for ``backend3``
301
302 Then the probability for each backend to be selected as most fitting would be:
303
304 * 1.0% for ``backend1``
305 * 39.8% for ``backend2``
306 * 59.2% for ``backend3``
307
308
309 Backends Exclusion
310 ------------------
311
312 When using a selection mode other than ``random``, backends are automatically excluded if their considered usage percentage is beyond the ``serverlist_mode_soft_usage_limit`` integer setting.
313
314 * ``partition_mode_soft_usage_limit``
315
316 If all backends are beyond the configured value, this feature is automatically deactivated. A backend is thus selected as if the setting was not set.
317
318 .. note::
319 Example of partition exclusion using ``serverlist_mode_soft_usage_limit``
320
321 In the same conditions than <xref linkend="exam-Deployment_Guide-Available_Selection_Modes_on_Frontend-Example_of_freespace_most_Selection_on_Frontend" />, using ``freespace-most`` selection mode, setting ``serverlist_mode_soft_usage_limit`` to ``49`` would exclude ``backend1`` and ``backend2`` since in that mode they have a disk usage of 50% and 55%. In other modes it would however only exclude ``backend1`` whose considered partition has a disk usage of 50%, while on ``backend2`` the considered partition has a disk usage of 30%.
322
323 .. warning::
324
325 **Exclusion is not absolute**
326
327 Backends are only excluded when creating a new user mailbox according to the configured selection mode.
328
329 If you explicitly specify an excluded backend, the mailbox will be created on that backend.
330
331 Mailboxes previously created on such backends are of course still accessible.
332
333
334 Backends Usage Data Reset
335 -------------------------
336
337 By default backends usage data are retrieved only once upon service initialization. This only concerns selection modes other than ``random``.
338
339 If you tend to use the same service instance for a long lapse of time and performs a large amount of mailboxes creation, it may be useful to configure the ``serverlist_mode_usage_reinit`` so that backends usage data are refreshed after the configured number of creation requests.
340