| 1 |
:tocdepth: 2 |
| 2 |
|
| 3 |
.. _imap-features-mailbox-distribution: |
| 4 |
|
| 5 |
==================== |
| 6 |
Mailbox Distribution |
| 7 |
==================== |
| 8 |
|
| 9 |
Prior to Cyrus IMAP version 2.5.0, when creating a mailbox, should no |
| 10 |
target partition have been specified, the mailbox is either created on: |
| 11 |
|
| 12 |
* the configured ``defaultserver``, or |
| 13 |
|
| 14 |
* the server that is found to have the most free disk space, and |
| 15 |
|
| 16 |
* the configured ``defaultpartition``, or |
| 17 |
|
| 18 |
* the most fitting partition if ``defaultpartition`` is not |
| 19 |
configured. |
| 20 |
|
| 21 |
New configuration options are available since Cyrus IMAP 2.5.0, that |
| 22 |
allow more weighted and better balanced backend and partition |
| 23 |
selection. |
| 24 |
|
| 25 |
Partition Selection Mode |
| 26 |
======================== |
| 27 |
|
| 28 |
Among the partitions, how the most fitting one is selected depends on |
| 29 |
the configured selection mode, using the ``partition_select_mode`` |
| 30 |
setting in :cyrusman:`imapd.conf(5)`. |
| 31 |
|
| 32 |
.. rubric:: Available Selection Modes |
| 33 |
|
| 34 |
.. sidebar:: Example Scenario for Selection Modes |
| 35 |
|
| 36 |
Suppose that the partitions configured are: |
| 37 |
|
| 38 |
+-----------+---------+--------+-----+ |
| 39 |
| Partition | Total | Free | % | |
| 40 |
+===========+=========+========+=====+ |
| 41 |
| ``part1`` | 1000 GB | 400 GB | 40% | |
| 42 |
+-----------+---------+--------+-----+ |
| 43 |
| ``part2`` | 1000 GB | 600 GB | 60% | |
| 44 |
+-----------+---------+--------+-----+ |
| 45 |
| ``part3`` | 100 GB | 30 GB | 30% | |
| 46 |
+-----------+---------+--------+-----+ |
| 47 |
| ``part4`` | 100 GB | 70 GB | 70% | |
| 48 |
+-----------+---------+--------+-----+ |
| 49 |
|
| 50 |
* **random** |
| 51 |
|
| 52 |
Choice is (pseudo-)random. Each partition has the same probability |
| 53 |
of being selected. |
| 54 |
|
| 55 |
* **freespace-most** |
| 56 |
|
| 57 |
The partition which has the most absolute free space (counted in KiB |
| 58 |
units) is selected. |
| 59 |
|
| 60 |
In the example scenario, ``part2`` would be selected as most |
| 61 |
fitting, since 600 GB of free space is the biggest of all |
| 62 |
partitions. |
| 63 |
|
| 64 |
* **freespace-percent-most** |
| 65 |
|
| 66 |
The partition which has the most relative free space (counted in |
| 67 |
percentiles) is selected. |
| 68 |
|
| 69 |
In the example scenario, ``part4`` would be selected as most |
| 70 |
fitting, since 70% of free space is the biggest of all partitions. |
| 71 |
|
| 72 |
* **freespace-percent-weighted** |
| 73 |
|
| 74 |
For each partition, the percentage of free space is its weight. Then |
| 75 |
a weighted choice is performed to select one of those partitions. |
| 76 |
|
| 77 |
As such, the more free space the partition has, the higher its |
| 78 |
chances to be selected. |
| 79 |
|
| 80 |
In the example scenario, the weight of each partition would be: |
| 81 |
|
| 82 |
* 40 for ``part1`` |
| 83 |
* 60 for ``part2`` |
| 84 |
* 30 for ``part3`` |
| 85 |
* 70 for ``part4`` |
| 86 |
|
| 87 |
The sum of all weights being 200, the probability for each |
| 88 |
partition to be selected as most fitting is: |
| 89 |
|
| 90 |
* 20% for ``part1`` |
| 91 |
* 30% for ``part2`` |
| 92 |
* 15% for ``part3`` |
| 93 |
* 35% for ``part4`` |
| 94 |
|
| 95 |
Out of 20 (hypothetically empty) mailboxes to be created, |
| 96 |
chances are: |
| 97 |
|
| 98 |
* 4 are created on ``part1`` |
| 99 |
* 6 are created on ``part2`` |
| 100 |
* 3 are created on ``part3`` |
| 101 |
* 7 are created on ``part4`` |
| 102 |
|
| 103 |
.. NOTE:: |
| 104 |
|
| 105 |
In ``freespace-percent-weighted`` mode, partitions percentage |
| 106 |
usages converge towards 100%. So if they have different usages, |
| 107 |
those differences will stay and only really diminish upon |
| 108 |
reaching 100% of usage. |
| 109 |
|
| 110 |
You may also observe growing differences between partitions |
| 111 |
usage when those partitions do not have the same total disk |
| 112 |
space. |
| 113 |
|
| 114 |
**freespace-percent-weighted-delta** |
| 115 |
|
| 116 |
As for ``freespace-percent-weighted``, a weight is associated to |
| 117 |
each partition. It is computed as follows: |
| 118 |
|
| 119 |
.. math:: |
| 120 |
|
| 121 |
free - leastfree + 0.5 |
| 122 |
|
| 123 |
Then a weighted choice is performed to select one of those |
| 124 |
partitions. |
| 125 |
|
| 126 |
As such, considering the percentages of usage, the more the |
| 127 |
partition is lagging behind the most used partition (which is the |
| 128 |
one with the lowest **percentage** of free space), the higher its |
| 129 |
chances are to be selected. |
| 130 |
|
| 131 |
.. NOTE:: |
| 132 |
|
| 133 |
The added 0.5 in partitions weight causes the selection to get |
| 134 |
smoother the more partitions get close to each other. |
| 135 |
|
| 136 |
In the example scenario, the weight of each partition would be: |
| 137 |
|
| 138 |
* For ``part1``: :math:`40 - 30 + 0.5 = 10.5` |
| 139 |
* For ``part2``: :math:`60 - 30 + 0.5 = 30.5` |
| 140 |
* For ``part3``: :math:`30 - 30 + 0.5 = 00.5` |
| 141 |
* For ``part4``: :math:`70 - 30 + 0.5 = 40.5` |
| 142 |
|
| 143 |
The sum of all weights amounting to 82, the probability for each |
| 144 |
partition to be selected as most fitting would be: |
| 145 |
|
| 146 |
* For ``part1``: 12.8% |
| 147 |
* For ``part2``: 37.2% |
| 148 |
* For ``part3``: 00.6% |
| 149 |
* For ``part4``: 49.4% |
| 150 |
|
| 151 |
.. NOTE:: |
| 152 |
|
| 153 |
In ``freespace-percent-weighted-delta`` mode, partitions |
| 154 |
percentage usages converge towards the most used one, after |
| 155 |
which partitions usages grow equally. |
| 156 |
|
| 157 |
Special Cases |
| 158 |
------------- |
| 159 |
|
| 160 |
Q: What happens when two partitions are equal as most fitting? |
| 161 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 162 |
|
| 163 |
Suppose you are using the ``freespace-most`` selection mode, that two |
| 164 |
(or more) partitions have the same free size, and that this freespace |
| 165 |
happens to be the biggest one of all configured partitions. |
| 166 |
|
| 167 |
In that case, only one of those partitions will be selected. You may not |
| 168 |
know in advance which one will be: it depends of the order in which |
| 169 |
configured partitions are stored in memory (``hashtable``). In |
| 170 |
particular, it may not be the first one that appears listed in your |
| 171 |
:cyrusman:`imapd.conf(5)` configuration file. |
| 172 |
|
| 173 |
Also note that since the selected partition will now have less free |
| 174 |
space, it shall not be seen as most fitting next time. |
| 175 |
|
| 176 |
Q: What happens when two partitions point to the same device? |
| 177 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 178 |
|
| 179 |
Suppose you are using the ``freespace-most`` or |
| 180 |
``freespace-percent-most`` selection mode, and that two (or more) |
| 181 |
partitions actually point to a different directory on the same |
| 182 |
filesystem. |
| 183 |
|
| 184 |
In that case, only one of those partitions will be checked, as if the |
| 185 |
others were not configured. Cyrus IMAP uses the device id of the |
| 186 |
partition hierarchy for this. You may not know in advance which one will |
| 187 |
be: it depends of the order in which configured partitions are stored in |
| 188 |
memory (``hashtable``). In particular, it may not be the first one that |
| 189 |
appears listed in your :cyrusman:`imapd.conf(5)` configuration file. |
| 190 |
|
| 191 |
Excluding Partitions |
| 192 |
-------------------- |
| 193 |
|
| 194 |
**partition_select_exclude** |
| 195 |
|
| 196 |
Partitions listed in the ``partition_select_exclude`` setting in |
| 197 |
:cyrusman:`imapd.conf(5)` are permanently excluded from being |
| 198 |
selected. |
| 199 |
|
| 200 |
Listed partition's names are separated by spaces and/or commas. Only |
| 201 |
configured partition names (and not paths) are expected. |
| 202 |
|
| 203 |
With the following configuration in :cyrusman:`imapd.conf(5)`, |
| 204 |
``parta`` and ``partb`` would be permanently excluded from being |
| 205 |
selected: |
| 206 |
|
| 207 |
.. parsed-literal:: |
| 208 |
|
| 209 |
partition-part1: /path/to/part1 |
| 210 |
partition-part2: /path/to/part2 |
| 211 |
partition-part3: /path/to/part3 |
| 212 |
partition-part4: /path/to/part4 |
| 213 |
partition-parta: /path/to/parta |
| 214 |
partition-partb: /path/to/partb |
| 215 |
|
| 216 |
partition_mode_exclude: parta partb |
| 217 |
|
| 218 |
**partition_select_soft_usage_limit** |
| 219 |
|
| 220 |
When using a selection mode other than ``random``, partitions are |
| 221 |
automatically excluded if their usage percentage is beyond the |
| 222 |
``partition_select_soft_usage_limit`` integer setting. |
| 223 |
|
| 224 |
If all partitions are beyond the configured value, this feature is |
| 225 |
automatically deactivated. A partition is thus selected as if the |
| 226 |
setting was not configured. |
| 227 |
|
| 228 |
In the example scenario, setting ``partition_mode_soft_usage_limit`` |
| 229 |
to ``50`` would exclude partitions ``part1`` and ``part3`` since |
| 230 |
their disk usages are 60% and 70% respectively. |
| 231 |
|
| 232 |
But setting the option to ``20`` would have no immediate effect on |
| 233 |
the selection of a partition, since the usage of all partitions is |
| 234 |
beyond 20%. |
| 235 |
|
| 236 |
.. IMPORTANT:: |
| 237 |
|
| 238 |
**Exclusion is not absolute** |
| 239 |
|
| 240 |
Partitions are only excluded when creating a new user mailbox |
| 241 |
according to the configured selection mode. |
| 242 |
|
| 243 |
If you explicitly specify an excluded partition, the mailbox will |
| 244 |
be created on that partition. |
| 245 |
|
| 246 |
Mailboxes previously created on such partitions are of course still |
| 247 |
accessible, and subfolders are by default still created on the same |
| 248 |
partition as the parent folder. |
| 249 |
|
| 250 |
Partition's Usage Information Reset |
| 251 |
----------------------------------- |
| 252 |
|
| 253 |
By default, partition's usage information is retrieved only once -- when |
| 254 |
the service first initializes. |
| 255 |
|
| 256 |
If you tend to use the same service instance for a long lapse of time |
| 257 |
and performs a large amount of mailbox creations, it may be useful to |
| 258 |
configure the ``partition_select_usage_reinit`` setting so that |
| 259 |
each partition's usage information is refreshed after the configured |
| 260 |
number of creation requests. |
| 261 |
|
| 262 |
.. NOTE:: |
| 263 |
|
| 264 |
This only concerns selection modes other than `random``. |
| 265 |
|
| 266 |
Backend Selection Mode (Cyrus IMAP Murder) |
| 267 |
========================================== |
| 268 |
|
| 269 |
Upon creating a user mailbox, if the target server is not given as |
| 270 |
extra parameter, the mailbox is created on either; |
| 271 |
|
| 272 |
* the configured ``defaultserver`` backend |
| 273 |
|
| 274 |
* the most fitting backend (partition) among the servers listed in the |
| 275 |
``serverlist`` setting, if ``defaultserver`` is not configured. |
| 276 |
|
| 277 |
.. rubric:: Related Settings |
| 278 |
|
| 279 |
* ``serverlist_select_mode`` |
| 280 |
* ``serverlist_select_usage_reinit`` |
| 281 |
* ``serverlist_select_soft_usage_limit`` |
| 282 |
|
| 283 |
Available Selection Modes on Frontend |
| 284 |
------------------------------------- |
| 285 |
|
| 286 |
**random** |
| 287 |
|
| 288 |
The selection is (pseudo-)random. Each backend has the same |
| 289 |
probability of being selected. |
| 290 |
|
| 291 |
.. sidebar:: Example Scenario for Selection Modes |
| 292 |
|
| 293 |
Suppose that the partitions configured are: |
| 294 |
|
| 295 |
+---------+-----------+---------+---------+-----+ |
| 296 |
| Backend | Partition | Total | Free | % | |
| 297 |
+=========+===========+=========+=========+=====+ |
| 298 |
| ``be1`` | | 2000 GB | 1000 GB | 50% | |
| 299 |
+---------+-----------+---------+---------+-----+ |
| 300 |
| | ``part1`` | 1000 GB | 500 GB | 50% | |
| 301 |
+---------+-----------+---------+---------+-----+ |
| 302 |
| | ``part2`` | 1000 GB | 500 GB | 50% | |
| 303 |
+---------+-----------+---------+---------+-----+ |
| 304 |
| ``be2`` | | 2000 GB | 900 GB | 45% | |
| 305 |
+---------+-----------+---------+---------+-----+ |
| 306 |
| | ``part1`` | 1000 GB | 200 GB | 20% | |
| 307 |
+---------+-----------+---------+---------+-----+ |
| 308 |
| | ``part2`` | 1000 GB | 700 GB | 70% | |
| 309 |
+---------+-----------+---------+---------+-----+ |
| 310 |
| ``be3`` | | 200 GB | 110 GB | 55% | |
| 311 |
+---------+-----------+---------+---------+-----+ |
| 312 |
| | ``part1`` | 100 GB | 30 GB | 30% | |
| 313 |
+---------+-----------+---------+---------+-----+ |
| 314 |
| | ``part2`` | 100 GB | 80 GB | 80% | |
| 315 |
+---------+-----------+---------+---------+-----+ |
| 316 |
|
| 317 |
**freespace-most** |
| 318 |
|
| 319 |
The backend which has the most absolute free space (counted in KiB |
| 320 |
units) is selected. |
| 321 |
|
| 322 |
.. NOTE:: |
| 323 |
|
| 324 |
The considered free space is the sum of the free space on all |
| 325 |
available partitions on the backend. |
| 326 |
|
| 327 |
In the example scenario, ``be1`` would be selected as most fitting, |
| 328 |
since 1000 GB of free space is the most of all backends. |
| 329 |
|
| 330 |
**freespace-percent-most** |
| 331 |
|
| 332 |
On each backend, only the partition with the most percentage of free |
| 333 |
space is considered. The selected backend is the one whose partition |
| 334 |
has the highest percentage of free space. |
| 335 |
|
| 336 |
In the example scenario, ``be3`` would be selected as most fitting, |
| 337 |
since it has a partition with 80% of free space which is the |
| 338 |
highest of all backends. |
| 339 |
|
| 340 |
**freespace-percent-weighted** |
| 341 |
|
| 342 |
On each backend, only the partition with the most percentage of free |
| 343 |
space is considered: this is the backend weight. Then a weighted |
| 344 |
choice is performed to select one of the backends. |
| 345 |
|
| 346 |
In the example scenario, the weight of each backend would be: |
| 347 |
|
| 348 |
* 50 for ``be1`` |
| 349 |
* 70 for ``be2`` |
| 350 |
* 80 for ``be3`` |
| 351 |
|
| 352 |
The sum of all weights being 200, the probability for each backend |
| 353 |
to be selected as most fitting would be: |
| 354 |
|
| 355 |
* 25% for ``be1`` |
| 356 |
* 35% for ``be2`` |
| 357 |
* 40% for ``be3`` |
| 358 |
|
| 359 |
**freespace-percent-weighted-delta** |
| 360 |
|
| 361 |
On each backend, only the partition with the most percentage of free |
| 362 |
space is considered. |
| 363 |
|
| 364 |
Like with ``freespace-percent-weighted``, a weight is associated to |
| 365 |
each backend. It is computed as follows: |
| 366 |
|
| 367 |
.. math:: |
| 368 |
|
| 369 |
free - leastfree + 0.5 |
| 370 |
|
| 371 |
Then a weighted choice is performed to select one of the backends. |
| 372 |
|
| 373 |
In the example scenario, the weight of each backend would be: |
| 374 |
|
| 375 |
* For ``be1``: :math:`50 - 50 + 0.5 = 0.5` |
| 376 |
* For ``be2``: :math:`70 - 50 + 0.5 = 20.5` |
| 377 |
* For ``be3``: :math:`80 - 50 + 0.5 = 30.5` |
| 378 |
|
| 379 |
Then the probability for each backend to be selected as most fitting would be: |
| 380 |
|
| 381 |
* 1.0% for ``be1`` |
| 382 |
* 39.8% for ``be2`` |
| 383 |
* 59.2% for ``be3`` |
| 384 |
|
| 385 |
Excluding Backends |
| 386 |
------------------ |
| 387 |
|
| 388 |
When using a selection mode other than ``random``, backends are |
| 389 |
automatically excluded if their considered usage percentage is beyond |
| 390 |
the ``serverlist_select_soft_usage_limit`` integer setting. |
| 391 |
|
| 392 |
Backend's Usage Information Reset |
| 393 |
--------------------------------- |
| 394 |
|
| 395 |
By default backends usage data are retrieved only once upon service |
| 396 |
initialization. |
| 397 |
|
| 398 |
.. NOTE:: |
| 399 |
|
| 400 |
This only concerns selection modes other than ``random``. |
| 401 |
|
| 402 |
If you tend to use the same service instance for a long lapse of time |
| 403 |
and performs a large amounts of mailbox creations, it may be useful to |
| 404 |
configure the ``serverlist_select_usage_reinit`` so that the backend's |
| 405 |
disk usage information is refreshed after the configured number of |
| 406 |
creation requests. |
| 407 |
|
| 408 |
Back to :ref:`imap-features` |