| 1 |
.. _murder-installation: |
| 2 |
|
| 3 |
============================================= |
| 4 |
Cyrus Murder: Installation and Administration |
| 5 |
============================================= |
| 6 |
|
| 7 |
Architecture |
| 8 |
============ |
| 9 |
|
| 10 |
:ref:`Overall structure of a Cyrus Murder <architecture_murder>`. |
| 11 |
|
| 12 |
The Cyrus Murder provides the ability to horizontally scale your Cyrus |
| 13 |
IMAP environment across multiple servers. It is similar in nature to |
| 14 |
various proxy solutions such as nginx or perdition with the difference |
| 15 |
being that Murder offers a uniform namespace. Those not currently using |
| 16 |
shared mailboxes and who don't intend to use shared mailboxes in the |
| 17 |
future, should probably just consider using a simple proxy solution. |
| 18 |
|
| 19 |
Before beginning Cyrus Murder configuration, we strongly recommended a |
| 20 |
thorough review of the :ref:`Cyrus Murder Concepts <murder_concepts>` |
| 21 |
guide. |
| 22 |
|
| 23 |
Terminology |
| 24 |
=========== |
| 25 |
|
| 26 |
Master |
| 27 |
In this document, Master always means the MUPDATE master server. |
| 28 |
|
| 29 |
Installation |
| 30 |
============ |
| 31 |
|
| 32 |
Cyrus IMAPd must be built with the ``--enable-murder`` configure |
| 33 |
option. This builds the proxyds and the associated utilities. |
| 34 |
|
| 35 |
.. Note:: |
| 36 |
Those using their distribution's packages may need to install a |
| 37 |
separate package for aggregation support. For example, on Debian |
| 38 |
and derived distros, install the ``cyrus-murder`` package. |
| 39 |
|
| 40 |
Requirements |
| 41 |
------------ |
| 42 |
|
| 43 |
* At least one Cyrus IMAP server instance. If there are more than |
| 44 |
one, their namespaces **must not** conflict. That is, all the |
| 45 |
mailbox names must be unique (or in different namespaces) |
| 46 |
|
| 47 |
* At least one machine that will become the first Frontend Server. |
| 48 |
|
| 49 |
* One machine to become the MUPDATE Master server. This can be the |
| 50 |
same as one of the frontend servers. |
| 51 |
|
| 52 |
Configuring the MUPDATE Master |
| 53 |
------------------------------ |
| 54 |
|
| 55 |
The MUPDATE Master server needs to be running the mupdate service in |
| 56 |
master mode. The MUPDATE master may be one of the cluster's frontend |
| 57 |
instances, in which case no slave mupdate process should be run on this |
| 58 |
instance. |
| 59 |
|
| 60 |
On the mupdate master :cyrusman:`cyrus.conf(5)` must include a line |
| 61 |
similar to the following in the SERVICES section:: |
| 62 |
|
| 63 |
mupdate cmd="/usr/cyrus/bin/mupdate -m" listen=3905 prefork=1 |
| 64 |
|
| 65 |
Note the ``-m`` option to tell mupdate that it should start in master |
| 66 |
mode. |
| 67 |
|
| 68 |
The MUPDATE Master will also need at least a skeleton |
| 69 |
:cyrusman:`imapd.conf(5)` that defines the config directory, a bogus |
| 70 |
partition-default and the ``admin``\ s who can authenticate to the server. |
| 71 |
Slave mupdate servers as well as the back end servers will need to be |
| 72 |
able to authenticate as admins on the master. |
| 73 |
|
| 74 |
Here is a very simple :cyrusman:`imapd.conf(5)` for a master server:: |
| 75 |
|
| 76 |
configdirectory: /imap/conf |
| 77 |
partition-default: /tmp |
| 78 |
|
| 79 |
admins: mupdateslave1 backend1 |
| 80 |
|
| 81 |
SASL must also be configured as needed to properly allow authentication. |
| 82 |
|
| 83 |
Setting up the backends to push changes to the MUPDATE Master |
| 84 |
------------------------------------------------------------- |
| 85 |
|
| 86 |
On the backends, configuration to be a part of a murder is easy. Simply |
| 87 |
set the ``mupdate_server`` option in :cyrusman:`imapd.conf(5)` and add |
| 88 |
an entry to :cyrusman:`cyrus.conf(5)` to push the mailboxes list to |
| 89 |
the MUPDATE Master. |
| 90 |
|
| 91 |
Depending on the authentication mechanisms used, some or all of the |
| 92 |
following settings in :cyrusman:`imapd.conf(5)` may be required: |
| 93 |
|
| 94 |
* mupdate_username |
| 95 |
* mupdate_authname |
| 96 |
* mupdate_realm |
| 97 |
* mupdate_password |
| 98 |
* servername |
| 99 |
|
| 100 |
Once these settings are made, any mailbox operation on the backend will |
| 101 |
be sent to the mupdate master for confirmation and entry into the |
| 102 |
mupdate database. |
| 103 |
|
| 104 |
At least one user/group must be configured using the ``proxyservers`` |
| 105 |
:cyrusman:`imapd.conf(5)` option. This user should not be an |
| 106 |
administrator, as that would give anyone compromising this credential |
| 107 |
full administrative control on all back ends. |
| 108 |
|
| 109 |
.. Note:: |
| 110 |
For lmtp to work in a murder, the proxyservers entries must also |
| 111 |
appear in the lmtp_admins entry. |
| 112 |
|
| 113 |
Example of the :cyrusman:`imapd.conf(5)` settings discussed thus far:: |
| 114 |
|
| 115 |
# How this server identifies itself within the murder |
| 116 |
servername: mailbox.example.com |
| 117 |
# Who's permitted to authenticate for which purposes |
| 118 |
admins: cyrus |
| 119 |
proxyservers: mailproxy |
| 120 |
lmtp_admins: mailproxy |
| 121 |
# Auth credentials for MUPDATE Master |
| 122 |
mupdate_server: postman.example.com |
| 123 |
mupdate_username: postman |
| 124 |
mupdate_authname: postman |
| 125 |
mupdate_password: <secret> |
| 126 |
|
| 127 |
All proxy user(s) must exist within the authentication domain of both |
| 128 |
the MUPDATE Master and the back end, as well. |
| 129 |
|
| 130 |
.. Note:: |
| 131 |
|
| 132 |
Do not set ``proxyservers`` on frontends. |
| 133 |
|
| 134 |
Exporting the database from the backend |
| 135 |
--------------------------------------- |
| 136 |
|
| 137 |
The existing mailboxes database must be exported to the MUPDATE Master. |
| 138 |
Use the :cyrusman:`ctl_mboxlist(8)` command to do so. For the first |
| 139 |
synchronization, change to the cyrus user, and run ``ctl_mboxlist -m``. |
| 140 |
|
| 141 |
.. Important:: |
| 142 |
One should use ``ctl_mboxlist -mw`` (dry run) first to be sure of |
| 143 |
understanding all the operations that this command will perform, |
| 144 |
since it does require that all mailboxes are unique in the murder |
| 145 |
namespace and could lead to deletions of conflicting mailboxes on |
| 146 |
other back ends already in the murder. |
| 147 |
|
| 148 |
If everything is configured properly, the mailbox database of the |
| 149 |
current host will upload to the mupdate master. If there are problems, |
| 150 |
the most likely cause is a misconfiguration of the authentication |
| 151 |
settings, or :cyrusman:`mupdate(8)` might not be running on the master. |
| 152 |
Using :cyrusman:`mupdatetest(1)` may be helpful in this case (it |
| 153 |
establishes an authenticated connection to the mupdate server, if it |
| 154 |
can). |
| 155 |
|
| 156 |
It is also useful to have the backends automatically resync the state |
| 157 |
of their local mailboxes database with the master on start up. This is |
| 158 |
configured by adding the following to the START section of |
| 159 |
:cyrusman:`cyrus.conf(5)` on the backends:: |
| 160 |
|
| 161 |
mupdatepush cmd="ctl_mboxlist -m" |
| 162 |
|
| 163 |
This will perform synchronization with the mupdate master each time the |
| 164 |
backend restarts, bringing the mupdate database up to date with the |
| 165 |
contents of the backend (and performing ACTIVATE and DELETES as needed |
| 166 |
to do so). |
| 167 |
|
| 168 |
.. Warning:: |
| 169 |
|
| 170 |
If somehow a mailbox exists on two (or more) backend servers, each |
| 171 |
time one of them synchronizes its database that backend server will |
| 172 |
become authoritative. Though this should not happen during normal |
| 173 |
operation of the murder (because of the consistency guarantees of |
| 174 |
the MUPDATE protocol, and the fact that mailbox operations are |
| 175 |
denied if the mupdate master is down), it is possible when first |
| 176 |
creating the mupdate database or when bringing a new backend server |
| 177 |
into the murder. |
| 178 |
|
| 179 |
Configuring the front ends |
| 180 |
-------------------------- |
| 181 |
|
| 182 |
Configuring the front ends is a two step process. First, define |
| 183 |
mupdate_server (and friends) as done for the backends above. However, |
| 184 |
as the frontends only talk to the mupdate master via a slave running on |
| 185 |
the local machine, also set up a slave on the same machine, in the |
| 186 |
SERVICES section of :cyrusman:`cyrus.conf(5)`, like so:: |
| 187 |
|
| 188 |
# mupdate database service - must prefork at least 1 |
| 189 |
mupdate cmd="mupdate" listen=3905 prefork=1 |
| 190 |
|
| 191 |
As this is a threaded service, prefork at least 1 so that the database |
| 192 |
synchronizes at startup. Otherwise, the service will not start running |
| 193 |
until after receiving a mupdate client connection to the slave (which |
| 194 |
is not a recommended configuration at this point). |
| 195 |
|
| 196 |
The front end SERVICES section should now look like this:: |
| 197 |
|
| 198 |
mupdate cmd="mupdate" listen=3905 prefork=1 |
| 199 |
|
| 200 |
imap cmd="imap" listen="imap" prefork=5 |
| 201 |
imaps cmd="imap -s" listen="imaps" prefork=1 |
| 202 |
pop3 cmd="pop3d" listen="pop3" prefork=0 |
| 203 |
pop3s cmd="pop3d -s" listen="pop3s" prefork=0 |
| 204 |
nntp cmd="nntpd" listen="nntp" prefork=0 |
| 205 |
nntps cmd="nntpd -s" listen="nntps" prefork=0 |
| 206 |
http cmd="httpd" listen="http" prefork=0 |
| 207 |
https cmd="httpd -s" listen="https" prefork=0 |
| 208 |
sieve cmd="timsieved" listen="sieve" prefork=0 |
| 209 |
lmtp cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=0 |
| 210 |
|
| 211 |
Note that timsieved does not need a proxy daemon, the managesieve |
| 212 |
protocol deals with the murder with referrals to the backends |
| 213 |
internally. |
| 214 |
|
| 215 |
Additionally, entries in :cyrusman:`imapd.conf(5)` are required to |
| 216 |
indicate the proxy auth name and passwords (if using a SASL mechanism |
| 217 |
that requires them) to the backends. |
| 218 |
|
| 219 |
For example, if the backends are ``mail1.andrew.cmu.edu`` and |
| 220 |
``mail2.andrew.cmu.edu`` with passwords of ``foo`` and ``bar``, and an |
| 221 |
auth name of ``mailproxy``:: |
| 222 |
|
| 223 |
mail1_password: foo |
| 224 |
mail2_password: bar |
| 225 |
proxy_authname: mailproxy |
| 226 |
|
| 227 |
For SASL mechanisms not using authnames or passwords (e.g. |
| 228 |
GSSAPI), the password options are not required. Note the use of |
| 229 |
the same authname as configured in the proxyservers line of the |
| 230 |
backend's :cyrusman:`imapd.conf(5)` above. |
| 231 |
|
| 232 |
Upon starting :cyrusman:`master(8)` on the frontend, the local |
| 233 |
mailboxes database should automatically synchronize with the contents |
| 234 |
of the MUPDATE master, and it's ready to go. Clients should connect to |
| 235 |
the frontends, and the frontends will proxy or refer as applicable to |
| 236 |
the backend servers. |
| 237 |
|
| 238 |
Additional backend configuration |
| 239 |
-------------------------------- |
| 240 |
|
| 241 |
Authentication system expecting usernames, passwords, etc, to |
| 242 |
authenticate, will also need to specify proxy_authname (and friends) in |
| 243 |
the backend imapd.confs. This is so the backends can authenticate to |
| 244 |
each other to facilitate mailbox moves. (Backend machines will need to |
| 245 |
be full admins). |
| 246 |
|
| 247 |
Delivering mail |
| 248 |
--------------- |
| 249 |
|
| 250 |
To deliver mail to a Murder, configure MTAs just as before, but instead |
| 251 |
of connecting directly to lmtpd on a back end, they should connect to |
| 252 |
lmtpproxyd on any front end. Remote MTAs may connect to the lmtpproxyd |
| 253 |
running on any front end machine (listening on a TCP socket), or |
| 254 |
install master and lmtpproxyd on your SMTP servers to connect via Unix |
| 255 |
domain socket. |
| 256 |
|
| 257 |
Administration |
| 258 |
============== |
| 259 |
|
| 260 |
Keeping the database synced |
| 261 |
--------------------------- |
| 262 |
|
| 263 |
Consistency in the database is maintained by pushing the current status |
| 264 |
of the backends to the master, and having the frontends stay up to date |
| 265 |
with the master's database. Since the frontends resync themselves |
| 266 |
entirely when they startup, downtime should not be a problem. |
| 267 |
(While they are up they should be continuously receiving database |
| 268 |
updates, as well when they lose connection to the master, they will |
| 269 |
try to reconnect and resync their database upon reconnection) |
| 270 |
|
| 271 |
Provided that the namespace of the backend servers is kept discrete |
| 272 |
(with no mailboxes existing on the same server), it is not a big deal |
| 273 |
to resync the mupdate master using ``ctl_mboxlist -m``. If two servers |
| 274 |
do have the same mailbox, this will need to be resolved before database |
| 275 |
consistency can be guaranteed. |
| 276 |
|
| 277 |
Moving Mailboxes between backends |
| 278 |
--------------------------------- |
| 279 |
|
| 280 |
There is currently no 100% foolproof way to do this, however, if you |
| 281 |
issue a rename command to a frontend (as you would to move a mailbox |
| 282 |
between partitions), and replace the partition name with the name of |
| 283 |
the new backend, it will move the mailbox to the indicated backend. You |
| 284 |
can also use the format ``backend.domain.com!partition`` to move to a |
| 285 |
specific partition (otherwise the default partition will be used). |
| 286 |
|
| 287 |
In cyradm, this looks like:: |
| 288 |
|
| 289 |
cyrus.andrew.cmu.edu> rename user.bcyrus user.bcyrus mail2.andrew.cmu.edu!u2 |
| 290 |
|
| 291 |
Note that since seen state is stored per-user, it is possible that when |
| 292 |
moving a shared mailbox users will have strange effects. The general |
| 293 |
rule is that moving an INBOX will move the entire user (including all |
| 294 |
sub-mailboxes to the INBOX, and seen state, and subscriptions, and |
| 295 |
sieve scripts, etc). The seen state is merged with the seen state on |
| 296 |
the new backend, so that no data is lost (seen state is also the only |
| 297 |
part left behind on the source backend). In the case of any other |
| 298 |
mailbox, however, only that individual mailbox is moved. If it is a |
| 299 |
quota root, the new quota root is instantiated on the new server, but |
| 300 |
otherwise quotas can appear to be violated, since each backend only |
| 301 |
takes care of its own quota. |
| 302 |
|
| 303 |
In general, it's better to leave trees of mailboxes on the same server, |
| 304 |
and not move submailboxes of inboxes between servers. |
| 305 |
|
| 306 |
Adding additional backend servers |
| 307 |
--------------------------------- |
| 308 |
|
| 309 |
This is very easy to do, simply configure an empty backend server and |
| 310 |
set its ``mupdate_server`` parameter to point at the mupdate master. |
| 311 |
Then, issue mailbox creates to it as you would any other backend |
| 312 |
server. |
| 313 |
|
| 314 |
Distributing Mailboxes between Back Ends |
| 315 |
---------------------------------------- |
| 316 |
|
| 317 |
Several options exist within :cyrusman:`imapd.conf(5)` to aid in the |
| 318 |
distribution of new users and mailboxes within a murder; across servers |
| 319 |
and partitions. We recommend exploring these: |
| 320 |
|
| 321 |
.. include:: /imap/reference/manpages/configs/imapd.conf.rst |
| 322 |
:start-after: startblob partition_select_mode |
| 323 |
:end-before: endblob partition_select_mode |
| 324 |
|
| 325 |
.. include:: /imap/reference/manpages/configs/imapd.conf.rst |
| 326 |
:start-after: startblob partition_select_exclude |
| 327 |
:end-before: endblob partition_select_exclude |
| 328 |
|
| 329 |
.. include:: /imap/reference/manpages/configs/imapd.conf.rst |
| 330 |
:start-after: startblob partition_select_usage_reinit |
| 331 |
:end-before: endblob partition_select_usage_reinit |
| 332 |
|
| 333 |
.. include:: /imap/reference/manpages/configs/imapd.conf.rst |
| 334 |
:start-after: startblob partition_select_soft_usage_limit |
| 335 |
:end-before: endblob partition_select_soft_usage_limit |
| 336 |
|
| 337 |
.. include:: /imap/reference/manpages/configs/imapd.conf.rst |
| 338 |
:start-after: startblob serverlist |
| 339 |
:end-before: endblob serverlist |
| 340 |
|
| 341 |
.. include:: /imap/reference/manpages/configs/imapd.conf.rst |
| 342 |
:start-after: startblob serverlist_select_mode |
| 343 |
:end-before: endblob serverlist_select_mode |
| 344 |
|
| 345 |
.. include:: /imap/reference/manpages/configs/imapd.conf.rst |
| 346 |
:start-after: startblob serverlist_select_usage_reinit |
| 347 |
:end-before: endblob serverlist_select_usage_reinit |
| 348 |
|
| 349 |
.. include:: /imap/reference/manpages/configs/imapd.conf.rst |
| 350 |
:start-after: startblob serverlist_select_soft_usage_limit |
| 351 |
:end-before: endblob serverlist_select_soft_usage_limit |
| 352 |
|
| 353 |
|
| 354 |
Backups |
| 355 |
------- |
| 356 |
|
| 357 |
.. :todo: |
| 358 |
xxx, need to write stuff. You don't need to really backup the data |
| 359 |
on the mupdate master or slaves, since this data can all be |
| 360 |
generated directly from the backends quite easily. |
| 361 |
|
| 362 |
Gotchas |
| 363 |
======= |
| 364 |
|
| 365 |
**Clients dealing with a pool of frontend servers** |
| 366 |
Some clients may not be terribly efficient caching connections to a |
| 367 |
pool of imap servers, this isn't a problem, as such, but it may |
| 368 |
mean that you will see many more authentications than you are used |
| 369 |
to. |
| 370 |
|
| 371 |
**Kerberos issues** |
| 372 |
If you are using kerberos authentication, you will want to ensure |
| 373 |
that all your machines are keyed properly, as we have seen problems |
| 374 |
with different clients trying to authenticate to different services |
| 375 |
(e.g. imap.imap-pool instead of imap.pool-frontend-1), so test the |
| 376 |
clients in use in your environment and be sure that they work with |
| 377 |
whatever keying scheme you use. |
| 378 |
|
| 379 |
**Clients dealing with referrals** |
| 380 |
Some clients (we've had particular trouble with pine, though most |
| 381 |
of these issues have now been resolved and new versions should be |
| 382 |
OK (that is, pine > 4.44), but as referrals have not been |
| 383 |
extensively used by any IMAP server until now, referrals are very |
| 384 |
likely to not work correctly or have surprising effects. |
| 385 |
|
| 386 |
**Clients dealing with getting a NO on LSUB commands** |
| 387 |
Some clients (Outlook, for example) may behave poorly if an LSUB |
| 388 |
command returns a NO, which may be the case if the backend server |
| 389 |
with the user's inbox is down. We have, for example, seen this |
| 390 |
result in the deletion of the disconnected message cache. |
| 391 |
|
| 392 |
**Behavior of cyradm / some mailbox operations** |
| 393 |
The behaviour of some administrative commands might be slightly |
| 394 |
unexpected. For example, you can only issue a SETQUOTA to a |
| 395 |
frontend server if the entire mailbox tree underneath where you are |
| 396 |
setting the quota exists on the same backend server, otherwise you |
| 397 |
will need to connect directly to the backend servers to perform the |
| 398 |
needed changes. Similarly, mailboxes will be created on the same |
| 399 |
backend server that their parent is in. In order to create them on |
| 400 |
a different server (or to create a new top level mailbox) you will |
| 401 |
need to connect directly to the desired backend server. |
| 402 |
|
| 403 |
**Subscriptions** |
| 404 |
If users want subscribe to a mailbox other than on their backend |
| 405 |
home server, they won't be able to, unless you set |
| 406 |
``allowallsubscribe: t`` in the backend imapd.confs. This |
| 407 |
essentially lets any string be subscribed to successfully. |
| 408 |
|
| 409 |
**Restarting the mupdate master** |
| 410 |
Because ``ctl_cyrusdb -r`` clears reservations on mailbox, if you |
| 411 |
restart the mupdate master (and run recovery), then this could (we |
| 412 |
suspect, very rarely) lead to inconsistencies in the mupdate |
| 413 |
database. |
| 414 |
|
| 415 |
Troubleshooting |
| 416 |
=============== |
| 417 |
|
| 418 |
**Mailbox operations are being denied** |
| 419 |
This is an indication that the mupdate master may be down. Restart |
| 420 |
it. |
| 421 |
|
| 422 |
**Mailbox operations are not being seen by one or more frontends** |
| 423 |
This indicates that the mupdate process on a slave may have died, |
| 424 |
you may need to restart master. Alternatively, mupdate will retry |
| 425 |
connections every 20 seconds or so for about 20 attempts if the |
| 426 |
master does go down. |
| 427 |
|
| 428 |
**A frontend's mailboxes.db is corrupt or out of sync** |
| 429 |
Restart master on the frontend, and have the mupdate process |
| 430 |
resynch the local database. You may need to remove the local |
| 431 |
mailboxes database if the corruption is extreme. |
| 432 |
|
| 433 |
**A mailbox's location keeps switching between two (or more) backend hosts.** |
| 434 |
It probably actually exists on both hosts. Delete the mailbox from |
| 435 |
all but one of the hosts, and run a ``ctl_mboxlist -m`` on the one |
| 436 |
where you want it to actually live. |
| 437 |
|
| 438 |
**Databases are never created on the frontends/slaves** |
| 439 |
Check to ensure that the mupdate slave process is started, (is |
| 440 |
prefork=1) |
| 441 |
|
| 442 |
**mupdate crashes with SIGSERV when using STARTTLS** |
| 443 |
The OpenSSL code in Cyrus Imap is for single-threaded applications and |
| 444 |
mupdate is a multi-threaded application. Do not encrypt the communication |
| 445 |
with mupdate. See also the discussion “SIGSEGV in cyrus-imapd 3.0.7 mupdate” |
| 446 |
on `cyrus-devel from July 2018 <https://lists.andrew.cmu.edu/pipermail/cyrus-devel/2018-July/>`_ |
| 447 |
and https://github.com/cyrusimap/cyrus-imapd/issues/2774 . |