| 1 |
.. _architecture: |
| 2 |
|
| 3 |
================================== |
| 4 |
System Architecture |
| 5 |
================================== |
| 6 |
|
| 7 |
High Level Architecture |
| 8 |
======================= |
| 9 |
|
| 10 |
A birds-eye view of Cyrus. |
| 11 |
|
| 12 |
.. image:: images/architecture.jpg |
| 13 |
:height: 400 px |
| 14 |
:width: 636 px |
| 15 |
:alt: High level architecture diagram. |
| 16 |
:align: center |
| 17 |
|
| 18 |
Mail is delivered over smtp to the MTA_ (Message/Mail Transfer Agent). |
| 19 |
This then is delivered to **Cyrus imapd** over lmtp_ (Local Mail |
| 20 |
Transfer Protocol). |
| 21 |
|
| 22 |
Cyrus processes the inbound message and makes it available to the user |
| 23 |
over POP3, IMAP or even NNTP. Cyrus does not provide outbound SMTP |
| 24 |
services: these are hooked back into the MTA. |
| 25 |
|
| 26 |
Cyrus usually uses **saslauthd** (Cyrus SASL) to provide authentication |
| 27 |
services. It is not the only way to provide authentication, but it is |
| 28 |
the usual option. |
| 29 |
|
| 30 |
Cyrus stores the mailspool, indexes and search data on disk. While |
| 31 |
these are inherently files, their structure and function is |
| 32 |
database-centric and should be treated as such. *(Do not attempt to |
| 33 |
manually edit these files. That way lies madness.)* Cyrus itself |
| 34 |
provides no inherent backup capacity: this must be configured |
| 35 |
externally using tools best suited for your environment. |
| 36 |
|
| 37 |
For administrative actions on your server - such as creating users, |
| 38 |
editing mailbox details, etc - use :cyrusman:`cyradm(8)`. This is a |
| 39 |
command, not a daemon, and it communicates with cyrus imapd via the |
| 40 |
IMAP protocol, and authenticating as an admin user. |
| 41 |
|
| 42 |
For security between the user and cyrus, usually SSL is applied. |
| 43 |
|
| 44 |
.. todo: |
| 45 |
- undecided on whether to include idled in here. At the moment I've left it out. |
| 46 |
|
| 47 |
.. _MTA: https://en.wikipedia.org/wiki/Message_transfer_agent |
| 48 |
.. _lmtp: https://en.wikipedia.org/wiki/Local_Mail_Transfer_Protocol |
| 49 |
.. _nginx: http://nginx.org/en/ |
| 50 |
|
| 51 |
.. _architecture_murder: |
| 52 |
|
| 53 |
Cyrus Murder |
| 54 |
============ |
| 55 |
|
| 56 |
Cyrus Murder, or IMAP Aggregation, provides the ability to split a |
| 57 |
common IMAP mailbox namespace(s) across multiple backends. Cyrus |
| 58 |
murder is not :ref:`replication <architecture_replication>`; it is load |
| 59 |
and resource sharing for performance. |
| 60 |
|
| 61 |
.. image:: images/image2-murder.jpg |
| 62 |
:height: 416 px |
| 63 |
:width: 496 px |
| 64 |
:alt: Cyrus Murder architecture. |
| 65 |
:align: center |
| 66 |
|
| 67 |
Consider a set of frontends (F1..Fn) which contain no user data. They |
| 68 |
are essentially stateless. Users access mailboxes from any frontend. |
| 69 |
Frontends are IMAP/POP3/Sieve/LMTP proxies. |
| 70 |
|
| 71 |
There is also a set of backends (B1..Bn) which manage user data |
| 72 |
stores. Unlike the frontends, each backends is unique: each holds a |
| 73 |
subset of data. Backends are IMAP/POP3/Sieve/LMTP servers. |
| 74 |
|
| 75 |
So when a user logs in to a frontend (say F1), how does F1 know from |
| 76 |
which backend to fetch the user's data? This is where **mupdate** |
| 77 |
comes in: the **mupdate** master holds the mapping of mailboxes to back |
| 78 |
ends. Any time a backend has a change to their user data (i.e. mailbox |
| 79 |
add/delete/move/subscribe), it sends the change to the **mupdate** |
| 80 |
master which then notifies all the frontends of the latest mapping. |
| 81 |
|
| 82 |
A user's data is not necessarily all stored on a single backend, |
| 83 |
either: it can be split across disk partitions or backends. |
| 84 |
|
| 85 |
The mapping on mupdate goes something like this: |
| 86 |
|
| 87 |
=============== =============== |
| 88 |
<user>.<folder> <backend>!<disk partition> |
| 89 |
=============== =============== |
| 90 |
u1 b2!default |
| 91 |
u2 b2!default |
| 92 |
u2.Trash b1!data |
| 93 |
u3 b1!bigdisk |
| 94 |
=============== =============== |
| 95 |
|
| 96 |
Mupdate is multithreaded. |
| 97 |
|
| 98 |
Should each datastore be configured to contain the complete set of all |
| 99 |
data, and any frontend can also behave as a backend (and vice versa), |
| 100 |
this is known as **Unified Murder**: where everything has everything. |
| 101 |
|
| 102 |
See the :ref:`Cyrus Murder installation and admin guide <murder>` |
| 103 |
for more information. |
| 104 |
|
| 105 |
.. _architecture_replication: |
| 106 |
|
| 107 |
Replication |
| 108 |
=========== |
| 109 |
Replication is not Aggregation or :ref:`Cyrus Murder<architecture_murder>`. |
| 110 |
Replication provides high availability and hot backups. It is designed to |
| 111 |
replicate the mailstore on a standalone Cyrus install, or multiple backend |
| 112 |
servers in a :ref:`murder <architecture_murder>` configuration. It does |
| 113 |
not replicate mupdate master servers: frontends have no state to replicate. |
| 114 |
|
| 115 |
.. image:: images/image3-replication.jpg |
| 116 |
:height: 385 px |
| 117 |
:width: 507 px |
| 118 |
:alt: Cyrus replication architecture |
| 119 |
:align: center |
| 120 |
|
| 121 |
A replication master is configured with a number of **channels**: each |
| 122 |
channel defines the set of replicas the master is replicating to, and |
| 123 |
the configuration (credentials on the replica host, port and hostname) |
| 124 |
for how to communicate to that replica. |
| 125 |
|
| 126 |
Each channel (ie: each replica) has its own set of log files on the |
| 127 |
master, and its own **sync_client** instance which processes those log |
| 128 |
files and sends them to the replica's **sync_server**. |
| 129 |
|
| 130 |
When a replication master generates a change, it calls **sync_log**. |
| 131 |
This writes the change to all the log files (one per channel). The log |
| 132 |
files are rotated every few seconds. The $PID on the log file set is |
| 133 |
that of the particular sync_client who processed the change. |
| 134 |
|
| 135 |
Replication is idempotent: you can safely run the same log multiple times. |
| 136 |
|
| 137 |
See the :ref:`Replication installation and admin guide <replication>` |
| 138 |
for more information. |
| 139 |
|
| 140 |
Channels |
| 141 |
-------- |
| 142 |
|
| 143 |
A channel is a way of describing the linkage between a replication master and one |
| 144 |
of its replicas. It encompasses the configuration on the master to know |
| 145 |
which sync logs to write to, and the configuration on the master to |
| 146 |
know the imap credentials on the replica to allow it to send details to |
| 147 |
the replica. It is the port configuration on the replica to know where |
| 148 |
to listen for change updates. |
| 149 |
|
| 150 |
There are two standard channel configurations: |
| 151 |
|
| 152 |
1. Single master keeping all replicas up to date. |
| 153 |
|
| 154 |
2. Single master updates the replicas via chaining. (master updates |
| 155 |
replica 1, which updates replica 2) |
| 156 |
|
| 157 |
The only real benefit to chaining is bandwidth use reduction - if |
| 158 |
you have two replicas in a different data centre, you can chain them and |
| 159 |
avoid sending all the data over the link twice. You can always re- |
| 160 |
establish replication to the second replica by creating a direct channel |
| 161 |
and running ``sync_client -A`` to make sure everything is up-to-date. |
| 162 |
|
| 163 |
Log file |
| 164 |
-------- |
| 165 |
|
| 166 |
The log file is a list of users and/or mailboxes which have been |
| 167 |
altered. When sync_log is enabled, all of the daemons which might |
| 168 |
alter a mailbox or user will write a line to this log each time they do |
| 169 |
so. That means the obvious suspects -- imapd, pop3d, timsieved, lmtpd, |
| 170 |
etc. -- but also cyr_expire and friends. |
| 171 |
|
| 172 |
So when sync_client processes a sync_log, it needs to look at an actual |
| 173 |
copy of the user/mailbox in order to determine its current state, and |
| 174 |
needs to look at both copies to work out what to replicate between |
| 175 |
them. It uses sync_send_lookup to ask for the current state of an item, |
| 176 |
and sync_response_parse to process the response to then make suitable |
| 177 |
changes to perform the synchronisation. |
| 178 |
|
| 179 |
Sync client supports doing a single user with '-u', a single mailbox |
| 180 |
with '-m', etc. All the entries in the sync_log file are triggers to |
| 181 |
replicate with that same value, so a line ``USER [email protected]`` is |
| 182 |
the same as running ``sync_client -u [email protected]``. |
| 183 |
|
| 184 |
.. _architecture_invoking: |
| 185 |
|
| 186 |
Invoking Cyrus |
| 187 |
============== |
| 188 |
|
| 189 |
Cyrus has two primary configuration files which control its operation: |
| 190 |
|
| 191 |
1. /etc/imapd.conf |
| 192 |
2. /etc/cyrus.conf |
| 193 |
|
| 194 |
imapd.conf |
| 195 |
---------- |
| 196 |
|
| 197 |
Cyrus is invoked via ``imapd -C path/to/imapd.conf``. |
| 198 |
|
| 199 |
The format of :cyrusman:`imapd.conf(5)` is in ``key: value`` pairs. |
| 200 |
|
| 201 |
cyrus.conf |
| 202 |
---------- |
| 203 |
|
| 204 |
This is the config file for the Cyrus master process. The format of |
| 205 |
:cyrusman:`cyrus.conf(5)` defines |
| 206 |
|
| 207 |
1. **START**: What processes to begin once at startup, |
| 208 |
2. **SERVICES**: Which daemons to spawn |
| 209 |
3. **EVENTS**: What processes to periodically invoke, similar to cron. |