Debian Codegraph @

doc/html/_sources/imap/reference/architecture.rst.txt

raw
1 .. _architecture:
2
3 ==================================
4 System Architecture
5 ==================================
6
7 High Level Architecture
8 =======================
9
10 A birds-eye view of Cyrus.
11
12 .. image:: images/architecture.jpg
13 :height: 400 px
14 :width: 636 px
15 :alt: High level architecture diagram.
16 :align: center
17
18 Mail is delivered over smtp to the MTA_ (Message/Mail Transfer Agent).
19 This then is delivered to **Cyrus imapd** over lmtp_ (Local Mail
20 Transfer Protocol).
21
22 Cyrus processes the inbound message and makes it available to the user
23 over POP3, IMAP or even NNTP. Cyrus does not provide outbound SMTP
24 services: these are hooked back into the MTA.
25
26 Cyrus usually uses **saslauthd** (Cyrus SASL) to provide authentication
27 services. It is not the only way to provide authentication, but it is
28 the usual option.
29
30 Cyrus stores the mailspool, indexes and search data on disk. While
31 these are inherently files, their structure and function is
32 database-centric and should be treated as such. *(Do not attempt to
33 manually edit these files. That way lies madness.)* Cyrus itself
34 provides no inherent backup capacity: this must be configured
35 externally using tools best suited for your environment.
36
37 For administrative actions on your server - such as creating users,
38 editing mailbox details, etc - use :cyrusman:`cyradm(8)`. This is a
39 command, not a daemon, and it communicates with cyrus imapd via the
40 IMAP protocol, and authenticating as an admin user.
41
42 For security between the user and cyrus, usually SSL is applied.
43
44 .. todo:
45 - undecided on whether to include idled in here. At the moment I've left it out.
46
47 .. _MTA: https://en.wikipedia.org/wiki/Message_transfer_agent
48 .. _lmtp: https://en.wikipedia.org/wiki/Local_Mail_Transfer_Protocol
49 .. _nginx: http://nginx.org/en/
50
51 .. _architecture_murder:
52
53 Cyrus Murder
54 ============
55
56 Cyrus Murder, or IMAP Aggregation, provides the ability to split a
57 common IMAP mailbox namespace(s) across multiple backends. Cyrus
58 murder is not :ref:`replication <architecture_replication>`; it is load
59 and resource sharing for performance.
60
61 .. image:: images/image2-murder.jpg
62 :height: 416 px
63 :width: 496 px
64 :alt: Cyrus Murder architecture.
65 :align: center
66
67 Consider a set of frontends (F1..Fn) which contain no user data. They
68 are essentially stateless. Users access mailboxes from any frontend.
69 Frontends are IMAP/POP3/Sieve/LMTP proxies.
70
71 There is also a set of backends (B1..Bn) which manage user data
72 stores. Unlike the frontends, each backends is unique: each holds a
73 subset of data. Backends are IMAP/POP3/Sieve/LMTP servers.
74
75 So when a user logs in to a frontend (say F1), how does F1 know from
76 which backend to fetch the user's data? This is where **mupdate**
77 comes in: the **mupdate** master holds the mapping of mailboxes to back
78 ends. Any time a backend has a change to their user data (i.e. mailbox
79 add/delete/move/subscribe), it sends the change to the **mupdate**
80 master which then notifies all the frontends of the latest mapping.
81
82 A user's data is not necessarily all stored on a single backend,
83 either: it can be split across disk partitions or backends.
84
85 The mapping on mupdate goes something like this:
86
87 =============== ===============
88 <user>.<folder> <backend>!<disk partition>
89 =============== ===============
90 u1 b2!default
91 u2 b2!default
92 u2.Trash b1!data
93 u3 b1!bigdisk
94 =============== ===============
95
96 Mupdate is multithreaded.
97
98 Should each datastore be configured to contain the complete set of all
99 data, and any frontend can also behave as a backend (and vice versa),
100 this is known as **Unified Murder**: where everything has everything.
101
102 See the :ref:`Cyrus Murder installation and admin guide <murder>`
103 for more information.
104
105 .. _architecture_replication:
106
107 Replication
108 ===========
109 Replication is not Aggregation or :ref:`Cyrus Murder<architecture_murder>`.
110 Replication provides high availability and hot backups. It is designed to
111 replicate the mailstore on a standalone Cyrus install, or multiple backend
112 servers in a :ref:`murder <architecture_murder>` configuration. It does
113 not replicate mupdate master servers: frontends have no state to replicate.
114
115 .. image:: images/image3-replication.jpg
116 :height: 385 px
117 :width: 507 px
118 :alt: Cyrus replication architecture
119 :align: center
120
121 A replication master is configured with a number of **channels**: each
122 channel defines the set of replicas the master is replicating to, and
123 the configuration (credentials on the replica host, port and hostname)
124 for how to communicate to that replica.
125
126 Each channel (ie: each replica) has its own set of log files on the
127 master, and its own **sync_client** instance which processes those log
128 files and sends them to the replica's **sync_server**.
129
130 When a replication master generates a change, it calls **sync_log**.
131 This writes the change to all the log files (one per channel). The log
132 files are rotated every few seconds. The $PID on the log file set is
133 that of the particular sync_client who processed the change.
134
135 Replication is idempotent: you can safely run the same log multiple times.
136
137 See the :ref:`Replication installation and admin guide <replication>`
138 for more information.
139
140 Channels
141 --------
142
143 A channel is a way of describing the linkage between a replication master and one
144 of its replicas. It encompasses the configuration on the master to know
145 which sync logs to write to, and the configuration on the master to
146 know the imap credentials on the replica to allow it to send details to
147 the replica. It is the port configuration on the replica to know where
148 to listen for change updates.
149
150 There are two standard channel configurations:
151
152 1. Single master keeping all replicas up to date.
153
154 2. Single master updates the replicas via chaining. (master updates
155 replica 1, which updates replica 2)
156
157 The only real benefit to chaining is bandwidth use reduction - if
158 you have two replicas in a different data centre, you can chain them and
159 avoid sending all the data over the link twice. You can always re-
160 establish replication to the second replica by creating a direct channel
161 and running ``sync_client -A`` to make sure everything is up-to-date.
162
163 Log file
164 --------
165
166 The log file is a list of users and/or mailboxes which have been
167 altered. When sync_log is enabled, all of the daemons which might
168 alter a mailbox or user will write a line to this log each time they do
169 so. That means the obvious suspects -- imapd, pop3d, timsieved, lmtpd,
170 etc. -- but also cyr_expire and friends.
171
172 So when sync_client processes a sync_log, it needs to look at an actual
173 copy of the user/mailbox in order to determine its current state, and
174 needs to look at both copies to work out what to replicate between
175 them. It uses sync_send_lookup to ask for the current state of an item,
176 and sync_response_parse to process the response to then make suitable
177 changes to perform the synchronisation.
178
179 Sync client supports doing a single user with '-u', a single mailbox
180 with '-m', etc. All the entries in the sync_log file are triggers to
181 replicate with that same value, so a line ``USER [email protected]`` is
182 the same as running ``sync_client -u [email protected]``.
183
184 .. _architecture_invoking:
185
186 Invoking Cyrus
187 ==============
188
189 Cyrus has two primary configuration files which control its operation:
190
191 1. /etc/imapd.conf
192 2. /etc/cyrus.conf
193
194 imapd.conf
195 ----------
196
197 Cyrus is invoked via ``imapd -C path/to/imapd.conf``.
198
199 The format of :cyrusman:`imapd.conf(5)` is in ``key: value`` pairs.
200
201 cyrus.conf
202 ----------
203
204 This is the config file for the Cyrus master process. The format of
205 :cyrusman:`cyrus.conf(5)` defines
206
207 1. **START**: What processes to begin once at startup,
208 2. **SERVICES**: Which daemons to spawn
209 3. **EVENTS**: What processes to periodically invoke, similar to cron.