| 1 |
.. _replication: |
| 2 |
|
| 3 |
============================================ |
| 4 |
Replication: Installation and Administration |
| 5 |
============================================ |
| 6 |
|
| 7 |
Architecture |
| 8 |
============ |
| 9 |
|
| 10 |
:ref:`Overall structure of replication <architecture_replication>`. |
| 11 |
|
| 12 |
Terminology |
| 13 |
=========== |
| 14 |
Host |
| 15 |
A computer chassis, slot or cabinet. A machine. |
| 16 |
|
| 17 |
Server or Service |
| 18 |
A program or process which "serves" a particular protocol. |
| 19 |
Typically a server will listen on a network port or Unix socket |
| 20 |
for communications from a client (local or remote). |
| 21 |
|
| 22 |
Client |
| 23 |
A program or process which talks to a server or servers for a given |
| 24 |
protocol. The initiator of a client/server session. |
| 25 |
|
| 26 |
Instance |
| 27 |
A particular instance or iteration of a program, which may be one, |
| 28 |
or one of many, providing similar services to different consumers. |
| 29 |
|
| 30 |
Master |
| 31 |
In this document, Master always means the source of data to be |
| 32 |
replicated. |
| 33 |
|
| 34 |
Replica |
| 35 |
The target of data replication is the Replica, which refers both to |
| 36 |
an instance of sync_server and to the resultant dataset. |
| 37 |
|
| 38 |
Operating Modes |
| 39 |
=============== |
| 40 |
|
| 41 |
Cyrus replication supports two modes of operation: Rolling and |
| 42 |
Periodic. The difference is that rolling replication is a more or less |
| 43 |
continuous process whereas periodic replication occurs on demand, |
| 44 |
triggered by some manual or automated process such as |
| 45 |
:manpage:`cron(8)`. |
| 46 |
|
| 47 |
Rolling Replication |
| 48 |
------------------- |
| 49 |
|
| 50 |
Rolling replication is enabled by setting ``sync_log`` to True in |
| 51 |
:cyrusman:`imapd.conf(5)`. With ``sync_log: true``, any process which |
| 52 |
alters the mail spool will update the ``sync_log`` files with details |
| 53 |
as to which mailbox(es) or users have been affected by their actions. |
| 54 |
In this way the ``sync_log`` acts as a command file for the |
| 55 |
:cyrusman:`sync_client(8)` process(es). |
| 56 |
|
| 57 |
The log files are stored in ``{configdirectory}/sync/log`` for single |
| 58 |
channel systems (see :ref:`replication-channels` for more information) |
| 59 |
and are rotated on a regular basis by Cyrus. Multi-channel deployments |
| 60 |
will have a separate ``sync_log`` file for each, stored as |
| 61 |
``{configdirectory}/sync/<channel>/log``. |
| 62 |
|
| 63 |
Upon completing a log file, ``sync_client`` will go to sleep, or, if |
| 64 |
processing took longer than ``sync_repeat_interval`` seconds, will |
| 65 |
start over again on the next log. |
| 66 |
|
| 67 |
.. Note:: |
| 68 |
Any unsuccessful run of sync_client will result in the incomplete |
| 69 |
remains of the original log file being left behind as "log-<$PID>". |
| 70 |
This may be re-run as needed. |
| 71 |
|
| 72 |
.. Note:: |
| 73 |
Please also see :ref:`below for other uses of sync_log |
| 74 |
<replication-other-uses>`. |
| 75 |
|
| 76 |
Periodic Replication |
| 77 |
-------------------- |
| 78 |
|
| 79 |
With ``sync_log`` set to the default False, replication must be |
| 80 |
triggered either by manually running :cyrusman:`sync_client(8)`, or by |
| 81 |
doing so via ``cron`` or an entry in :cyrusman:`cyrus.conf(5)`. |
| 82 |
|
| 83 |
In either event, command line switches control the operation of |
| 84 |
``sync_client``. |
| 85 |
|
| 86 |
Once the process completes its work, it will exit. |
| 87 |
|
| 88 |
Idempotency |
| 89 |
----------- |
| 90 |
|
| 91 |
Synchronization itself is idempotent in either mode, so log files may |
| 92 |
be "replayed" without concern of damage to the replica's mail spools. |
| 93 |
|
| 94 |
Sync Chains |
| 95 |
----------- |
| 96 |
|
| 97 |
Cyrus supports chained replication, in which one replica replicates to |
| 98 |
another. I.e. A replicates to B; B replicates to C. If you wish to |
| 99 |
use this approach, please see the ``sync_log_chain`` setting: |
| 100 |
|
| 101 |
.. include:: /imap/reference/manpages/configs/imapd.conf.rst |
| 102 |
:start-after: startblob sync_log_chain |
| 103 |
:end-before: endblob sync_log_chain |
| 104 |
|
| 105 |
Note that sync_log_chain is to be set on the middle server(s) in a |
| 106 |
chain, not on the first or last. |
| 107 |
|
| 108 |
Transport |
| 109 |
========= |
| 110 |
|
| 111 |
Older versions (pre-3.0) of Cyrus used the dedicated ``csync`` |
| 112 |
transport -- typically over TCP port 2005 -- and server process -- |
| 113 |
:cyrusman:`sync_server(8)` -- for replication. This is no longer |
| 114 |
necessary. |
| 115 |
|
| 116 |
From v3.0 forward, the :cyrusman:`sync_client(8)` will default to using |
| 117 |
IMAP protocol for transport, and an IMAP instance on the replica will |
| 118 |
process the synchronization instructions. If you wish, you may |
| 119 |
override this by setting the ``sync_try_imap`` setting in |
| 120 |
:cyrusman:`imapd.conf(5)` to False. |
| 121 |
|
| 122 |
.. include:: /imap/reference/manpages/configs/imapd.conf.rst |
| 123 |
:start-after: startblob sync_try_imap |
| 124 |
:end-before: endblob sync_try_imap |
| 125 |
|
| 126 |
Installation |
| 127 |
============ |
| 128 |
|
| 129 |
One must :ref:`build Cyrus IMAPd <compiling>` with the |
| 130 |
``--enable-replication`` configure option. This builds the replication |
| 131 |
client/server applications and utilities. |
| 132 |
|
| 133 |
.. Note:: |
| 134 |
Those using their distribution's packages may need to install a |
| 135 |
separate package for replication support. For example, on Debian |
| 136 |
and derived distros, install the ``cyrus-replication`` package. |
| 137 |
|
| 138 |
Requirements |
| 139 |
------------ |
| 140 |
|
| 141 |
1. At least one Cyrus IMAP server instance to be the **master**. |
| 142 |
2. At least one Cyrus IMAP server instance to be the **replica**. |
| 143 |
|
| 144 |
.. Note:: |
| 145 |
Sample configurations for both "master" and "replica" instances are |
| 146 |
included in the standard distribution. |
| 147 |
|
| 148 |
Replica server configuration |
| 149 |
---------------------------- |
| 150 |
|
| 151 |
The **replica** is a standalone server instance which listens for and |
| 152 |
processes synchronization messages from a single **master** server. The |
| 153 |
replica server needs to be configured to accept synchronization |
| 154 |
messages via IMAP or the (deprecated) :cyrusman:`sync_server(8)` |
| 155 |
process. |
| 156 |
|
| 157 |
.. Important:: |
| 158 |
Within a Cyrus :ref:`Murder <architecture_murder>` environment, |
| 159 |
replicas must **not** be configured to invoke |
| 160 |
:cyrusman:`ctl_mboxlist(8)` on startup (pushing the local mailbox |
| 161 |
list to the **Mupdate Master**). This may only be done on the |
| 162 |
Master instance. |
| 163 |
|
| 164 |
.. Important:: |
| 165 |
It is critically important that all partitions existing on a master |
| 166 |
server instance also be present on all replica instances. Failure |
| 167 |
to ensure this will result in crashes and could lead to lost data |
| 168 |
on the replicas. |
| 169 |
|
| 170 |
1. :ref:`Configure a standalone server <installing>`. |
| 171 |
|
| 172 |
2. If using the deprecated sync_server scheme, add the following line |
| 173 |
to the ``/etc/services`` file. Note that the port number is |
| 174 |
arbitrary as long as its not being used by any other services on the |
| 175 |
network. |
| 176 |
|
| 177 |
:: |
| 178 |
|
| 179 |
csync 2005/tcp |
| 180 |
|
| 181 |
3. If using the deprecated sync_server scheme, add a line similar to |
| 182 |
the following in the SERVICES section of :cyrusman:`cyrus.conf(5)`: |
| 183 |
|
| 184 |
:: |
| 185 |
|
| 186 |
syncserver cmd="/usr/cyrus/bin/sync_server" listen="csync" |
| 187 |
|
| 188 |
4. Start/restart ``/usr/cyrus/bin/master``. |
| 189 |
|
| 190 |
Master server configuration |
| 191 |
--------------------------- |
| 192 |
|
| 193 |
The **master** server is a standalone or backend Cyrus IMAP server |
| 194 |
instance which is actively serving mailboxes to clients. This server |
| 195 |
needs to be configured to synchronize its mailstore with a **replica** |
| 196 |
server via an instance of :cyrusman:`sync_client(8)`. |
| 197 |
|
| 198 |
If using the deprecated sync_server scheme, add the following line to |
| 199 |
the ``/etc/services`` file. |
| 200 |
|
| 201 |
:: |
| 202 |
|
| 203 |
csync 2005/tcp |
| 204 |
|
| 205 |
.. Note:: |
| 206 |
The port number **MUST** be the same as that used on the replica |
| 207 |
server. |
| 208 |
|
| 209 |
Specify the hostname of the replica server and how to authenticate to |
| 210 |
it in :cyrusman:`imapd.conf(5)` using these options: |
| 211 |
|
| 212 |
* sync_host |
| 213 |
* sync_port |
| 214 |
* sync_authname |
| 215 |
* sync_realm |
| 216 |
* sync_password |
| 217 |
|
| 218 |
.. Note:: |
| 219 |
``sync_authname`` **MUST** be an ``admin`` user on the replica. |
| 220 |
|
| 221 |
.. Note:: |
| 222 |
``sync_realm`` and ``sync_password`` may not be necessary |
| 223 |
depending on the SASL mechanism used for authentication. |
| 224 |
|
| 225 |
.. Note:: |
| 226 |
See :ref:`replication-channels`, below, for details on how to use |
| 227 |
these settings to control syncing to multiple replicas. |
| 228 |
|
| 229 |
.. Important:: |
| 230 |
If using sync_log_channels for any other purpose, such as |
| 231 |
specifying the sync_log used by :cyrusman:`squatter(8)` command, |
| 232 |
you must *also* either specify a sync_log channel for replication, |
| 233 |
or specify the default "" (the two-character string U+22 U+22). |
| 234 |
|
| 235 |
Add invocation specifications to :cyrusman:`cyrus.conf(5)` to spawn |
| 236 |
:cyrusman:`sync_client(8)` as desired (for each channel used) as |
| 237 |
described below in Rolling Replication or Periodic Replication. |
| 238 |
|
| 239 |
Compression |
| 240 |
----------- |
| 241 |
|
| 242 |
If one runs replication over a WAN link, the trade-off between |
| 243 |
bandwidth and CPU usage will tilt strongly in favour of enabling |
| 244 |
compression to save bandwidth at a slight increase in CPU cost. Set |
| 245 |
the ``sync_compress`` value in :cyrusman:`imapd.conf(5)`:: |
| 246 |
|
| 247 |
sync_compress: On |
| 248 |
|
| 249 |
or pass the ``-z`` flag to :cyrusman:`sync_client(8)` in the service |
| 250 |
spec in :cyrusman:`cyrus.conf(5)`:: |
| 251 |
|
| 252 |
syncclient cmd="/usr/cyrus/bin/sync_client -r -z" |
| 253 |
|
| 254 |
Rolling Replication Configuration |
| 255 |
--------------------------------- |
| 256 |
|
| 257 |
**Rolling Replication** means that the master instance continuously |
| 258 |
synchronizes itself with a replica. |
| 259 |
|
| 260 |
To configure rolling replication, perform the following: |
| 261 |
|
| 262 |
1. Enable the ``sync_log`` option in :cyrusman:`imapd.conf(5)`. This |
| 263 |
allows the imapd, pop3d, nntpd, and lmtpd services to log |
| 264 |
synchronization actions which will be periodically serviced by |
| 265 |
sync_client:: |
| 266 |
|
| 267 |
sync_log: On |
| 268 |
|
| 269 |
2. Optionally, adjust the ``sync_repeat_interval`` in |
| 270 |
:cyrusman:`imapd.conf(5)`:: |
| 271 |
|
| 272 |
sync_repeat_interval: 300 |
| 273 |
|
| 274 |
3. Add a line similar to the following in the STARTUP section of |
| 275 |
:cyrusman:`cyrus.conf(5)`:: |
| 276 |
|
| 277 |
syncclient cmd="/usr/cyrus/bin/sync_client -r" |
| 278 |
|
| 279 |
Start/restart ``/usr/cyrus/bin/master``. |
| 280 |
|
| 281 |
.. Hint:: |
| 282 |
In a multi-channel mesh, the channel to be used by a given |
| 283 |
sync_client must be specified via the "-n <channel>" argument on |
| 284 |
the command line:: |
| 285 |
|
| 286 |
syncclient cmd="/usr/cyrus/bin/sync_client -r -n channel1" |
| 287 |
|
| 288 |
Terminating Rolling Replication |
| 289 |
------------------------------- |
| 290 |
|
| 291 |
To be able to stop rolling replication at any time, configure the |
| 292 |
``sync_shutdown_file`` option in :cyrusman:`imapd.conf(5)` to point to |
| 293 |
a non-existant file, the appearance of this file will trigger a |
| 294 |
shutdown of a :cyrusman:`sync_client(8)` instance:: |
| 295 |
|
| 296 |
sync_shutdown_file: /var/lib/imap/syncstop |
| 297 |
|
| 298 |
Tweaking Rolling Replication |
| 299 |
---------------------------- |
| 300 |
|
| 301 |
The default frequency of replication runs is 3 seconds. Lengthening |
| 302 |
this produces higher efficiency at the cost of slightly more stale data |
| 303 |
on the replica. Alter this via the sync_repeat_interval in |
| 304 |
:cyrusman:`imapd.conf(5)` or by using the "-d" argument in the |
| 305 |
invocation of :cyrusman:`sync_client(8)`. |
| 306 |
|
| 307 |
Periodic Replication Configuration |
| 308 |
---------------------------------- |
| 309 |
|
| 310 |
In Periodic Replication the sync_client instance must be spawned |
| 311 |
from time to time, causing replication to start at that time. This may |
| 312 |
be handled via a :manpage:`cron(8)` job, or by adding an entry to the |
| 313 |
EVENTS section of :cyrusman:`cyrus.conf(5)` like any of these:: |
| 314 |
|
| 315 |
EVENTS { |
| 316 |
<...> |
| 317 |
# Periodically sync ALL user mailboxes every 4 hours |
| 318 |
syncclient cmd="/usr/cyrus/bin/sync_client -A" period=240 |
| 319 |
|
| 320 |
# Periodically sync changes at specific times |
| 321 |
syncclient cmd="/usr/cyrus/bin/sync_client -A" at=0800 |
| 322 |
syncclient cmd="/usr/cyrus/bin/sync_client -A" at=1200 |
| 323 |
syncclient cmd="/usr/cyrus/bin/sync_client -A" at=1800 |
| 324 |
<...> |
| 325 |
} |
| 326 |
|
| 327 |
.. Note:: |
| 328 |
When using the "-A" flag (sync all users) no non-user |
| 329 |
mailboxes are synced. As the man page :cyrusman:`imapd.conf(5)` |
| 330 |
notes, "... this could be considered a bug and maybe it should do |
| 331 |
those mailboxes independently." |
| 332 |
|
| 333 |
Tweaking Replication |
| 334 |
-------------------- |
| 335 |
|
| 336 |
You may control the number of messages replicated in each batch, via |
| 337 |
the ``sync_batchsize`` setting: |
| 338 |
|
| 339 |
.. include:: /imap/reference/manpages/configs/imapd.conf.rst |
| 340 |
:start-after: startblob sync_batchsize |
| 341 |
:end-before: endblob sync_batchsize |
| 342 |
|
| 343 |
.. _replication-channels: |
| 344 |
|
| 345 |
Channels |
| 346 |
======== |
| 347 |
|
| 348 |
The Cyrus replication scheme is very flexible, and supports meshes in |
| 349 |
which masters running on various hosts may replicate to instances on |
| 350 |
other hosts. This is achieved by use of the Channels feature of the |
| 351 |
replication system. |
| 352 |
|
| 353 |
To employ channels, prefix any of the following sync\_ configuration |
| 354 |
options in :cyrusman:`imapd.conf(5)` with the channel name and an |
| 355 |
underscore "_" character as needed:: |
| 356 |
|
| 357 |
sync_authname |
| 358 |
sync_password |
| 359 |
sync_realm |
| 360 |
sync_host |
| 361 |
sync_port |
| 362 |
sync_repeat_interval |
| 363 |
sync_shutdown_file |
| 364 |
|
| 365 |
Then add the setting ``sync_log_channels`` with a list of the channels:: |
| 366 |
|
| 367 |
sync_log_channels: chan1 chan2 chan3 |
| 368 |
|
| 369 |
For example, a site using the same auth credentials for all servers has |
| 370 |
no need to specify unique per-channel settings for ``sync_authname``, |
| 371 |
``sync_password`` or ``sync_realm``, but might do the following for the |
| 372 |
rest of the sync related settings in :cyrusman:`imapd.conf(5)`:: |
| 373 |
|
| 374 |
sync_authname: replman |
| 375 |
sync_password: <secret> |
| 376 |
sync_log_channels: repl1 repl2 offsite |
| 377 |
## |
| 378 |
# The main replica |
| 379 |
repl1_sync_host: mailrepl1.example.org |
| 380 |
repl1_sync_repeat_interval: 180 |
| 381 |
repl1_shutdown_file: /run/cyrus/sync/repl1_shutdown |
| 382 |
## |
| 383 |
# A second replica used to feed the tape backup system |
| 384 |
repl2_sync_host: mailrepl2.example.org |
| 385 |
repl2_sync_repeat_interval: 180 |
| 386 |
repl2_shutdown_file: /run/cyrus/sync/repl2_shutdown |
| 387 |
## |
| 388 |
# An offsite replica which needs a different port and uses a slower |
| 389 |
# cycle rate |
| 390 |
offsite_sync_port: 19205 |
| 391 |
offsite_sync_host: mailoffsite.example.org |
| 392 |
offsite_sync_repeat_interval: 360 |
| 393 |
offsite_shutdown_file: /run/cyrus/sync/offsite_shutdown |
| 394 |
|
| 395 |
Then these entries in :cyrusman:`cyrus.conf(5)` would complete the |
| 396 |
exercise:: |
| 397 |
|
| 398 |
repl1sync cmd="/usr/cyrus/bin/sync_client -r -n repl1" |
| 399 |
repl2sync cmd="/usr/cyrus/bin/sync_client -r -n repl2" |
| 400 |
offsitesync cmd="/usr/cyrus/bin/sync_client -r -n offsite" |
| 401 |
|
| 402 |
Again, this is just an example for illustration. The system provides so |
| 403 |
much flexibility, and one can combine channels with chaining to achieve |
| 404 |
even more. |
| 405 |
|
| 406 |
.. _replication-other-uses: |
| 407 |
|
| 408 |
Other Considerations |
| 409 |
==================== |
| 410 |
|
| 411 |
.. Important:: |
| 412 |
This section is currently under development. If you believe you |
| 413 |
are impacted by these considerations, please check back with each |
| 414 |
release and follow the mailing list. |
| 415 |
|
| 416 |
The infrastructure provided by ``sync_log`` has now been leveraged by |
| 417 |
the Rolling Indexing capability introduced in v3.0. See |
| 418 |
:cyrusman:`squatter(8)` for more details (see the fourth mode synopsis). |
| 419 |
|
| 420 |
Specifically, the following new settings have been added to |
| 421 |
:cyrusman:`imapd.conf(5)` in support of this new use of ``sync_log``: |
| 422 |
|
| 423 |
.. include:: /imap/reference/manpages/configs/imapd.conf.rst |
| 424 |
:start-after: startblob sync_log_unsuppressable_channels |
| 425 |
:end-before: endblob sync_log_unsuppressable_channels |
| 426 |
|
| 427 |
Administration |
| 428 |
============== |
| 429 |
|
| 430 |
Manual replication |
| 431 |
------------------ |
| 432 |
|
| 433 |
To manually synchronize any part of the mailstore, run |
| 434 |
:cyrusman:`sync_client(8)` with the appropriate command line options. |
| 435 |
Note that manual synchronization DOES NOT interfere with rolling |
| 436 |
replication. |
| 437 |
|
| 438 |
For example: |
| 439 |
|
| 440 |
:: |
| 441 |
|
| 442 |
[root@skynet ~]# /usr/lib/cyrus-imapd/sync_client -S cyrus-replica.example.org -v -u [email protected] |
| 443 |
USER john^[email protected] |
| 444 |
|
| 445 |
One can run :cyrusman:`cyr_synclog(8)` instead, which will insert the |
| 446 |
record into the rolling replication log. |
| 447 |
|
| 448 |
Failover |
| 449 |
-------- |
| 450 |
|
| 451 |
.. :todo: |
| 452 |
Hmm! How does failover work? |
| 453 |
Clue: It's not automated (yet)... |