Debian Codegraph @

doc/html/_sources/imap/download/upgrade.rst.txt

raw
1 .. highlight:: none
2
3 .. _upgrade:
4
5 =================
6 Upgrading to 3.12
7 =================
8
9 .. note::
10
11 This guide assumes that you are familiar and comfortable with administration
12 of a Cyrus installation, and system administration in general.
13
14 It assumes you are installing from source or tarball. If you want to install
15 from package, use the upgrade instructions from the package provider.
16
17 .. contents:: Upgrading: an overview
18 :local:
19
20 1. Preparation
21 --------------
22
23 Things to consider **before** you begin:
24
25 Versions to upgrade from
26 ########################
27
28 Before upgrading to 3.12, your deployment should be running
29 **3.10.1 (or later)**
30
31 If your existing deployment is older than this, you should first upgrade
32 to 3.10.1, let it run for a while, resolve any issues that come up, and only
33 then upgrade to 3.12.
34
35 Installation from tarball
36 #########################
37
38 You will need to install from our packaged tarball. We provide a full list of
39 libraries that Debian requires, but we aren't able to test all platforms: you
40 may find you need to install additional or different libraries to support 3.12.
41
42 JMAP/CalDAV changes
43 ###################
44
45 .. _upgrade_email_query_reindex:
46
47 New JMAP Email/query filter conditions
48 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
49
50 3.12 adds the JMAP Email/query filter conditions ``messageId``, ``references``,
51 and ``inReplyTo``.
52
53 It is recommended to rebuild the Xapian index with :cyrusman:`squatter(8)` to
54 make use of these filter conditions. Otherwise, email queries having these
55 filters fall back to reading the MIME headers from disk, resulting in slower
56 search.
57
58 .. _upgrade_pcre2_support:
59
60 PCRE2 support
61 #############
62
63 Cyrus 3.12 will prefer PCRE2 over PCRE if both are installed. If you have both
64 installed and wish to use PCRE rather than PCRE2, run configure with
65 ``--disable-pcre2``.
66
67 If you haven't specifically installed libpcre2-dev (or whatever your system's
68 equivalent is), you might still have parts of pcre2 installed due to other
69 packages on your system depending on it. This can confuse configure into
70 thinking you have a usable PCRE2 when you don't. Either properly install
71 libpcre2-dev so Cyrus can use it, or configure Cyrus with ``--disable-pcre2``
72 so that it ignores the partial installation.
73
74 Please note that on Debian-based systems, PCRE (the old one, no longer
75 maintained) is called "pcre3". Yes, this is confusing.
76
77 How are you planning on upgrading?
78 ##################################
79
80 Ideally, you will do a sandboxed test installation of 3.12 using a snapshot
81 of your existing data before you switch off your existing installation.
82
83 Other possibilities are upgrading by replication, or upgrading in place.
84
85 **The rest of the instructions are written assuming a sandboxed 3.12
86 installation**, but you should read and understand them regardless of how
87 you intend to perform the upgrade.
88
89 Upgrade by replicating
90 ~~~~~~~~~~~~~~~~~~~~~~
91
92 If you're familiar with replication, and your current installation is 2.4 or
93 newer, you can set up your existing installation to replicate data to a new
94 3.12 installation and failover to the new installation when you're ready. The
95 replication protocol has been kept mostly backwards compatible.
96
97 If your old installation contains mailboxes or messages that are older than
98 2.4, they may not have GUID fields in their indexes (index version too old),
99 or they may have their GUID field set to zero. 3.12 will not accept message
100 replications without valid matching GUIDs, so you need to fix this on your
101 old installation first.
102
103 You can check for affected mailboxes by examining the output from the
104 :cyrusman:`mbexamine(8)` tool:
105
106 * mailboxes that report a 'Minor Version:' less than 10 will need to have
107 their index upgraded using :cyrusman:`reconstruct(8)` with the
108 `-V <version>` parameter to be at least 10.
109 * mailboxes containing messages that report 'GUID:0' will need to have
110 their GUIDs recalculated using :cyrusman:`reconstruct(8)` with the `-G`
111 parameter.
112
113 If you have a large amount of data, these reconstructs will take a long time,
114 so it's better to identify the mailboxes needing attention and target them
115 specifically. But if you have a small amount of data, it might be less work
116 to just `reconstruct -G -V max` everything.
117
118 If your old installation is on 3.0 or older and is using mailbox annotations,
119 you will have problems replicating to newer versions due to missing MODSEQ
120 (:issue:`4967`). There is an experimental patch in the comments on this issue
121 that might help for a one-off replication run into an empty replica, but it
122 will not help for updating a replica that already has data.
123
124 Upgrade in place
125 ~~~~~~~~~~~~~~~~
126
127 If you are upgrading in place, you will need to shut down Cyrus entirely while
128 you install the new package. You should probably also block logins or filewall
129 off internet access until you're completely finished so that you aren't
130 surprised by users reconnecting before the upgraded server is ready for them.
131
132 Do What As Who?
133 ###############
134
135 Since the various files, databases, directories, etc. used by Cyrus
136 must be readable and writable as the ``cyrus`` user, please make sure
137 to **always** perform Cyrus commands *as* the ``cyrus`` user, and not
138 as ``root``. In our documentation, we will always reference Cyrus
139 commands in this form -- :cyrusman:`cyr_info(8)` -- before using
140 examples of them, so you'll know that those commands **must** be run as
141 the ``cyrus`` user.
142
143 Doing so in most systems is as simple as using either the ``su`` or
144 ``sudo`` commands, like so::
145
146 su cyrus -c "/usr/local/bin/cyr_info conf-lint -C /etc/imapd.conf -M /etc/cyrus.conf"
147 sudo -u cyrus /usr/local/bin/cyr_info conf-lint -C /etc/imapd.conf -M /etc/cyrus.conf
148
149 In this document, however, there are also several command examples which
150 *should* or **must** be run as ``root``. These are always standard \*nix
151 commands, such as ``rsync`` or ``scp``.
152
153 We strongly recommend that you read this entire document before upgrading.
154
155 2. Install new 3.12 Cyrus
156 -------------------------
157
158 Download the release :ref:`3.12 package tarball <getcyrus>`.
159
160 Fetch the libraries for your platform. The full list (including all optional
161 packages) for Debian is::
162
163 sudo apt-get install -y autoconf automake autotools-dev bash-completion \
164 bison build-essential comerr-dev debhelper flex g++ git gperf groff \
165 heimdal-dev libbsd-resource-perl libclone-perl libconfig-inifiles-perl \
166 libcunit1-dev libdatetime-perl libdigest-sha-perl libencode-imaputf7-perl \
167 libfile-chdir-perl libglib2.0-dev libical-dev libio-socket-inet6-perl \
168 libio-stringy-perl libjansson-dev libldap2-dev libmysqlclient-dev \
169 libnet-server-perl libnews-nntpclient-perl libpam0g-dev libpcre2-dev \
170 libsasl2-dev libsqlite3-dev libssl-dev libtest-unit-perl libtool \
171 libunix-syslog-perl liburi-perl libxapian-dev libxml-generator-perl \
172 libxml-xpath-perl libxml2-dev libwrap0-dev libzephyr-dev lsb-base \
173 net-tools perl php-cli php-curl pkg-config po-debconf tcl-dev transfig \
174 uuid-dev vim wamerican wget xutils-dev zlib1g-dev sasl2-bin rsyslog sudo \
175 acl telnet
176
177 If you're on another platform and can provide the list of dependencies, please
178 let us know via a
179 `GitHub issue <https://github.com/cyrusimap/cyrus-imapd/issues>`_
180 or documentation pull request, or send mail to the
181 :ref:`developer list<feedback-mailing-lists>`.
182
183 Follow the :ref:`general install instructions <installing>`.
184
185 .. note::
186
187 It's best to ensure your new Cyrus *will not* start up automatically
188 if your server restarts in the middle of the upgrade.
189
190 How this is best achieved will depend upon your OS and distro, but may
191 involve something like ``systemctl disable cyrus-imapd`` or
192 ``update-rc.d cyrus-imapd disable``
193
194 3. Shut down existing Cyrus
195 ---------------------------
196
197 Shut down your existing Cyrus installation with its init script or
198 whatever method you normally use.
199
200 This is necessary to guarantee a clean data snapshot.
201
202 4. Backup and Copy existing data
203 --------------------------------
204
205 We recommend backing up all your data before continuing.
206
207 * Sieve scripts
208 * Config files
209 * Mail spool
210 * :ref:`Cyrus Databases <databases>`
211
212 Copy all of this to the new instance, using ``rsync`` or similar tools.
213
214 .. note::
215
216 Cyrus keeps its data and databases in various locations, some of
217 which may be tailored by your configuration. Please consult
218 :ref:`imap-admin-locations` for guidance on where data lives in your
219 current installation.
220
221 For example, to copy from an existing Debian or Ubuntu installation
222 using their standard locations, you might execute this series of
223 commands on the *new* server (where "oldimap" is the name of the old
224 server)::
225
226 rsync -aHv oldimap:/var/lib/cyrus/. /var/lib/cyrus/.
227 rsync -aHv oldimap:/var/spool/cyrus/. /var/spool/cyrus/.
228
229 You don't need to copy the following databases as Cyrus 3.12 will
230 recreate these for you automatically:
231
232 * duplicate delivery (deliver.db),
233 * TLS cache (tls_sessions.db),
234 * PTS cache (ptscache.db),
235 * STATUS cache (statuscache.db).
236
237 .. note::
238 You may wish to consider relocating these four databases to ephemeral
239 storage, such as ``/run/cyrus`` (Debian/Ubuntu) or ``/var/run/cyrus``
240 or whatever suitable tmpfs is provided on your distro. It will place
241 less IO load on your disks and run faster.
242
243 5. Copy config files and update
244 -------------------------------
245
246 Again, check the locations on your specific installation. For example,
247 on FreeBSD systems, the configuration files :cyrusman:`imapd.conf(5)`
248 and :cyrusman:`cyrus.conf(5)` are in ``/usr/local/etc``, rather than
249 ``/etc/``. Run this command on the *old* server::
250
251 scp /etc/cyrus.conf /etc/imapd.conf newimap:/etc/
252
253 Using the :cyrusman:`cyr_info(8)` command, check to see if your
254 imapd.conf file contains any deprecated options. Run this command on
255 the new server::
256
257 cyr_info conf-lint -C <path to imapd.conf> -M <path to cyrus.conf>
258
259 You need to provide both imapd.conf and cyrus.conf so that conf-lint knows
260 the names of all your services and can check service-specific overrides.
261
262 To check your entire system's configuration you can use the conf-all
263 action. This command takes all the system defaults, along with anything
264 you have provided overrides for in your config files::
265
266 cyr_info conf-all -C <path to imapd.conf> -M <path to cyrus.conf>
267
268 6. Upgrade specific items
269 -------------------------
270
271 Nothing special required when upgrading from 3.10.
272
273 7. Start new 3.12 Cyrus and verify
274 ----------------------------------
275
276 ::
277
278 sudo ./master/master -d
279
280 Check ``/var/log/syslog`` for errors so you can quickly understand potential
281 problems.
282
283 When you're satisfied version 3.12 is running and can see all its data
284 correctly, start the new Cyrus up with your regular init script.
285
286 If something has gone wrong, contact us on the
287 :ref:`mailing list <feedback-mailing-lists>`.
288 You can revert to backups and keep processing mail using your old version
289 until you're able to finish your 3.12 installation.
290
291 .. note::
292
293 If you've disabled your system startup scripts, as recommended in
294 step 2, remember to re-enable them. Use something like ``systemctl
295 enable cyrus-imapd`` or ``update-rc.d cyrus-imapd enable``
296
297 8. Reconstruct databases and cache
298 ----------------------------------
299
300 The following steps can each take a long time, so we recommend
301 running them one at a time (to reduce locking contention and high I/O load).
302
303 To upgrade all the mailboxes to the latest version. This will take hours,
304 possibly days.
305
306 ::
307
308 reconstruct -V max
309
310 To check (and correct) quota usage::
311
312 quota -f
313
314 If you've been using CalDAV/CardDAV/all of the DAV from earlier releases, then
315 the user.dav databases need to be reconstructed due to format changes.::
316
317 dav_reconstruct -a
318
319 If have the `reverseacls` feature enabled in :cyrusman:`imapd.conf(5)`, you may
320 need to regenerate the data it uses (which is stored in `mailboxes.db`). This
321 is automatically regenerated at startup by ``ctl_cyrusdb -r`` if the
322 `reverseacls` setting has changed. So, to force a regeneration:
323
324 1. Shut down Cyrus
325 2. Change `reverseacls` to `0` in :cyrusman:`imapd.conf(5)`
326 3. Run :cyrusman:`ctl_cyrusdb(8)` with the `-r` switch (or just start
327 Cyrus, assuming your :cyrusman:`cyrus.conf(5)` contains a
328 `ctl_cyrusdb -r` entry in the START section). The old RACL entries
329 will be removed
330 4. (If you started Cyrus, shut it down again)
331 5. Change `reverseacls` back to `1`
332 6. Start up Cyrus (or run `ctl_cyrusdb -r`). The RACL entries will
333 be rebuilt
334
335 There are fixes and improvements to caching and search indexing in 3.12.
336 You should consider running :cyrusman:`reconstruct(8)` across all mailboxes to
337 rebuild caches, and :cyrusman:`squatter(8)` to rebuild search indexes. This
338 will probably take a long time, so you may wish to only do it per-mailbox as
339 inconsistencies are discovered.
340
341 9. Do you want any new features?
342 --------------------------------
343
344 3.12 comes with many lovely new features. Consider which ones you want to
345 enable. Check the :ref:`3.12 release notes <imap-release-notes-3.12>` for the
346 full list.
347
348 10. Upgrade complete
349 --------------------
350
351 Your upgrade is complete, congratulations!
352
353 Special note for Murder configurations
354 --------------------------------------
355
356 If you upgrade murder frontends before you upgrade all the backends,
357 they may advertise features to clients which the backends don't support,
358 which will cause the commands to fail when they are proxied to the backend.
359
360 Generally accepted wisdom when upgrading a Murder configuration is to
361 upgrade all your back end servers first. This can be done one at a time.
362
363 Upgrade your mupdate master and front ends last.
364
365 Please note that you will be unable to set ANNOTATION-STORAGE or MAILBOX
366 quotas (formerly known as X-ANNOTATION-STORAGE and X-NUM_FOLDERS) in a
367 mixed-version murder environment until your frontends are upgraded to 3.10
368 (or later). Upgraded frontends know how to negotiate with older backends, but
369 older frontends do not know how to negotiate with newer backends.
370
371 If you wish to use XFER to transfer mailboxes from an existing backend to your
372 new 3.12 backend, you should first upgrade your existing backends to 3.10, 3.8,
373 3.6.1, 3.4.5, 3.2.11, or 3.0.18. These releases contain a patch such that XFER
374 will correctly recognise 3.8 and later destinations. Without this patch, XFER
375 will not recognise 3.12, and will downgrade mailboxes to the oldest supported
376 format (losing metadata) in transit.