| 1 |
.. _imap-admin-deleted-expired-expunged-purged: |
| 2 |
|
| 3 |
====================================================== |
| 4 |
When is What ... Deleted, Expired, Expunged or Purged? |
| 5 |
====================================================== |
| 6 |
|
| 7 |
The terminology sometimes plays administrators for fools -- which they |
| 8 |
are obviously not -- but an article clarifying what it is that is meant |
| 9 |
by either term of Deleted, Expired, Expunged or Purged goes a long way. |
| 10 |
|
| 11 |
**Deleted** |
| 12 |
|
| 13 |
A message has been flagged as ``\Deleted``. |
| 14 |
|
| 15 |
In the context of folders, *Deleted* only really applies to a folder |
| 16 |
having been removed (from the user's (IMAP client) view), as opposed |
| 17 |
to having been renamed to a hierarchy in a trash folder. |
| 18 |
|
| 19 |
**Expired** |
| 20 |
|
| 21 |
The index records of a message have been expired, and usually this |
| 22 |
means the message file has been purged as well. However the message |
| 23 |
file could be purged prior to index records being expired. |
| 24 |
|
| 25 |
**Expunged** |
| 26 |
|
| 27 |
The message (which has been flagged as ``\Deleted``) is also |
| 28 |
expunged, meaning that the user can in no way retrieve the message |
| 29 |
autonomously. |
| 30 |
|
| 31 |
**Purged** |
| 32 |
|
| 33 |
The message's index record may still exist (until they are expired), |
| 34 |
but the message file is removed from the filesystem, or in the |
| 35 |
context of folders, the mail folder is removed from the filesystem. |
| 36 |
|
| 37 |
Users and IMAP Clients Deleting Messages |
| 38 |
======================================== |
| 39 |
|
| 40 |
When a **message** is deleted by a user, this means that the user's IMAP |
| 41 |
client has in fact *flagged* the message with ``\Deleted``, or |
| 42 |
alternatively, the IMAP client has moved the message to a trash folder |
| 43 |
(and has at least flagged the original copy as ``\Deleted``). |
| 44 |
|
| 45 |
What is the exact behavior depends on the IMAP client software, and if |
| 46 |
so allowed, the user's preferences specified within the IMAP client |
| 47 |
software, and the Cyrus IMAP server configuration. |
| 48 |
|
| 49 |
Flagged As ``\Deleted`` |
| 50 |
----------------------- |
| 51 |
|
| 52 |
When a message is merely flagged with ``\Deleted``, the message itself |
| 53 |
as such remains available to the IMAP client, but the IMAP client used |
| 54 |
may not make it possible for the user to view a list of messages flagged |
| 55 |
with ``\Deleted``. As such, the user may interpret the message as |
| 56 |
removed and unavailable -- if the removal was accidental, a support call |
| 57 |
may be on its way. |
| 58 |
|
| 59 |
The message in fact still resides in the Cyrus IMAP mail spool, still |
| 60 |
resides in the same IMAP folder, and still resides on the filesystem. |
| 61 |
|
| 62 |
Only when the user (or the user's IMAP client as is often the case) |
| 63 |
issues an ``EXPUNGE`` against the folder, or a ``UID EXPUNGE`` against |
| 64 |
the message [#]_, will the message be actually removed -- at least from |
| 65 |
the user's perspective. It then becomes irretrievable even if the IMAP |
| 66 |
client allows the listing of messages flagged with ``\Deleted``. |
| 67 |
|
| 68 |
Issuing an ``EXPUNGE`` may come in the form of a button to "compact" the |
| 69 |
folder, or an IMAP client routine that is executed periodically or at |
| 70 |
the end of a session (e.g. as the client application is closed), such as |
| 71 |
an "Empty Trash folder" kind of option. |
| 72 |
|
| 73 |
It is here that the Cyrus IMAP server settings come in to play, most |
| 74 |
prominently the ``expunge_mode`` setting, which has three possible |
| 75 |
values: |
| 76 |
|
| 77 |
**delayed** (the default *since version 2.5.0*) |
| 78 |
|
| 79 |
The message files as well as the index records are retained for an |
| 80 |
undetermined period of time -- possibly indefinitely. |
| 81 |
|
| 82 |
A separate job (using :ref:`imap-reference-manpages-systemcommands-cyr_expire`) is |
| 83 |
responsible for actually removing index records and message files. |
| 84 |
|
| 85 |
**default** |
| 86 |
|
| 87 |
The message files are removed at the first opportunity, while the |
| 88 |
index records remain available to facilitate ``QRESYNC``. |
| 89 |
|
| 90 |
In this context also, when we say "message files are removed", we |
| 91 |
mean "purged from the filesystem". |
| 92 |
|
| 93 |
**immediate** |
| 94 |
|
| 95 |
The message files as well as the index records are removed at the |
| 96 |
earliest opportunity. |
| 97 |
|
| 98 |
In this context, when we say "message files are removed", we mean |
| 99 |
"purged from the filesystem". |
| 100 |
|
| 101 |
Exceptional circumstances aside, when ``immediate`` or ``default`` is |
| 102 |
the configured ``expunge_mode``, message files are often purged from the |
| 103 |
filesystem too quickly for anyone to recover. |
| 104 |
|
| 105 |
.. NOTE:: |
| 106 |
|
| 107 |
One such exceptional circumstance is a mailbox with multiple |
| 108 |
sessions keeping the mailbox open. Cyrus IMAP ensures no mailbox |
| 109 |
records disappear from underneath an existing open mailbox session. |
| 110 |
|
| 111 |
Moved to Trash Folder |
| 112 |
--------------------- |
| 113 |
|
| 114 |
Should the IMAP client normally, or allow the user to specify through |
| 115 |
preferences, that messages being deleted should be moved in to a trash |
| 116 |
folder, then the user will usually be able to recover from accidental |
| 117 |
deletion autonomously, for as long as a copy of the deleted message(s) |
| 118 |
resides in such trash folder. |
| 119 |
|
| 120 |
However, the trash folder would typically continue to grow and grow, and |
| 121 |
usually counts towards the user's resource usage (a.k.a. |
| 122 |
:ref:`imap-features-quota`); many IMAP clients therefore allow the user |
| 123 |
to specify a preference to empty the trash folder at the end of a |
| 124 |
session, or otherwise periodically. |
| 125 |
|
| 126 |
If the IMAP client does not support :rfc:`6851` (for ``UID MOVE``), the |
| 127 |
client may choose to ``COPY`` the message then flag the original with |
| 128 |
``\Deleted``, then ``EXPUNGE`` the folder or ``UID EXPUNGE`` |
| 129 |
(:rfc:`4315`) the message. |
| 130 |
|
| 131 |
This does not fare well in situations where the user is over quota, |
| 132 |
though, and (other) messages will need to be flagged as ``\Deleted`` and |
| 133 |
expunged, and/or folders within the quota root hierarchy will need to be |
| 134 |
deleted. |
| 135 |
|
| 136 |
Expunged Messages |
| 137 |
----------------- |
| 138 |
|
| 139 |
Messages in expunged folders, or messages that have been expunged |
| 140 |
individually, can not autonomously be restored by a user, and are gone |
| 141 |
permanently unless ``expunge_mode: delayed`` is used. |
| 142 |
|
| 143 |
Recovering expunged messages requires administrator assistance, who can |
| 144 |
use command-line tools such as :ref:`imap-reference-manpages-systemcommands-unexpunge` to |
| 145 |
list and restore messages expunged. See the documentation on |
| 146 |
:ref:`imap-reference-manpages-systemcommands-unexpunge` for a walk-through on how that |
| 147 |
works. |
| 148 |
|
| 149 |
With the use of ``expunge_mode: delayed``, a regular ``EVENT`` (see |
| 150 |
:cyrusman:`cyrus.conf(5)`) is responsible for triggering |
| 151 |
:ref:`imap-reference-manpages-systemcommands-cyr_expire`. This utility takes a parameter |
| 152 |
``-X <days>`` to delete from the filesystem any messages that had been |
| 153 |
expunged (by the user or the IMAP client) more than ``<days>`` days ago. |
| 154 |
|
| 155 |
In other words, using ``expunge_mode: delayed`` and |
| 156 |
:ref:`imap-reference-manpages-systemcommands-cyr_expire` allows an administrator to recover |
| 157 |
messages that have been deleted by the user less than ``<days>`` ago. |
| 158 |
|
| 159 |
.. NOTE:: |
| 160 |
|
| 161 |
This also offers a backup program the chance to obtain all message |
| 162 |
files. For a monthly full cycle, for example, one could choose to |
| 163 |
purge message files from the filesystem only after 69 days: two |
| 164 |
months plus the maximum margin for a first Saturday to Sunday night |
| 165 |
of the week. |
| 166 |
|
| 167 |
Deleting Folders |
| 168 |
================ |
| 169 |
|
| 170 |
When folders are deleted the IMAP client tends to either delete the |
| 171 |
folder, or rename the folder to a hierarchy in a trash folder. |
| 172 |
|
| 173 |
.. NOTE:: |
| 174 |
|
| 175 |
Note that deleting a folder ``A/B`` in a hierarchy ``A/B/C`` also |
| 176 |
deletes the folder ``A/B/C``. |
| 177 |
|
| 178 |
If the folder is not renamed to a hierarchy in a trash folder but |
| 179 |
instead removed directly, then the user has no way to autonomously |
| 180 |
recover from such event. |
| 181 |
|
| 182 |
This is where the Cyrus IMAP server settings come in to play, most |
| 183 |
prominently ``delete_mode``. |
| 184 |
|
| 185 |
The setting holds two values: |
| 186 |
|
| 187 |
**delayed** (the default *since version 2.5.0*) |
| 188 |
|
| 189 |
Mailboxes that are being deleted are not deleted from the |
| 190 |
filesystem, but instead renamed to a special mailbox hierarchy under |
| 191 |
the deleted prefix, to be removed later by |
| 192 |
:ref:`imap-reference-manpages-systemcommands-cyr_expire`. |
| 193 |
|
| 194 |
**immediate** |
| 195 |
|
| 196 |
In immediate mode, the mailbox is removed from the filesystem |
| 197 |
immediately. Note that for large folders, this can be a |
| 198 |
comparatively expensive operation. |
| 199 |
|
| 200 |
Where are the Messages? |
| 201 |
======================= |
| 202 |
|
| 203 |
This part of the documentation assumes that you have run with the |
| 204 |
default settings of ``delete_mode: delayed`` and |
| 205 |
``expunge_mode: delayed``. |
| 206 |
|
| 207 |
The result of a message having deleted in either of the former ways, or |
| 208 |
an entire folder having been deleted, is one of the following stages; |
| 209 |
|
| 210 |
* The message has only been flagged as ``\Deleted`` and the message |
| 211 |
nor the folder has been expunged. |
| 212 |
|
| 213 |
Result: The message resides in the original folder. |
| 214 |
|
| 215 |
* The message has only been flagged as ``\Deleted`` and either the |
| 216 |
message individually or the entire folder as a whole has been |
| 217 |
expunged. |
| 218 |
|
| 219 |
Result: The message resides in the original folder and can be |
| 220 |
retrieved using :ref:`imap-reference-manpages-systemcommands-unexpunge`. |
| 221 |
|
| 222 |
* The message has been copied to the trash folder and at least flagged |
| 223 |
``\Deleted`` in the source folder, and the original message or the |
| 224 |
entire folder in which the original message resided may or may not |
| 225 |
have been expunged. |
| 226 |
|
| 227 |
Similarly, the trash folder may or may not have been "emptied". |
| 228 |
|
| 229 |
Result: A copy of the message still exists in the original folder |
| 230 |
and can be retrieved using :ref:`imap-reference-manpages-systemcommands-unexpunge`. |
| 231 |
|
| 232 |
* The message was moved in to the trash folder, implying the original |
| 233 |
message is expunged from the source folder -- through ``UID MOVE`` |
| 234 |
or :rfc:`6851` support *since version 2.5.0*. |
| 235 |
|
| 236 |
The trash folder may or may not have been "emptied". |
| 237 |
|
| 238 |
Result: A copy of the message still exists in the original folder |
| 239 |
and can be retrieved using :ref:`imap-reference-manpages-systemcommands-unexpunge`. |
| 240 |
|
| 241 |
* The folder was moved to a hierarchy in the trash folder, and the |
| 242 |
trash folder has not yet been "emptied". |
| 243 |
|
| 244 |
Result: A copy of the message exists in the trash folder's |
| 245 |
hierarchy. |
| 246 |
|
| 247 |
* The folder was moved to a hierarchy in the trash folder, and the |
| 248 |
trash folder as subsequently been emptied. |
| 249 |
|
| 250 |
Result: The folder hierarchy has been renamed to the deleted |
| 251 |
namespace. |
| 252 |
|
| 253 |
.. rubric:: Footnotes |
| 254 |
|
| 255 |
.. [#] |
| 256 |
|
| 257 |
Only if the IMAP client supports :rfc:`4315`, the IMAP UIDPLUS |
| 258 |
Extension. |