| 1 |
.. _imap-developer-api-mailbox: |
| 2 |
|
| 3 |
.. Note: This document was converted from the original by Nic Bernstein |
| 4 |
(Onlight). Any formatting mistakes are my fault and not the |
| 5 |
original author's. Converted via the pandoc tool from HTML. |
| 6 |
|
| 7 |
Mailbox API |
| 8 |
=========== |
| 9 |
|
| 10 |
Intro |
| 11 |
----- |
| 12 |
|
| 13 |
The Mailbox API is implemented in ``imap/mailbox.h`` and |
| 14 |
``imap/mailbox.c``. It wraps the data structures of the |
| 15 |
``cyrus.header``, ``cyrus.index`` and ``cyrus.cache`` files in a |
| 16 |
psuedo-object-oriented way, allowing easy changes to the mailbox while |
| 17 |
keeping the internal cached data structures consistent. |
| 18 |
|
| 19 |
Opening and closing |
| 20 |
------------------- |
| 21 |
|
| 22 |
:: |
| 23 |
|
| 24 |
struct mailbox *mailbox = NULL; |
| 25 |
int r; |
| 26 |
const char *mboxname = "user.brong"; |
| 27 |
|
| 28 |
r = mailbox_open_iwl(mboxname, &mailbox); |
| 29 |
// or |
| 30 |
r = mailbox_open_irl(mboxname, &mailbox); |
| 31 |
// or |
| 32 |
r = mailbox_open_exclusive(mboxname, &mailbox); |
| 33 |
if (r) return r; |
| 34 |
|
| 35 |
do_stuff(mailbox); |
| 36 |
|
| 37 |
mailbox_close(&mailbox); |
| 38 |
|
| 39 |
It is always necessary to obtain an index lock when opening a mailbox, |
| 40 |
because the index header read must be consistent. The locks are as |
| 41 |
follows: |
| 42 |
|
| 43 |
+----------------------------+-------------+--------------+ |
| 44 |
| Function | Namelock | Index Lock | |
| 45 |
+============================+=============+==============+ |
| 46 |
| mailbox\_open\_iwl | Shared | Exclusive | |
| 47 |
+----------------------------+-------------+--------------+ |
| 48 |
| mailbox\_open\_irl | Shared | Shared | |
| 49 |
+----------------------------+-------------+--------------+ |
| 50 |
| mailbox\_open\_exclusive | Exclusive | Exclusive | |
| 51 |
+----------------------------+-------------+--------------+ |
| 52 |
|
| 53 |
It should never be necessary to call ``mailbox_open_exclusive``, but |
| 54 |
it's included for completeness. Use ``mailbox_open_iwl`` if you expect |
| 55 |
to need to write to the index (or even if you're not sure) and |
| 56 |
``mailbox_open_irl`` when you know you're only reading from the file and |
| 57 |
wish to allow other readers to work concurrently. |
| 58 |
|
| 59 |
Many actions are delayed until the mailbox is closed, or even until the |
| 60 |
*last* mailbox is closed for things that require an exclusive namelock |
| 61 |
to perform like deletion or repack. See below under "delayed actions" |
| 62 |
for more detail. |
| 63 |
|
| 64 |
To avoid opening the same file multiple times, the mailbox API refcounts |
| 65 |
open mailboxes. If you open the same mailbox again (i.e. a URL fetch or |
| 66 |
status command on the currently select mailbox) then the same mailbox |
| 67 |
will be returned. It must be unlocked (see below or the open command |
| 68 |
will return IMAP\_MAILBOX\_LOCKED). The matching close will reduce the |
| 69 |
refcount, and only the final close will do the cleanup actions. |
| 70 |
|
| 71 |
Locking and unlocking |
| 72 |
--------------------- |
| 73 |
|
| 74 |
You can keep a mailbox "open", maintaining the namelock, while releasing |
| 75 |
the index lock to allow other processes to make changes to the mailbox. |
| 76 |
By holding the namelock, you know that record numbers won't change, and |
| 77 |
the underlying message files won't be deleted. |
| 78 |
|
| 79 |
``mailbox_close`` will call ``mailbox_unlock_index`` if the index is |
| 80 |
still locked, so it is not necessary to explicitly unlock the index |
| 81 |
before closing. |
| 82 |
|
| 83 |
:: |
| 84 |
|
| 85 |
r = mailbox_unlock_index(mailbox, NULL); |
| 86 |
|
| 87 |
// sleep on user input... |
| 88 |
|
| 89 |
r = mailbox_lock_index(mailbox, LOCK_SHARED); |
| 90 |
// or |
| 91 |
r = mailbox_lock_index(mailbox, LOCK_EXCLUSIVE); |
| 92 |
|
| 93 |
For example, ``mailbox_unlock_index`` and ``mailbox_lock_index`` are |
| 94 |
used extensively by the index module, allowing an imap client to |
| 95 |
maintain a long lived connection selected to a mailbox and know that |
| 96 |
messages won't magically disappear from under it - yet at the same time |
| 97 |
allow new mail delivery to happen or other imap connections to query the |
| 98 |
mailbox. |
| 99 |
|
| 100 |
If you have built an accurate statuscache item for the locked mailbox, |
| 101 |
you can pass this as the second parameter to mailbox\_index\_unlock. If |
| 102 |
there have been any changes, mailbox\_index\_unlock will invalidated the |
| 103 |
statuscache. If you give it the new value, then it will store that value |
| 104 |
instead. For example: |
| 105 |
|
| 106 |
:: |
| 107 |
|
| 108 |
struct statusdata sdata; |
| 109 |
index_status(state, &sdata); |
| 110 |
/* RECENT is zero for everyone else because we wrote a new |
| 111 |
* recentuid! */ |
| 112 |
sdata.recent = 0; |
| 113 |
mailbox_unlock_index(state->mailbox, &sdata); |
| 114 |
|
| 115 |
See "delayed actions" below for delayed actions performed during an |
| 116 |
unlock. |
| 117 |
|
| 118 |
Creating, renaming and deleting |
| 119 |
------------------------------- |
| 120 |
|
| 121 |
**WARNING:** These functions only change the mailbox files on disk. They |
| 122 |
don't update the mailboxes.db records or contact murder servers. In most |
| 123 |
cases you are probably looking for the ``mboxlist_`` functions instead. |
| 124 |
|
| 125 |
Creating a mailbox is somewhat longwinded - as there are many optional |
| 126 |
parameters. |
| 127 |
|
| 128 |
:: |
| 129 |
|
| 130 |
int mailbox_create(const char *name, const char *part, const char *acl, |
| 131 |
const char *uniqueid, int options, unsigned uidvalidity, |
| 132 |
struct mailbox **mailboxptr); |
| 133 |
|
| 134 |
Most interesting to note is that on success, ``mailboxptr`` will contain |
| 135 |
the same mailbox that ``mailbox_open_exclusive`` above would have |
| 136 |
returned, with an exclusive namelock and an exclusive index lock. This |
| 137 |
allows you to perform other consistency operations after creating the |
| 138 |
mailbox with a full guarantee that no other process will even be able to |
| 139 |
know of the mailbox's existence! You can still roll-back by deleting the |
| 140 |
mailbox and the next process will get the namelock and see no mailbox |
| 141 |
with that name. |
| 142 |
|
| 143 |
:: |
| 144 |
|
| 145 |
int mailbox_rename_copy(struct mailbox *oldmailbox, |
| 146 |
const char *newname, const char *newpart, |
| 147 |
const char *userid, int ignorequota, |
| 148 |
int silent, |
| 149 |
struct mailbox **newmailboxptr); |
| 150 |
|
| 151 |
Very similar to mailbox\_create - the new mailbox is created with an |
| 152 |
exclusive name lock and returned. The old mailbox must be passed in with |
| 153 |
an **exclusive index lock** but is fine with a shared namelock, as it |
| 154 |
will be passed to ``mailbox_delete``. |
| 155 |
|
| 156 |
:: |
| 157 |
|
| 158 |
int mailbox_delete(struct mailbox **mailboxptr); |
| 159 |
|
| 160 |
Just like ``mailbox_close`` above, this closes the mailbox. Before it |
| 161 |
does so, it sets the OPT\_MAILBOX\_DELETED option flag in the index |
| 162 |
header. The interesting work is actually done in ``mailbox_close``. See |
| 163 |
below under "delayed actions". |
| 164 |
|
| 165 |
``mailbox_delete`` requires an exclusive index lock, but can complete |
| 166 |
quite happily with only a shared namelock. |
| 167 |
|
| 168 |
Reading and writing records |
| 169 |
--------------------------- |
| 170 |
|
| 171 |
Ok - so you have a mailbox, it's opened, and the index is locked. Time |
| 172 |
to start reading and writing some records! |
| 173 |
|
| 174 |
At the mailbox level there is no concept of "message numbers" from imap, |
| 175 |
only "record numbers". The canonical variable name to refer to record |
| 176 |
numbers is ``recno``. All records are read and written using |
| 177 |
``struct index_record`` values. |
| 178 |
|
| 179 |
Here at the API definitions used for reading and writing: |
| 180 |
|
| 181 |
:: |
| 182 |
|
| 183 |
int mailbox_read_index_record(struct mailbox *mailbox, |
| 184 |
uint32_t recno, |
| 185 |
struct index_record *record); |
| 186 |
int mailbox_rewrite_index_record(struct mailbox *mailbox, |
| 187 |
struct index_record *record); |
| 188 |
int mailbox_append_index_record(struct mailbox *mailbox, |
| 189 |
struct index_record *record); |
| 190 |
int mailbox_commit(mailbox); |
| 191 |
|
| 192 |
An example of iterating through a mailbox |
| 193 |
|
| 194 |
:: |
| 195 |
|
| 196 |
uint32_t recno; |
| 197 |
struct index_record record; |
| 198 |
int make_changes; |
| 199 |
|
| 200 |
/* DEPRECATED */ |
| 201 |
for (recno = 1; recno <= mailbox->i.num_records; recno++) { |
| 202 |
if (mailbox_read_index_record(mailbox, recno, &record)) |
| 203 |
fatal("invalid record", EX_SOFTWARE); // or return an error |
| 204 |
if (record.internal_flags & FLAG_INTERNAL_EXPUNGED) |
| 205 |
continue; // skip expunged records |
| 206 |
make_changes = do_stuff(mailbox, &record); |
| 207 |
if (make_changes) |
| 208 |
mailbox_rewrite_index_record(mailbox, &record); |
| 209 |
} |
| 210 |
|
| 211 |
/* the new way */ |
| 212 |
int make_change; |
| 213 |
const struct index_record *record; |
| 214 |
struct mailbox_iter *iter; |
| 215 |
|
| 216 |
iter = mailbox_iter_init(mailbox, 0, ITER_SKIP_EXPUNGED); |
| 217 |
while ((record = mailbox_iter_step(iter))) { |
| 218 |
make_changes = do_stuff(mailbox, record); |
| 219 |
if (make_changes) |
| 220 |
mailbox_rewrite_index_record(mailbox, record); |
| 221 |
} |
| 222 |
mailbox_iter_done(&iter); |
| 223 |
|
| 224 |
NOTE: ``mailbox_rewrite_index_record`` doesn't need a recno, as that's |
| 225 |
cached inside the index\_record struct. |
| 226 |
|
| 227 |
NOTE: You need an exclusively locked index to use rewrite or append, but |
| 228 |
only a shared index lock to use read. |
| 229 |
|
| 230 |
There are a range of consistency checks done to ensure that a rewrite |
| 231 |
doesn't violate IMAP semantics (an expunged message can never be |
| 232 |
unexpunged, UIDs can't change, etc) and the internal tracking counts and |
| 233 |
quota data are updated as well. They will be committed at unlock time, |
| 234 |
see "delayed actions" |
| 235 |
|
| 236 |
If you don't set the ``record.silent`` field to a true value before |
| 237 |
rewriting or appending, the ``record.modseq`` and |
| 238 |
``record.last_updated`` values will be changed. This allows condstore to |
| 239 |
work correctly. |
| 240 |
|
| 241 |
Appending |
| 242 |
~~~~~~~~~ |
| 243 |
|
| 244 |
To append a record, the file must have already been copied into place |
| 245 |
(XXX - plan to move to a stage based system where the mailbox API |
| 246 |
handles the staging, but that's not finished yet) and been parsed into |
| 247 |
the record struct. The UID must be set already, and must be greater than |
| 248 |
the UID of any existing record in the mailbox. There are a range of |
| 249 |
consistency checks done. |
| 250 |
|
| 251 |
The internal consistency counts are updated by append as well. |
| 252 |
|
| 253 |
Committing |
| 254 |
~~~~~~~~~~ |
| 255 |
|
| 256 |
When you have finished making any changes, you need to "commit". This |
| 257 |
will write the updated values for any index header fields, rewite the |
| 258 |
``cyrus.header`` file if needed and fsync all changes to disk. |
| 259 |
|
| 260 |
It is a fatal error to unlock (or close) a mailbox that has had changes |
| 261 |
without committing, as it can leave the mailbox in a corrupted state. |
| 262 |
|
| 263 |
Cache records |
| 264 |
~~~~~~~~~~~~~ |
| 265 |
|
| 266 |
Cache records are accessed through ``record.crec`` which is not filled |
| 267 |
by read\_index\_record. The cache file is only read and mapped into |
| 268 |
memory as needed, so you if you want to access cache records, the basic |
| 269 |
API is as follows: |
| 270 |
|
| 271 |
:: |
| 272 |
|
| 273 |
int mailbox_cacherecord(struct mailbox *mailbox, |
| 274 |
struct index_record *record); |
| 275 |
const char *cacheitem_base(struct index_record *record, int field); |
| 276 |
unsigned cacheitem_size(struct index_record *record, int field); |
| 277 |
struct buf *cacheitem_buf(struct index_record *record, int field); |
| 278 |
|
| 279 |
You must always call ``mailbox_cacherecord`` on a record before trying |
| 280 |
to access any of the cache items. "``field``" above is the individual |
| 281 |
field (there are 10) in the cache record. There's more information on |
| 282 |
those fields in the mailbox internal format documentation. |
| 283 |
|
| 284 |
:: |
| 285 |
|
| 286 |
for (recno = 1; recno <= mailbox->i.num_records; recno++) { |
| 287 |
if (mailbox_read_index_record(mailbox, recno, &record)) |
| 288 |
fatal("invalid record", EX_SOFTWARE); // or return an error |
| 289 |
if (record.internal_flags & FLAG_INTERNAL_EXPUNGED) |
| 290 |
continue; // skip expunged records |
| 291 |
if (mailbox_cacherecord(mailbox, &record)) |
| 292 |
fatal("failed to read cache", EX_SOFTWARE); |
| 293 |
... |
| 294 |
envelope_length = cacheitem_size(&record, CACHE_ENVELOPE); |
| 295 |
} |
| 296 |
|
| 297 |
See ``imap/mailbox.h`` for the full list of constants. |
| 298 |
|
| 299 |
Delayed Actions |
| 300 |
--------------- |
| 301 |
|
| 302 |
Here's the bit you've been waiting for! What happens during unlock and |
| 303 |
close |
| 304 |
|
| 305 |
first, unlock |
| 306 |
~~~~~~~~~~~~~ |
| 307 |
|
| 308 |
Anything that makes any changes sets the mailbox->has\_changed flag. If |
| 309 |
this is set, then before the index gets unlocked: |
| 310 |
|
| 311 |
- the updatenotifier (idle) is called |
| 312 |
- ``sync_log_mailbox`` (replication) gets called |
| 313 |
- the statuscache value gets erased (or replaced if you passed in an |
| 314 |
updated value). |
| 315 |
|
| 316 |
then: close |
| 317 |
~~~~~~~~~~~ |
| 318 |
|
| 319 |
next the index is unlocked (see above) |
| 320 |
|
| 321 |
third, any "unlink" commands scheduled for email files are run. These |
| 322 |
can't be done until after the mailbox\_commit to ensure consistency - |
| 323 |
the file isn't deleted until the record is written as unlinked! But we |
| 324 |
save the unlink until now so that other tasks aren't waiting for the |
| 325 |
index lock while the unlinks run. Unlink is expensive in IO and time. |
| 326 |
|
| 327 |
finally we check for MAILBOX\_NEEDS\_REPACK or MAILBOX\_DELETED option |
| 328 |
flags. If either is sets, then we make a non-blocking attempt to get an |
| 329 |
exclusive namelock. If the non-blocking attempt fails, then another |
| 330 |
process has the mailbox open, so save the cleanup for them! If it |
| 331 |
succeeds, then go ahead with either ``mailbox_delete_cleanup`` or |
| 332 |
``mailbox_index_repack`` as appropriate. |
| 333 |
|
| 334 |
After this it's just a matter of releasing malloc'd memory and finally |
| 335 |
releasing the name lock. |