Debian Codegraph @

doc/html/_sources/imap/developer/API/mailbox-api.rst.txt

raw
1 .. _imap-developer-api-mailbox:
2
3 .. Note: This document was converted from the original by Nic Bernstein
4 (Onlight). Any formatting mistakes are my fault and not the
5 original author's. Converted via the pandoc tool from HTML.
6
7 Mailbox API
8 ===========
9
10 Intro
11 -----
12
13 The Mailbox API is implemented in ``imap/mailbox.h`` and
14 ``imap/mailbox.c``. It wraps the data structures of the
15 ``cyrus.header``, ``cyrus.index`` and ``cyrus.cache`` files in a
16 psuedo-object-oriented way, allowing easy changes to the mailbox while
17 keeping the internal cached data structures consistent.
18
19 Opening and closing
20 -------------------
21
22 ::
23
24 struct mailbox *mailbox = NULL;
25 int r;
26 const char *mboxname = "user.brong";
27
28 r = mailbox_open_iwl(mboxname, &mailbox);
29 // or
30 r = mailbox_open_irl(mboxname, &mailbox);
31 // or
32 r = mailbox_open_exclusive(mboxname, &mailbox);
33 if (r) return r;
34
35 do_stuff(mailbox);
36
37 mailbox_close(&mailbox);
38
39 It is always necessary to obtain an index lock when opening a mailbox,
40 because the index header read must be consistent. The locks are as
41 follows:
42
43 +----------------------------+-------------+--------------+
44 | Function | Namelock | Index Lock |
45 +============================+=============+==============+
46 | mailbox\_open\_iwl | Shared | Exclusive |
47 +----------------------------+-------------+--------------+
48 | mailbox\_open\_irl | Shared | Shared |
49 +----------------------------+-------------+--------------+
50 | mailbox\_open\_exclusive | Exclusive | Exclusive |
51 +----------------------------+-------------+--------------+
52
53 It should never be necessary to call ``mailbox_open_exclusive``, but
54 it's included for completeness. Use ``mailbox_open_iwl`` if you expect
55 to need to write to the index (or even if you're not sure) and
56 ``mailbox_open_irl`` when you know you're only reading from the file and
57 wish to allow other readers to work concurrently.
58
59 Many actions are delayed until the mailbox is closed, or even until the
60 *last* mailbox is closed for things that require an exclusive namelock
61 to perform like deletion or repack. See below under "delayed actions"
62 for more detail.
63
64 To avoid opening the same file multiple times, the mailbox API refcounts
65 open mailboxes. If you open the same mailbox again (i.e. a URL fetch or
66 status command on the currently select mailbox) then the same mailbox
67 will be returned. It must be unlocked (see below or the open command
68 will return IMAP\_MAILBOX\_LOCKED). The matching close will reduce the
69 refcount, and only the final close will do the cleanup actions.
70
71 Locking and unlocking
72 ---------------------
73
74 You can keep a mailbox "open", maintaining the namelock, while releasing
75 the index lock to allow other processes to make changes to the mailbox.
76 By holding the namelock, you know that record numbers won't change, and
77 the underlying message files won't be deleted.
78
79 ``mailbox_close`` will call ``mailbox_unlock_index`` if the index is
80 still locked, so it is not necessary to explicitly unlock the index
81 before closing.
82
83 ::
84
85 r = mailbox_unlock_index(mailbox, NULL);
86
87 // sleep on user input...
88
89 r = mailbox_lock_index(mailbox, LOCK_SHARED);
90 // or
91 r = mailbox_lock_index(mailbox, LOCK_EXCLUSIVE);
92
93 For example, ``mailbox_unlock_index`` and ``mailbox_lock_index`` are
94 used extensively by the index module, allowing an imap client to
95 maintain a long lived connection selected to a mailbox and know that
96 messages won't magically disappear from under it - yet at the same time
97 allow new mail delivery to happen or other imap connections to query the
98 mailbox.
99
100 If you have built an accurate statuscache item for the locked mailbox,
101 you can pass this as the second parameter to mailbox\_index\_unlock. If
102 there have been any changes, mailbox\_index\_unlock will invalidated the
103 statuscache. If you give it the new value, then it will store that value
104 instead. For example:
105
106 ::
107
108 struct statusdata sdata;
109 index_status(state, &sdata);
110 /* RECENT is zero for everyone else because we wrote a new
111 * recentuid! */
112 sdata.recent = 0;
113 mailbox_unlock_index(state->mailbox, &sdata);
114
115 See "delayed actions" below for delayed actions performed during an
116 unlock.
117
118 Creating, renaming and deleting
119 -------------------------------
120
121 **WARNING:** These functions only change the mailbox files on disk. They
122 don't update the mailboxes.db records or contact murder servers. In most
123 cases you are probably looking for the ``mboxlist_`` functions instead.
124
125 Creating a mailbox is somewhat longwinded - as there are many optional
126 parameters.
127
128 ::
129
130 int mailbox_create(const char *name, const char *part, const char *acl,
131 const char *uniqueid, int options, unsigned uidvalidity,
132 struct mailbox **mailboxptr);
133
134 Most interesting to note is that on success, ``mailboxptr`` will contain
135 the same mailbox that ``mailbox_open_exclusive`` above would have
136 returned, with an exclusive namelock and an exclusive index lock. This
137 allows you to perform other consistency operations after creating the
138 mailbox with a full guarantee that no other process will even be able to
139 know of the mailbox's existence! You can still roll-back by deleting the
140 mailbox and the next process will get the namelock and see no mailbox
141 with that name.
142
143 ::
144
145 int mailbox_rename_copy(struct mailbox *oldmailbox,
146 const char *newname, const char *newpart,
147 const char *userid, int ignorequota,
148 int silent,
149 struct mailbox **newmailboxptr);
150
151 Very similar to mailbox\_create - the new mailbox is created with an
152 exclusive name lock and returned. The old mailbox must be passed in with
153 an **exclusive index lock** but is fine with a shared namelock, as it
154 will be passed to ``mailbox_delete``.
155
156 ::
157
158 int mailbox_delete(struct mailbox **mailboxptr);
159
160 Just like ``mailbox_close`` above, this closes the mailbox. Before it
161 does so, it sets the OPT\_MAILBOX\_DELETED option flag in the index
162 header. The interesting work is actually done in ``mailbox_close``. See
163 below under "delayed actions".
164
165 ``mailbox_delete`` requires an exclusive index lock, but can complete
166 quite happily with only a shared namelock.
167
168 Reading and writing records
169 ---------------------------
170
171 Ok - so you have a mailbox, it's opened, and the index is locked. Time
172 to start reading and writing some records!
173
174 At the mailbox level there is no concept of "message numbers" from imap,
175 only "record numbers". The canonical variable name to refer to record
176 numbers is ``recno``. All records are read and written using
177 ``struct index_record`` values.
178
179 Here at the API definitions used for reading and writing:
180
181 ::
182
183 int mailbox_read_index_record(struct mailbox *mailbox,
184 uint32_t recno,
185 struct index_record *record);
186 int mailbox_rewrite_index_record(struct mailbox *mailbox,
187 struct index_record *record);
188 int mailbox_append_index_record(struct mailbox *mailbox,
189 struct index_record *record);
190 int mailbox_commit(mailbox);
191
192 An example of iterating through a mailbox
193
194 ::
195
196 uint32_t recno;
197 struct index_record record;
198 int make_changes;
199
200 /* DEPRECATED */
201 for (recno = 1; recno <= mailbox->i.num_records; recno++) {
202 if (mailbox_read_index_record(mailbox, recno, &record))
203 fatal("invalid record", EX_SOFTWARE); // or return an error
204 if (record.internal_flags & FLAG_INTERNAL_EXPUNGED)
205 continue; // skip expunged records
206 make_changes = do_stuff(mailbox, &record);
207 if (make_changes)
208 mailbox_rewrite_index_record(mailbox, &record);
209 }
210
211 /* the new way */
212 int make_change;
213 const struct index_record *record;
214 struct mailbox_iter *iter;
215
216 iter = mailbox_iter_init(mailbox, 0, ITER_SKIP_EXPUNGED);
217 while ((record = mailbox_iter_step(iter))) {
218 make_changes = do_stuff(mailbox, record);
219 if (make_changes)
220 mailbox_rewrite_index_record(mailbox, record);
221 }
222 mailbox_iter_done(&iter);
223
224 NOTE: ``mailbox_rewrite_index_record`` doesn't need a recno, as that's
225 cached inside the index\_record struct.
226
227 NOTE: You need an exclusively locked index to use rewrite or append, but
228 only a shared index lock to use read.
229
230 There are a range of consistency checks done to ensure that a rewrite
231 doesn't violate IMAP semantics (an expunged message can never be
232 unexpunged, UIDs can't change, etc) and the internal tracking counts and
233 quota data are updated as well. They will be committed at unlock time,
234 see "delayed actions"
235
236 If you don't set the ``record.silent`` field to a true value before
237 rewriting or appending, the ``record.modseq`` and
238 ``record.last_updated`` values will be changed. This allows condstore to
239 work correctly.
240
241 Appending
242 ~~~~~~~~~
243
244 To append a record, the file must have already been copied into place
245 (XXX - plan to move to a stage based system where the mailbox API
246 handles the staging, but that's not finished yet) and been parsed into
247 the record struct. The UID must be set already, and must be greater than
248 the UID of any existing record in the mailbox. There are a range of
249 consistency checks done.
250
251 The internal consistency counts are updated by append as well.
252
253 Committing
254 ~~~~~~~~~~
255
256 When you have finished making any changes, you need to "commit". This
257 will write the updated values for any index header fields, rewite the
258 ``cyrus.header`` file if needed and fsync all changes to disk.
259
260 It is a fatal error to unlock (or close) a mailbox that has had changes
261 without committing, as it can leave the mailbox in a corrupted state.
262
263 Cache records
264 ~~~~~~~~~~~~~
265
266 Cache records are accessed through ``record.crec`` which is not filled
267 by read\_index\_record. The cache file is only read and mapped into
268 memory as needed, so you if you want to access cache records, the basic
269 API is as follows:
270
271 ::
272
273 int mailbox_cacherecord(struct mailbox *mailbox,
274 struct index_record *record);
275 const char *cacheitem_base(struct index_record *record, int field);
276 unsigned cacheitem_size(struct index_record *record, int field);
277 struct buf *cacheitem_buf(struct index_record *record, int field);
278
279 You must always call ``mailbox_cacherecord`` on a record before trying
280 to access any of the cache items. "``field``" above is the individual
281 field (there are 10) in the cache record. There's more information on
282 those fields in the mailbox internal format documentation.
283
284 ::
285
286 for (recno = 1; recno <= mailbox->i.num_records; recno++) {
287 if (mailbox_read_index_record(mailbox, recno, &record))
288 fatal("invalid record", EX_SOFTWARE); // or return an error
289 if (record.internal_flags & FLAG_INTERNAL_EXPUNGED)
290 continue; // skip expunged records
291 if (mailbox_cacherecord(mailbox, &record))
292 fatal("failed to read cache", EX_SOFTWARE);
293 ...
294 envelope_length = cacheitem_size(&record, CACHE_ENVELOPE);
295 }
296
297 See ``imap/mailbox.h`` for the full list of constants.
298
299 Delayed Actions
300 ---------------
301
302 Here's the bit you've been waiting for! What happens during unlock and
303 close
304
305 first, unlock
306 ~~~~~~~~~~~~~
307
308 Anything that makes any changes sets the mailbox->has\_changed flag. If
309 this is set, then before the index gets unlocked:
310
311 - the updatenotifier (idle) is called
312 - ``sync_log_mailbox`` (replication) gets called
313 - the statuscache value gets erased (or replaced if you passed in an
314 updated value).
315
316 then: close
317 ~~~~~~~~~~~
318
319 next the index is unlocked (see above)
320
321 third, any "unlink" commands scheduled for email files are run. These
322 can't be done until after the mailbox\_commit to ensure consistency -
323 the file isn't deleted until the record is written as unlinked! But we
324 save the unlink until now so that other tasks aren't waiting for the
325 index lock while the unlinks run. Unlink is expensive in IO and time.
326
327 finally we check for MAILBOX\_NEEDS\_REPACK or MAILBOX\_DELETED option
328 flags. If either is sets, then we make a non-blocking attempt to get an
329 exclusive namelock. If the non-blocking attempt fails, then another
330 process has the mailbox open, so save the cleanup for them! If it
331 succeeds, then go ahead with either ``mailbox_delete_cleanup`` or
332 ``mailbox_index_repack`` as appropriate.
333
334 After this it's just a matter of releasing malloc'd memory and finally
335 releasing the name lock.