Debian Codegraph @

doc/html/_sources/imap/developer/guidance/mailbox-format.rst.txt

raw
1 .. _imap-developer-guidance-mailbox-format:
2
3 Cyrus IMAP Server: Mailbox File Formats
4 =======================================
5
6 Intro
7 -----
8
9 This documentation refers to the "version 12" cyrus index format and
10 associated mailbox files.
11
12 No external tools should make use of this information. The only
13 supported method of access to the mail store is through the standard
14 interfaces: IMAP, POP, NNTP, LMTP, etc.
15
16 A cyrus mailbox is a directory in the filesystem. It contains the
17 following files:
18
19 - zero or more message files
20 - the ``cyrus.header`` metadata file
21 - the ``cyrus.index`` metadata file
22 - the ``cyrus.cache`` metadata file
23 - zero or one ``cyrus.squat`` search indexes
24 - zero or more subdirectories
25
26 With "split metadata" configuration, the mailbox may actually be split
27 between multiple disks, with the files being in the same relative
28 directory on the meta disk. See the ``imapd.conf`` option
29 ``metapartition_files`` for more information
30
31 Message Files
32 -------------
33
34 The message files are named by their UID, followed by a ".", so UID 423
35 would be named "``423.``". They are stored in wire-format: lines are
36 terminated by CRLF and binary data is not allowed.
37
38 ``cyrus.header``
39 ----------------
40
41 This file contains mailbox-wide information that does not change often.
42 Its format:
43
44 ::
45
46 <Mailbox Header Magic String>
47 <Quota Root>\t<Mailbox Unique ID String>\n
48 <Space-separated list of user flags>\n
49 <Mailbox ACL>\n
50
51 The Mailbox Unique ID String is used for non-owner per-user \\Seen flags
52 so they remain with the mailbox during renames, and also by the
53 replication subsystem to detect mailbox renames.
54
55 The ACL is a copy of the value stored in mailboxes.db, and isn't
56 actually used.
57
58 Locking Considerations
59 ~~~~~~~~~~~~~~~~~~~~~~
60
61 The ``cyrus.index`` file must be locked in exclusive mode while making
62 changes to the ``cyrus.header`` file to ensure consistency. All changes
63 are made by rewriting the entire file and renaming the new version into
64 place.
65
66 ``cyrus.cache``
67 ---------------
68
69 The ``cyrus.cache`` file is a pure cache of information that's also
70 present in the message files. It exists to make ENVELOPE and specific
71 header fetches more efficient, as well as to assist with searches and
72 sorts.
73
74 If a ``cyrus.cache`` file is missing or corrupted, it can be
75 re-generated by running a ``reconstruct`` on the mailbox.
76
77 The format is 10 individual records each prefixed with a 32 bit length
78 value in network byte order. The offset of each message's cache record
79 is stored in the ``cyrus.index`` file (documented below). The records in
80 a cyrus.cache file are of variable length, depending on the contents of
81 the associated message.
82
83 The first 4 bytes of the cyrus.cache file are a "generation number"
84 which must match the first 4 bytes of the associated cyrus.index file.
85 In the past this was used to track consistency between the files, but
86 the name locking scheme and per-record CRC check in cyrus 2.4 and above
87 means this is just a backup consistency check rather than an essential
88 format feature.
89
90 ::
91
92 +------------------------------------------------------------------------+
93 |Gen # (32bits)|Size 1 (32bits)|Data 1 |
94 +------------------------------------------------------------------------+
95 | |Size 2 (32bits)|Data 2 |Size 3 (32bits)| Data 3 |
96 +------------------------------------------------------------------------+
97 | ..... |
98 +------------------------------------------------------------------------+
99
100 While there are occasional changes to the cache format, this information
101 is NOT stored in the cyrus.cache file. Instead, there is a
102 "cache\_version" field in the cyrus.index record, so multiple different
103 versions of cache data may exist in the same cache file.
104
105 The order of fields per record in the cache file is as follows: (keep in
106 mind that they are all preceded by a 4 byte network byte order size).
107
108 Envelope Response
109 Raw IMAP response for a request for the envelope.
110 Bodystructure Response
111 Raw IMAP response for a request for the bodystructure.
112 Body Response
113 Raw IMAP response for an (old style) request for the body.
114 Binary Bodystructure
115 Offsets into the message file to pull out various body parts.
116 Because of the nature of MIME parts, this is somewhat recursive.
117
118 This looks like the following (starting the octet following the
119 cache field size). All of the fields are bit32s.
120
121 ::
122
123 [
124 [Number of message parts+1 for the rfc822 header if present]
125 [
126 [Offset in the message file of the header of this part]
127 [Size (octets) of the header of this part]
128 [Offset in the message file of the content of this part]
129 [Size (octets) of the content of this part]
130 [Encoding Type of this part]
131 ]
132 (repeat for each part as well as once for the headers)
133 [zero *or* number of sub-parts in the case of a multipart.
134 if nonzero, this is a recursion into the top structure]
135 (repeat for each part)
136 ]
137
138 Note if this is not a message/rfc822, than the values for the sizes
139 of the part 0 are -1 (to indicate that it doesn't exist). Sub-parts
140 are not possible for a part 0, so they aren't included when finding
141 recursive entries.
142
143 The offset and size info for both the mime header and content part
144 are useful in order to do fast indexing on the appropriate parts of
145 the message file when a client does a FETCH request for
146 BODY[HEADER], or BODY[2.MIME].
147
148 Note that the top level RFC822 headers are a treated as a separate
149 part from their body text ("0" or "HEADER").
150
151 In the case of a multipart/alternative, the content size & offset
152 refers to the size of the entire mime part.
153
154 A very simple message (with a single text/plain part) would
155 therefore look like:
156
157 ::
158
159 [[2][rfc822 header][text/plain body part info][0]]
160
161 A simple multipart/alternative message might look like:
162
163 ::
164
165 [[3][rfc822 header][text/plain message part info]
166 [second message part info][0][0]]
167
168 A message with an attachment that has two subparts:
169
170 ::
171
172 [[3][rfc822 header info][rfc822 first body part info][attachment info][0][
173 [3][NIL header info][sub part 1 info][sub part 2 info][0][0]]]
174
175 A message with an attached message/rfc822 message with the following
176 total structure:
177
178 ::
179
180 message/rfc822
181 0 headers; content-type: multipart/mixed
182 1 text/plain
183 2 message/rfc822
184 0 headers; content-type: multipart/alternative
185 1 text/plain
186 2 text/html
187
188 ::
189
190 [[3][rfc822 header part 0][text/plain part 1][overall attachment info][0][
191 [3][rfc822 header part 2.0][text/plain part 2.1][text/html part 2.2]
192 [0][0]]]
193
194 Cache Header
195 Any cached header fields. The exact set of fields here depends on
196 the cache record version - there is a function in ``imap/mailbox.c``
197 to determine if a named header would be cached based on the version.
198 These are in the same format they would appear in the message file:
199
200 ::
201
202 HeaderName: headerdata\r\n
203
204 Examples include: References, In-Reply-To, etc.
205
206 From
207 The from header.
208 To
209 The to header.
210 Cc
211 The CC header.
212 Bcc
213 The BCC header.
214 Subject
215 The Subject header.
216
217 Locking Considerations
218 ~~~~~~~~~~~~~~~~~~~~~~
219
220 The ``cyrus.index`` file must be locked in exclusive mode while making
221 changes to the ``cyrus.cache`` file to ensure consistency. All new cache
222 records are created by reading the current end-of-file offset, appending
223 the new cache record, and storing that start offset into the associated
224 cyrus.index record.
225
226 ``cyrus.index``
227 ---------------
228
229 The cyrus.index file is NOT just a cache - it stores information not
230 present in the message file!
231
232 The cyrus.index file consists of a fixed width header, followed by fixed
233 width records. In the past, it would be rewritten on every expunge, but
234 since Cyrus 2.4 the expunged records remain in the cyrus.index file for
235 a configurable time to support QRESYNC and more efficient delayed
236 expunge.
237
238 The cyrus.index file is the "heart" of the mailbox format - containing
239 checksums (CRC32) of everything else, and the most frequently updated
240 fields. All fields are stored in network byte order and aligned on 4
241 byte boundaries. Due to some 64 bit values being stored, the header and
242 individual records are aligned on 8 byte boundaries.
243
244 The overall format looks sort of like this:
245
246 ::
247
248 cyrus.index:
249 +----------------+
250 | Mailbox Header |
251 +----------------+
252 | Msg: Num 1 |
253 +----------------+
254 | Msg: Num 2 |
255 +----------------+
256 | ... |
257 +----------------+
258
259 The basic idea being that there is one header, and then all the message
260 records are evenly spaced throughout the file. All of the message
261 records are at well-known offsets, making any part of the file
262 accessable at roughly equal speed.
263
264 Locking Considerations
265 ~~~~~~~~~~~~~~~~~~~~~~
266
267 ``cyrus.index`` files can not be repacked (i.e. records can not change
268 UID for a particular offset, and the file can't be rewritten or deleted)
269 unless there's an exclusive namelock held for the mailbox name. This is
270 to avoid race conditions and simplify the use of mailboxes. Whenever a
271 mailbox is opened, the caller holds a shared namelock on the mailbox
272 name for the duration of the "mailbox object"'s existence.
273
274 All reads of a ``cyrus.index`` file must be done with a lock held, and
275 all writes must be done with an exclusive lock held. This ensures CRC32
276 checksums of individual headers and records are always consistent. There
277 are no direct "offset" reads done any more, instead the mailbox API
278 provides a way to read an entire cyrus.index header or cyrus.index
279 record into a struct, performing consistency checks. Writes are also
280 done with a complete record struct.
281
282 Detail of ``cyrus.index`` header
283 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
284
285 The index header contains the following information, in order:
286
287 Generation Number (4 bytes)
288 A number that is basically the "revision number" of the mailbox. It
289 must match between the cache and index files. This is to ensure that
290 if we fail to sync both the cache and index files and a crash
291 happens (so that only one is synced), we do not provide bad data to
292 the user. This is also backed by having individual cache checksums
293 on each record.
294 Format (4 bytes)
295 Basically obsolete (indicates netnews or regular).
296 Minor Version (4 bytes)
297 Indicates the version number of the index file. This can be used for
298 on-the-fly upgrades of the index and cache files.
299 Start Offset (4 bytes)
300 Size of index header.
301 Record Size (4 bytes)
302 Size of an index record.
303 Num Records (4 bytes)
304 How many records are in this index (including records for expunged
305 records. See below for "Exists" which has moved from pre-version 12
306 files.
307 Last Appenddate (4 bytes)
308 (time\_t) of the last time a message was appended
309 Last UID (4 bytes)
310 Highest UID of all messages in the mailbox (UIDNEXT - 1).
311 Quota Mailbox Used (8 bytes)
312 Total amount of storage used by all of the messages in the mailbox.
313 Platforms that don't support 64-bit integers only use the last 4
314 bytes.
315 POP3 Last Login (4 bytes)
316 (time\_t) of the last pop3 login to this INBOX, used to enforce the
317 "poptimeout" ``imapd.conf`` option.
318 UIDvalidity (4 bytes)
319 The UID validitiy of this mailbox. Cyrus currently uses the
320 ``time()`` when this mailbox was created.
321 Deleted, Answered, and Flagged (4 bytes each)
322 Counts of how many messages have each flag.
323 Mailbox Options (4 bytes)
324 Bitmask of mailbox options, consisting of any combination of the
325 following:
326
327 POP3\_NEW\_UIDL
328 Flag signalling that we're using "*uidvalidity*.\ *uid*" instead
329 of just "*uid*" for the output of the POP3 UIDL command.
330 IMAP\_SHAREDSEEN
331 Flag signalling that we're supporting a shared \\Seen flag on
332 the mailbox.
333 IMAP\_DUPDELIVER
334 Flag signalling that we're allowing duplicate delivery of
335 messages to the mailbox, overriding system-wide duplicate
336 suppression.
337 MAILBOX\_NEEDS\_REPACK
338 Flag signalling that the mailbox is due to be repacked. During
339 mailbox\_close() every process will attempt to take an exclusive
340 namelock on the mailbox and repack.
341 MAILBOX\_DELETED
342 Flag signalling that the mailbox is deleted. This can be set
343 with a shared namelock, and indicates to all other users of the
344 mailbox that they need to close it and attempt cleanup. The last
345 process to close the mailbox will perform the final cleanup
346 under an exclusive namelock, giving the other processes a chance
347 to finish their current operation first without files
348 disappearing from under them!
349
350 Leaked Cache (4 bytes)
351 Number of leaked records in the cache file.
352 Highest ModSeq (8 bytes)
353 Highest Modification Sequence of all the messages in the mailbox
354 (CONDSTORE).
355 Deleted ModSeq (8 bytes)
356 Lowest Modification Sequence before which expunged message data may
357 have been purged from the mailbox and forgotten (CONDSTORE/QRESYNC
358 support).
359 Exists (4 bytes)
360 See NumRecords above. This is the count of non-expunged records in
361 the mailbox and corresponds to the IMAP status item "EXISTS".
362 First Expunged (4 bytes)
363 lowest modified time of an expunged message in this mailbox (or zero
364 if there are no expunged messages) - used to determine if the
365 mailbox needs repacking.
366 Last Repack Time (4 bytes)
367 a timestamp for the last repack, to ensure repacks aren't done too
368 close together if expunges were closely spaced
369 Header File CRC (4 bytes)
370 CRC32 value of the bytes in the ``cyrus.header`` file for this
371 mailbox. Must be rewritten whenever the cyrus.header file is changed
372 (see locking considerations above - this is why the cyrus.index must
373 be exclusively locked!)
374 Sync CRC (4 bytes)
375 An XOR of the CRC32 of a specially generated value for each of the
376 non-expunged records in this mailbox. This is a cached value which
377 allows the replication subsystem to quickly determine that all
378 non-expunged records in a mailbox are in sync and detect possible
379 "split brain" scenarios with low bandwidth use.
380 Recent UID (4 bytes)
381 The highest UID last time an IMAP client logged in as the mailbox
382 owner (or anybody if SHAREDSEEN is enabled) selected this mailbox.
383 Used to generate the \\Recent flags in IMAP
384 Recent Time (4 bytes)
385 Used for consistency with the seen\_db code, but probably not
386 actually necessary. Oh well
387 Header CRC (4 bytes)
388 Must always be the LAST record of the header. This is the CRC32 of
389 the actual bytes on disk (network order format) for the rest of the
390 cyrus.index. By keeping it last, it can be easily calculated with
391 the following snippet of code:
392 ``crc = crc32_map(buf, OFFSET_HEADER_CRC);`` - i.e. crc32 from the
393 start of the buffer to just before this field.
394
395 There are also spare fields in the index header, to allow for future
396 expansion without forcing an upgrade of the file, and to round up to be
397 divisible by 8 bytes.
398
399 Detail of ``cyrus.index`` records
400 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
401
402 These records start immediately following the ``cyrus.index`` header,
403 and are all fixed size. They are in-order by uid of the message.
404
405 UID (4 bytes)
406 UID of the message
407 INTERNALDATE (4 bytes)
408 INTERNALDATE of the message (where possible, this matches the
409 creation and modification times of the file on disk to help
410 reconstruct in the event of data loss)
411 SENTDATE (4 bytes)
412 Contents of the Date: header chomped to day resolution with timezone
413 stripped.
414 SIZE (4 bytes)
415 Size of the whole message (in octets)
416 HEADER SIZE (4 bytes)
417 Size of the message header (in octets)
418 GMTIME (4 bytes)
419 Contents of the Date: header at 1 second resolution and converted to
420 GMT (for sort)
421 CACHE\_OFFSET (4 bytes)
422 Offset into the ``cyrus.cache`` file for the beginning of this
423 message's cache entry.
424 LAST UPDATED (4 bytes)
425 (time\_t) of the last time this record was changed
426 SYSTEM FLAGS (4 bytes)
427 Bitmask showing which system flags are set/unset
428 USER FLAGS (MAX\_USER\_FLAGS / 32 bytes)
429 Bitmask showing which user flags are set/unset (bits correspond to
430 positions in the cyrus.header flag list, i.e. (1<<0) == the flag
431 name
432 CONTENT\_LINES (4 bytes)
433 Number of text lines contained in the message content (body).
434 CACHE\_VERSION (4 bytes)
435 Indicates the version number of the cache record for the message
436 (determines which headers are cached, see list in mailbox.c).
437 GUID (MESSAGE\_GUID\_SIZE bytes)
438 Globally Unique IDentifier of the message (used by replication
439 engine). This is the sha1 value of the bytes as stored on disk.
440 MODSEQ (8 bytes)
441 Modification Sequence of the message (CONDSTORE).
442 CACHE\_CRC (4 bytes)
443 This is the CRC32 of all the bytes of the cache record (all 10
444 fields) as stored on disk. Again, calculated over the exact bytes
445 stored in the ``cyrus.cache`` file.
446 RECORD\_CRC (4 bytes)
447 Like the header CRC - this is the CRC32 of all the bytes in on-disk
448 order that exist in this record. Records are always rewritten as the
449 entire record, including the updated CRC, so it's always consistent
450 if you have a lock on the ``cyrus.index`` file, because writers will
451 wait until they get an exclusive lock to make modifications.
452
453 Notes
454 -----
455
456 - Expunge is super quick now - it's just a flag update!
457 - Append is relatively fast (it only adds to the end of both the cache
458 and index files and modifies the index header)
459 - Message unlinks always happen during the "close" phase - which may be
460 noticed when you select another mailbox, but otherwise are delayed
461 from the actual action. With delayed expunge, the unlinks are pushed
462 off to cyr\_expire which is a background task, and will never be
463 noticed by the user.
464 - Message delivery is something like this:
465
466 #. write/sync message file
467 #. write/sync new ``cyrus.cache`` record
468 #. write/sync new ``cyrus.index`` record
469 #. calculate, write, sync new ``cyrus.index`` header
470 #. acknowledge message delivery
471
472 The message isn't delivered until the new index header is written. In
473 case of a crash before the new index header is written, any previous
474 writes will be overwritten on the next delivery (and will not be
475 noticed by the readers).
476
477 Note that certain power failure situations (power failure in the
478 middle of a disk sector write) could cause a mailbox to need
479 reconstruction (possibly even losing some flag state). These failure
480 modes are not possible in the "Hardware RAID disk model" (which we
481 will describe somewhere else when we get around to it).
482
483 Future considerations
484 ---------------------
485
486 - Cache all header fields? (or all up to Xk?) This could greatly
487 improve speeds of clients that just ask for everything, but also
488 increases the expense of rewriting the cache file (as well as the
489 size it takes on disk).
490 - Reformat cache file to use a
491 (size)(size)(size)(size)(data)(data)(data) format. This makes
492 accesses anywhere in the cache file equally fast, as opposed to
493 having to iterate through all the entires for a given message to get
494 to the last one. Note that either way is still O(1) so maybe it
495 doesn't matter much.
496 - It would be useful to store a uniqueid -> mailbox name index, so that
497 we could fix arbitron again.