Debian Codegraph @

format_docs/pdb/mobi.txt

raw
1 from (https://wiki.mobileread.com/wiki/MOBI)
2
3 About
4 -----
5
6 MOBI is the format used by the MobiPocket Reader. It may have a .mobi
7 extension or it may have a .prc extension. The extension can be changed by the
8 user to either of the accepted forms. In either case it may be DRM protected or
9 non-DRM. The .prc extension is used because the PalmOS doesn't support any file
10 extensions except .prc or .pdb. Note that Mobipocket prohibits their DRM format
11 to be used on dedicated eBook readers that support other DRM formats.
12
13
14 Description
15 -----------
16
17 MOBI format was originally an extension of the PalmDOC format by adding
18 certain HTML like tags to the data. Many MOBI formatted documents still use
19 this form. However there is also a high compression version of this file format
20 that compresses data to a larger degree in a proprietary manner. There are some
21 third party programs that can read the eBooks in the original MOBI format but
22 there are only a few third party program that can read the eBooks in the new
23 compressed form. The higher compression mode is using a huffman coding scheme
24 that has been called the Huff/cdic algorithm.
25
26 From time to time features have been added to the format so new files may have
27 problems if you try and read them with a down level reader. Currently the
28 source files follow the guidelines in the Open eBook format.
29
30 Note that AZW for the Amazon Kindle is the same format as MOBI except that it
31 uses a slightly different DRM scheme.
32
33
34 Format
35 ------
36
37 Like PalmDOC, the Mobipocket file format is that of a standard Palm Database
38 Format file. The header of that format includes the name of the database
39 (usually the book title and sometimes a portion of the authors name) which is
40 up to 31 bytes of data. The files are identified as Creator ID of MOBI and a
41 Type of BOOK.
42
43
44 PalmDOC Header
45 --------------
46
47 The first record in the Palm Database Format gives more information about the
48 Mobipocket file. The first 16 bytes are almost identical to the first sixteen
49 bytes of a PalmDOC format file.
50
51 bytes content comments
52 2 Compression 1 == no compression, 2 = PalmDOC compression,
53 17480 = HUFF/CDIC compression.
54 2 Unused Always zero
55 4 text length Uncompressed length of the entire text of the book
56 2 record count Number of PDB records used for the text of the book.
57 2 record size Maximum size of each record containing text, always
58 4096.
59 4 Current Position Current reading position, as an offset into the
60 uncompressed text
61
62 There are two differences from a Palm DOC file. There's an additional
63 compression type (17480), and the Current Position bytes are used for a
64 different purpose:
65
66 bytes content comments
67 2 Encryption Type 0 == no encryption, 1 = Old Mobipocket Encryption,
68 2 = Mobipocket Encryption.
69 2 Unknown Usually zero
70
71 The old Mobipocket Encryption scheme only allows the file to be registered
72 with one PID, unlike the current encryption scheme that allows multiple PIDs to
73 be used in a single file. Unless specifically mentioned, all the encryption
74 information on this page refers to the current scheme.
75
76
77 MOBI Header
78 -----------
79
80 Most Mobipocket file also have a MOBI header in record 0 that follows these
81 16 bytes, and newer formats also have an EXTH header following the MOBI header,
82 again all in record 0 of the PDB file format.
83
84 The MOBI header is of variable length and is not documented. Some fields have
85 been tentatively identified as follows:
86
87 offset bytes content comments
88 16 4 identifier The characters M O B I
89 20 4 header length The length of the MOBI header, including
90 the previous 4 bytes
91 24 4 Mobi type The kind of Mobipocket file this is
92 2 Mobipocket Book
93 3 PalmDoc Book
94 4 Audio
95 257 News
96 258 News_Feed
97 259 News_Magazine
98 513 PICS
99 514 WORD
100 515 XLS
101 516 PPT
102 517 TEXT
103 518 HTML
104 28 4 text Encoding 1252 = CP1252 (WinLatin1); 65001 = UTF-8
105 32 4 Unique-ID Some kind of unique ID number (random?)
106 36 4 Generator version Potentially the version of the
107 Mobipocket-generation tool. Always >=
108 the value of the "format version" field
109 and <= the version of mobigen used to
110 produce the file.
111 40 40 Reserved All 0xFF. In case of a dictionary, or
112 some newer file formats, a few bytes are
113 used from this range of 40 0xFFs
114 80 4 First Non-book index? First record number (starting with 0)
115 that's not the book's text
116 84 4 Full Name Offset Offset in record 0 (not from start of
117 file) of the full name of the book
118 88 4 Full Name Length Length in bytes of the full name of the
119 book
120 92 4 Language Book language code. Low byte is main
121 language 09= English, next byte is
122 dialect, 08 = British, 04 = US
123 96 4 Input Language Input language for a dictionary
124 100 4 Output Language Output language for a dictionary
125 104 4 Format version Potentially the version of the
126 Mobipocket format used in this file.
127 Always >= 1 and <= the value of the
128 "generator version" field.
129 108 4 First Image record First record number (starting with 0)
130 that contains an image. Image records
131 should be sequential. If there are
132 no images this will be 0xffffffff.
133 112 4 HUFF record Record containing Huff information
134 used in HUFF/CDIC decompression.
135 116 4 HUFF count Number of Huff records.
136 122 4 DATP record Unknown: Records starts with DATP.
137 124 4 DATP count Number of DATP records.
138 128 4 EXTH flags Bitfield. if bit 6, 0x40 is set, then
139 there's an EXTH record
140 The following records are only present if the mobi header is long enough.
141 132 36 ? 32 unknown bytes, if MOBI is long enough
142 168 4 DRM Offset Offset to DRM key info in DRMed files.
143 0xFFFFFFFF if no DRM
144 172 4 DRM Count Number of entries in DRM info.
145 174 4 DRM Size Number of bytes in DRM info.
146 176 4 DRM Flags Some flags concerning the DRM info.
147 180 6 ?
148 186 2 Last Image record Possible value with the last image
149 record. If there are no images in the
150 book this will be 0xffff.
151 188 4 ?
152 192 4 FCIS record Unknown. Record starts with FCIS.
153 196 4 ?
154 200 4 FLIS record Unknown. Records starts with FLIS.
155 204 ? ? Bytes to the end of the MOBI header,
156 including the following if the header
157 length >= 228. ( 244 from start of
158 record)
159 242 2 Extra Data Flags A set of binary flags, some of which
160 indicate extra data at the end of each
161 text block. This only seems to be valid
162 for Mobipocket format version 5 and 6
163 (and higher?), when the header length
164 is 228 (0xE4) or 232 (0xE8).
165
166
167 EXTH Header
168 -----------
169
170 If the MOBI header indicates that there's an EXTH header, it follows immediately
171 after the MOBI header. since the MOBI header is of variable length, this isn't
172 at any fixed offset in record 0. Note that some readers will ignore any EXTH
173 header info if the mobipocket version number specified in the MOBI header is 2
174 or less (perhaps 3 or less).
175
176 The EXTH header is also undocumented, so some of this is guesswork.
177
178 bytes content comments
179 4 identifier the characters E X T H
180 4 header length the length of the EXTH header, including the previous 4 bytes
181 4 record Count The number of records in the EXTH header. the rest of the EXTH header consists of repeated EXTH records to the end of the EXTH length.
182 EXTH record start Repeat until done.
183 4 record type Exth Record type. Just a number identifying what's stored in the record
184 4 record length length of EXTH record = L , including the 8 bytes in the type and length fields
185 L-8 record data Data.
186 EXTH record end Repeat until done.
187
188 There are lots of different EXTH Records types. Ones found so far in Mobipocket
189 files are listed here, with possible meanings. Hopefully the table will be
190 filled in as more information comes to light.
191
192 record type usual length name comments
193 1 drm_server_id
194 2 drm_commerce_id
195 3 drm_ebookbase_book_id
196 100 author
197 101 publisher
198 102 imprint
199 103 description
200 104 isbn
201 105 subject
202 106 publishingdate
203 107 review
204 108 contributor
205 109 rights
206 110 subjectcode
207 111 type
208 112 source
209 113 asin
210 114 versionnumber
211 115 sample
212 116 startreading
213 117 3 adult Mobipocket Creator adds this if Adult only is checked; contents: "yes"
214 118 retail price As text, e.g. "4.99"
215 119 retail price currency As text, e.g. "USD"
216 201 4 coveroffset Add to first image field in Mobi Header to find PDB record containing the cover image
217 202 4 thumboffset Add to first image field in Mobi Header to find PDB record containing the thumbnail cover image
218 203 hasfakecover
219 204 4 Creator Software Records 204-207 are usually the same for all books from a certain source, e.g. 1-6-2-41 for Baen and 201-1-0-85 for project gutenberg, 200-1-0-85 for amazon when converted to a 32 bit integer.
220 205 4 Creator Major Version
221 206 4 Creator Minor Version
222 207 4 Creator Build Number
223 208 watermark
224 209 tamper proof keys Used by the Kindle (and Android app) for generating book-specific PIDs.
225 300 fontsignature
226 401 1 clippinglimit
227 402 publisherlimit
228 403 403 Unknown 1 - Text to Speech disabled; 0 - Text to Speech enabled
229 404 1 404 ttsflag
230 501 4 cdetype PDOC - Personal Doc;
231 EBOK - ebook;
232 502 lastupdatetime
233 503 updatedtitle
234
235 And now, at the end of Record 0 of the PDB file format, we usually get the full
236 file name, the offset of which is given in the MOBI header.
237
238
239 Variable-width integers
240 -----------------------
241
242 Some parts of the Mobipocket format encode data as variable-width integers.
243 These integers are represented big-endian with 7 bits per byte in bits 1-7. They
244 may be either forward-encoded, in which case only the LSB has bit 8 set, or
245 backward-encoded, in which case only the MSB has bit 8 set. For example, the
246 number 0x11111 would be represented forward-encoded as:
247
248 0x04 0x22 0x91
249
250 And backward-encoded as:
251
252 0x84 0x22 0x11
253
254
255 Trailing entries
256 ----------------
257
258 The Extra Data Flags field of the MOBI header indicates which, if any, trailing
259 entries are appended to the end of each text record. Each set bit in the field
260 indicates a trailing entry. The entries appear to occur in bit-order; e.g.,
261 trailing entry 1 immediately follows the text content and entry 16 occurs at
262 the very end of the record. The effect and exact details of most of these
263 entries is unknown. The trailing entries indicated by bits 2-16 appear to
264 follow a common format. That format is:
265
266 <data><size>
267
268 Where <size> is the size of the entire trailing entry (including the size of
269 <size>) as a backward-encoded Mobipocket variable-width integer.
270
271 Only a few bits have been identified
272
273 bit Data at end of records
274 0x0001 Multi-byte character overlaps
275 0x0002 Some data to help with indexing
276 0x0004 Some data about uncrossable breaks
277
278
279 Multibyte character overlap
280 ---------------------------
281
282 When bit 1 of the Extra Data Flags field is set, each record is followed by a
283 trailing entry containing any extra bytes necessary to complete a multibyte
284 character which crosses the record boundary. The bytes do not participate in
285 compression regardless which compression scheme is used for the file. However,
286 unlike the trailing data bytes, the multibytes (including the count byte) do
287 get included in any encryption. The overlapping bytes then re-appear as normal
288 content at the beginning of the following record. The trailing entry ends with
289 a byte containing a count of the overlapping bytes plus additional flags.
290
291 offset bytes content comments
292 0 0-3 N terminal bytes
293 of a multibyte
294 character
295 N 1 Size & flags bits 1-2 encode N, use of bits 3-8 is unknown
296
297
298 PalmDOC Compression
299 -------------------
300
301 PalmDOC uses LZ77 compression techniques. DOC files can contain only compressed
302 text. The format does not allow for any text formatting. This keeps files small,
303 in keeping with the Palm philosophy. However, extensions to the format can use
304 tags, such as HTML or PML, to include formatting within text. These extensions
305 to PalmDoc are not interchangeable and are the basis for most eBook Reader
306 formats on Palm devices.
307
308 LZ77 algorithms achieve compression by replacing portions of the data with
309 references to matching data that has already passed through both encoder and
310 decoder. A match is encoded by a pair of numbers called a length-distance pair,
311 which is equivalent to the statement "each of the next length characters is
312 equal to the character exactly distance characters behind it in the uncompressed
313 stream." (The "distance" is sometimes called the "offset" instead.)
314
315 In the PalmDoc format, a length-distance pair is always encoded by a two-byte
316 sequence. Of the 16 bits that make up these two bytes, 11 bits go to encoding
317 the distance, 3 go to encoding the length, and the remaining two are used to
318 make sure the decoder can identify the first byte as the beginning of such a
319 two-byte sequence. The exact algorithm needed to decode the compressed text can
320 be found on the PalmDOC page.
321
322 PalmDOC data is always divided into 4096 byte blocks and the blocks are acted
323 upon independently.
324
325 PalmDOC does have support for bookmarks. These pointers are named and refer to
326 an offset location in a file. If the file is edited these locations may no
327 longer refer to the correct locations. Some reading programs allow the user to
328 enter or edit these bookmarks while others treat them as a TOC. Some reading
329 programs may ignore them entirely. They are stored at the end of the file itself
330 so the full file needs to be scanned when loaded to find them.
331
332
333 Image Records
334 -------------
335
336 If the file contains images, they follow the text blocks, with each image using a
337 single block. The 4096-byte record size in the PalmDoc header applies only to
338 text records; image records may be larger.
339
340
341 Magic Records
342 -------------
343
344 In some cases, MobiPocket Creator adds a 2-zero-byte record after the text
345 records in a file. This record is not included in the "record count" of text
346 records in the PalmDoc header, and is also not used as the "first non-book
347 index" in the MOBI header. (If the 2-zero-byte record is present, the index of
348 the following block is used as the "first non-book index".)
349
350 MobiPocket Creator also ends files with three records: 'FLIS', 'FCIS', and
351 'end-of-file', in that order. The 'FLIS' and 'FCIS' records do not seem to be
352 necessary for MobiPocket Reader or the Amazon Kindle 2 to read the file. The
353 'end-of-file' record might be necessary.
354
355
356 FLIS Record
357 -----------
358
359 The FLIS record appears to have a fixed value. The meaning of the values is not known.
360
361 offset bytes content comments
362 0 4 identifier the characters F L I S (0x46 0x4c 0x49 0x53)
363 4 4 ? fixed value: 8
364 8 2 ? fixed value: 65
365 10 2 ? fixed value: 0
366 12 4 ? fixed value: 0
367 16 4 ? fixed value: -1
368 20 2 ? fixed value: 1
369 22 2 ? fixed value: 3
370 24 4 ? fixed value: 3
371 28 4 ? fixed value: 1
372 32 4 ? fixed value: -1
373
374
375 FCIS Record
376 -----------
377
378 The FCIS record appears to have mostly fixed values.
379
380 offset bytes content comments
381 0 4 identifier the characters F C I S (0x46 0x43 0x49 0x53)
382 4 4 ? fixed value: 20
383 8 4 ? fixed value: 16
384 12 4 ? fixed value: 1
385 16 4 ? fixed value: 0
386 20 4 ? text length (the same value as "text length" in the PalmDoc header)
387 24 4 ? fixed value: 0
388 28 4 ? fixed value: 32
389 32 4 ? fixed value: 8
390 36 2 ? fixed value: 1
391 38 2 ? fixed value: 1
392 40 4 ? fixed value: 0
393
394
395 End-of-file Record
396 ------------------
397
398 The end-of-file record is a fixed 4-byte record. While the last two bytes
399 appear to be a CRLF marker, the meaning of the first two bytes is unknown.
400
401 offset bytes content comments
402 0 1 ? fixed value: 233 (0xe9)
403 1 1 ? fixed value: 142 (0x8e)
404 2 1 ? fixed value: 13 (0x0d)
405 3 1 ? fixed value: 10 (0x0a)
406
407
408 SRCS Record
409 -----------
410
411 kindlegen creates a record whose content is a zip archive of all source files
412 (i.e., .opf, .ncx, .htm, .jpg, ...) given to the command and puts it in the
413 generated MOBI file. The record begins with the "SRCS" signature and is
414 located just before the #End-of-file Record.
415
416 MOBI files created with Mobipocket creator, Amazon's Personal Document Service,
417 or Kindle Direct Publishing (former Amazon DTP) don't include SRCS record.
418 In a past, kindlegen had an undocumented option to suppress this record, but
419 the option was removed in 2010.
420
421 offset bytes content comments
422 0 4 identifier "SRCS" (0x53 0x52 0x43 0x53)
423 4 4 ? fixed value(?): 0x00000010
424 8 4 ? fixed value(?): 0x0000002f
425 12 4 ? fixed value(?): 0x00000001
426 16 zip The zip archive continues to the end of this record
427
428
429 MBP
430 ---
431
432 This is the extension used on a side file (auxiliary) for MOBI formatted eBooks.
433 It is used to store metadata used by the library software and also to store
434 user entered data like bookmarks, annotations, last read position. This file is
435 created automatically by the reader program when the eBook is first opened and
436 has a .mbp extension. The Library management software in MobiPocket uses this
437 file to get information displayed in the library window such as title and author
438 so that it won't have to open the larger eBook file.
439