| 1 |
The zTXT Format |
| 2 |
--------------- |
| 3 |
|
| 4 |
The zTXT format is relatively straightforward. The simplest zTXT contains a |
| 5 |
Palm database header, followed by zTXT record #0, followed by the compressed |
| 6 |
data. The compressed data can be in one of two formats: one long data stream, |
| 7 |
or split into chunks for random access. If there are any bookmarks, they occupy |
| 8 |
the record immediately after the compressed data. If there are any annotations, |
| 9 |
the annotation index occupies the record immediately after the bookmarks with |
| 10 |
each annotation in the index having a record immediately after the annotation |
| 11 |
index. Here are diagrams of a simple zTXT and a full featured zTXT: |
| 12 |
|
| 13 |
DB Header |
| 14 |
0 Record 0 |
| 15 |
1 |
| 16 |
2 |
| 17 |
3 |
| 18 |
... Compressed Data |
| 19 |
36 |
| 20 |
37 |
| 21 |
38 |
| 22 |
|
| 23 |
DB Header |
| 24 |
0 Record 0 |
| 25 |
1 |
| 26 |
2 |
| 27 |
3 |
| 28 |
... Compressed Data |
| 29 |
36 |
| 30 |
37 |
| 31 |
38 |
| 32 |
39 Bookmarks |
| 33 |
40 Annotation Index |
| 34 |
41 Annotation 1 |
| 35 |
42 Annotation 2 |
| 36 |
43 Annotation 3 |
| 37 |
|
| 38 |
|
| 39 |
Compression Modes |
| 40 |
----------------- |
| 41 |
|
| 42 |
zTXT version 1.40 and later supports two modes of compression. Mode 1 is a |
| 43 |
random access mode, and mode 2 consists of one long data stream. Both modes |
| 44 |
work on 8K (the default record size) blocks of text. |
| 45 |
|
| 46 |
Please note, however, that as of Weasel Reader version 1.60 the old style |
| 47 |
(mode 2) zTXT format is no longer supported. makeztxt and libztxt still support |
| 48 |
creating these documents for backwards compatibility, but you should not use |
| 49 |
mode 2 if possible. |
| 50 |
|
| 51 |
|
| 52 |
Mode 1 |
| 53 |
------ |
| 54 |
|
| 55 |
In mode one, 8K blocks of text are compressed into an equal number of blocks of |
| 56 |
compressed data. Using the Z_FULL_FLUSH flush mode with zLib allows for random |
| 57 |
access among the blocks of data. In order for this to function, the first block |
| 58 |
must be decompressed first, and after that any block in the file may be |
| 59 |
decompressed in any order. In mode 1, the blocks of compressed data will likely |
| 60 |
not all have the same size. |
| 61 |
|
| 62 |
|
| 63 |
Mode 2 |
| 64 |
------ |
| 65 |
|
| 66 |
In zTXT versions before 1.40, this was the only method of compression. This |
| 67 |
mode involves compressing the entire input buffer into a single output buffer |
| 68 |
and then splitting the resulting buffer into 8K segments. This mode requires |
| 69 |
that all of the compressed data be decompressed in one pass. Since there are no |
| 70 |
real 'blocks' of data, the resulting output can be of any blocksize, though |
| 71 |
typically the default of 8K should be fine. The advantage to mode 2 is that it |
| 72 |
will give about 10% - 15% more compression. |
| 73 |
|
| 74 |
|
| 75 |
zTXT Record #0 Definition (version 1.44) |
| 76 |
---------------------------------------- |
| 77 |
|
| 78 |
Record 0 provides all of the information about the zTXT contents. Be sure it is |
| 79 |
correct, lest firey death rain down upon your program. |
| 80 |
|
| 81 |
typedef struct zTXT_record0Type { |
| 82 |
UInt16 version; |
| 83 |
UInt16 numRecords; |
| 84 |
UInt32 size; |
| 85 |
UInt16 recordSize; |
| 86 |
UInt16 numBookmarks; |
| 87 |
UInt16 bookmarkRecord; |
| 88 |
UInt16 numAnnotations; |
| 89 |
UInt16 annotationRecord; |
| 90 |
UInt8 flags; |
| 91 |
UInt8 reserved; |
| 92 |
UInt32 crc32; |
| 93 |
UInt8 padding[0x20 - 24]; |
| 94 |
} zTXT_record0; |
| 95 |
|
| 96 |
|
| 97 |
Structure Elements |
| 98 |
------------------ |
| 99 |
|
| 100 |
UInt16 version; |
| 101 |
|
| 102 |
This is mostly just informational. Your program can figure out what features |
| 103 |
might be available from the version. However, the remaining parts of the |
| 104 |
structure are designed such that their value will be 0 if that particular |
| 105 |
feature is not present, so that is the correct way to test. The version is |
| 106 |
stored as two 8 bit integers. For example, version 1.42 is 0x012A. |
| 107 |
|
| 108 |
UInt16 numRecords; |
| 109 |
|
| 110 |
This is the number of DATA records only and does not include record 0, |
| 111 |
bookmarks, or annotations. With compression mode 1, this is also the number of |
| 112 |
uncompressed text records. With mode 2, you must decompress the file to figure |
| 113 |
out how many text records there will be. |
| 114 |
|
| 115 |
UInt32 size; |
| 116 |
|
| 117 |
The size in bytes of the uncompressed data in the zTXT. Check this value with |
| 118 |
the amount of free storage memory on the Palm to make sure there's enough room |
| 119 |
to decompress the data in full or in part. |
| 120 |
|
| 121 |
UInt16 recordSize; |
| 122 |
|
| 123 |
recordSize is the size in bytes of a text record. This field is important, as |
| 124 |
the size of text and decompression buffers is based on this value. It is used |
| 125 |
by Weasel to navigate though the text so it can map absolute offsets to record |
| 126 |
numbers. 8192 is the default. With compression mode 1, this is the amount of |
| 127 |
data inside each compressed record (except maybe the last one), but the actual |
| 128 |
compressed records will likely have varying sizes. In mode 2, both compressed |
| 129 |
records and the resulting text records are all of this size (except, again, the |
| 130 |
last record). |
| 131 |
|
| 132 |
UInt16 numBookmarks; |
| 133 |
|
| 134 |
The definitive count of how many bookmarks are stored in the bookmark index |
| 135 |
record. See the section on bookmarks below. |
| 136 |
|
| 137 |
UInt16 bookmarkRecord; |
| 138 |
|
| 139 |
If there are any bookmarks, this is set to the record index number that |
| 140 |
contains the bookmark listing, otherwise it is 0. |
| 141 |
|
| 142 |
UInt16 numAnnotations; |
| 143 |
|
| 144 |
Like the bookmark count, this is the definitive count of how many annotations |
| 145 |
are in the annotation index and how many annotation records follow it. See the |
| 146 |
section on annotation below. |
| 147 |
|
| 148 |
UInt16 annotationRecord; |
| 149 |
|
| 150 |
If there are any annotations, this is set to the record index number that |
| 151 |
contains the annotation index, otherwise it is 0. |
| 152 |
|
| 153 |
UInt8 flags; |
| 154 |
|
| 155 |
These flags indicate various features of the zTXT database. flags is a bitmask |
| 156 |
and at present the only two defined bits are: |
| 157 |
|
| 158 |
ZTXT_RANDOMACCESS (0x01) |
| 159 |
If the zTXT was compressed according to the method in mode 1, then it |
| 160 |
supports random access and this should be set. |
| 161 |
ZTXT_NONUNIFORM (0x02) |
| 162 |
Setting this bit indicates that the text records within the zTXT database |
| 163 |
are not of uniform length. That is, when the blocks of text are |
| 164 |
decompressed they will not have identical block sizes. If this is not set, |
| 165 |
the compressed blocks are assumed to all have the same size when |
| 166 |
decompressed (typically 8K) except for the last block which can be smaller. |
| 167 |
|
| 168 |
UInt32 crc32; |
| 169 |
|
| 170 |
A CRC32 value for checking data integrity. This value is computer over all text |
| 171 |
data record only and does not include record 0 nor any bookmark/annotation |
| 172 |
records. The current implementation in makeztxt/Weasel computes this value |
| 173 |
using the crc32 function in zLib which should be the standard CRC32 definition. |
| 174 |
|
| 175 |
UInt8 padding[0x20 - 24]; |
| 176 |
|
| 177 |
zTXT record zero is 32 bytes in length, so the unused portion is padded. |
| 178 |
|
| 179 |
|
| 180 |
zTXT Bookmarks |
| 181 |
-------------- |
| 182 |
|
| 183 |
zTXT bookmarks are stored in a simple array in a record at the end of a zTXT. |
| 184 |
The format is as follows: |
| 185 |
|
| 186 |
#define MAX_BMRK_LENGTH 20 |
| 187 |
|
| 188 |
typedef struct GPlmMarkType { |
| 189 |
UInt32 offset; |
| 190 |
Char title[MAX_BMRK_LENGTH]; |
| 191 |
} GPlmMark; |
| 192 |
|
| 193 |
In the structure, offset is counted as an absolute offset into the text. The |
| 194 |
bookmarks must be sorted in ascending order. |
| 195 |
|
| 196 |
If there are no bookmarks, then the bookmark index does not exist. When the |
| 197 |
user creates the first bookmark, the record containing the index will then be |
| 198 |
created. If there are annotations, when the bookmark record is created it must |
| 199 |
go before the annotation index. This will require incrementing annotationRecord |
| 200 |
in record 0 to point to the new record index. |
| 201 |
|
| 202 |
Similarly, when all bookmarks are deleted the bookmark index record is also |
| 203 |
deleted. If there are annotations, annotationRecord in record 0 must be |
| 204 |
decremented to point to the new index. |
| 205 |
|
| 206 |
|
| 207 |
zTXT Annotations |
| 208 |
---------------- |
| 209 |
|
| 210 |
zTXT annotations have a format almost identical to that of the bookmark index: |
| 211 |
|
| 212 |
typedef struct GPlmAnnotationType { |
| 213 |
UInt32 offset; |
| 214 |
Char title[MAX_BMRK_LENGTH]; |
| 215 |
} GPlmAnnotation; |
| 216 |
|
| 217 |
Like the bookmarks, offset is an absolute offset into the text. The annotation |
| 218 |
index is organized just as the bookmarks are, as a single array in a record. |
| 219 |
Note that this structure does NOT store the actual annotation text. |
| 220 |
|
| 221 |
The text of each annotation is stored in its own record immediately following |
| 222 |
the index. So, the first annotation in the index will occupy the first record |
| 223 |
following the index, and the second annotation will be in the second record |
| 224 |
following the index, and so on. The text of each annotation is limited to |
| 225 |
4096 bytes. |
| 226 |
|