Debian Codegraph @

README

raw
1 README for JTE 1.22
2
3 Steve McIntyre <[email protected]>
4 20 November 2019
5
6 License - GPL v2+. See the file COPYING for more details.
7
8 JTE - Jigdo Template Export
9 ===========================
10
11 Introduction - jigdo and JTE
12 ----------------------------
13
14 Jigdo is a useful tool to help in the distribution of large files like CD and
15 DVD images. See Richard Atterer's site for more details. Debian CDs and DVD ISO
16 images are published on the web in jigdo format to allow end users to download
17 them more efficiently.
18
19 Jigdo is generic and powerful - it can be used for any large files that are
20 made up of smaller files. However, to be this generic is costly. Creating jigdo
21 files from ISO images is quite inefficient - to work out which files are
22 included in the ISO image, jigdo has to calculate and compare checksums of
23 every possible file and every extent in the image. Essentially it has to
24 brute-force the image. It can take a long time to do this for a large image
25 (imagine a 4.5GB DVD image or a 30+GB Blu-Ray image).
26
27 I first started looking for ways to improve this back in 2004:
28
29 1. Modify jigdo so it knew about the internals of ISO images and could
30 efficiently scan them (bad, not very generic for jigdo)
31 2. Write a helper tool to dump extra information for jigdo to use alongside
32 the ISO image (I had a helper tool written, but modifying jigdo to use this
33 looked HARD)
34 3. Patch mkisofs/genisoimage to write .jigdo and .template files alongside the
35 ISO image
36
37 I completed the third of these options, and called it JTE (or Jigdo Template
38 Export). The code worked fine, and ran in a very small fraction of the time
39 taken to run genisoimage and jigdo separately. The output .jigdo and .template
40 files worked correctly, i.e. jigdo-file and the wrapper script jigdo-mirror
41 accept them and would generate an ISO image that exactly matches the original.
42
43 Debian used that code for a number of years within genisoimage, but we've since
44 switched over to using xorriso instead for our image building instead. It has a
45 lot of useful features that we want compared to genisoimage, not least a
46 friendly and engaged author in Thomas Scmitt!
47
48 Thomas and I and George Danchev worked together to package up my JTE code into
49 libjte such that xorriso could use it effectively. Xorriso has been capable of
50 generating jigdo files since 2010.
51
52 In late 2019, I took over maintenance of the jigdo upstream code and added
53 support for a new (v2) jigdo data format, using SHA256 instead of MD5
54 internally. See my jigdo page for more details about that. I have also updated
55 the JTE codebase to support this new format, of course.
56
57 As genisoimage is effectively dead at this point, I took the decision to not
58 add the jigdo v2 support into the genisoimage codebase. If you need to generate
59 jigdo v2 format, either use jigdo itself or xorriso if you'd like the
60 performance benefit of the libjte integration.
61
62 JTE includes a few tools:
63
64 • jigit-mkimage, a simple and very fast tool to reconstruct image files from
65 .jigdo and .template files. It doesn't have any logic to cope with
66 downloading missing files, but will list the missing files that are needed.
67 It is also much faster for people (like me!) who already have full local
68 mirrors.
69 • parallel-sums is a simple extra utility to generate checksums quickly and
70 efficiently, reading file data only once and calculating checksums using
71 multiple algorithms in parallel using threads.
72 • jigsum, jigsum-sha256 and rsyncsum are checksum tools which will output
73 checksums in jigdo's base64-like format rather than the normal hexadecimal
74 format. Useful for debugging jigdo issues.
75 • jigdump is a tool to dump the contents of a jigdo template or .iso.tmp
76 file. Useful for debugging jigdo issues.
77 • mkjigsnap is a utility to help with maintaining the "snapshots" that jigdo
78 needs if you're going to be keeping data around for users in the long term.
79 We use this on some Debian systems.
80
81 Why the "jigit" name? The packages and source are named jigit to match the name
82 of a long-dead wrapper script.
83
84 ----------------------------------------------------------------------
85
86 Download
87 --------
88
89 The jigit source package (and hence the various binary packages it builds) is
90 included in the main Debian archive, so your best bet is to get binary packages
91 from there. Check for the current version(s) using tracker.debian.org).
92
93 Source and backported versions are in the download area [1] alongside
94 the current ChangeLog. All the files for download are PGP-signed for
95 safety. You can find my keys online if you need them [2].
96
97 jigit is maintained in git [3].
98
99 [1] https://www.einval.com/~steve/software/JTE/download/
100 [2] https://www.einval.com/~steve/pgp/
101 [3] https://git.einval.com/cgi-bin/gitweb.cgi?p=jigit.git.
102
103 ----------------------------------------------------------------------
104
105 How to use JTE
106
107 To use the jigdo creation code, specify the location of the output .jigdo and
108 .template files alongside the ISO image. You can also specify the minimum size
109 beneath which files will just be dropped into the binary template file data
110 rather than listed as separate files to be found on the mirror, and exclude
111 patterns to ignore certain files in the same way. And paths in the original
112 filesystem can be mapped onto more global namespaces using the [Servers]
113 section in the .jigdo file. For example:
114
115 genisoimage -J -r -o /home/steve/test1.iso \
116 -jigdo-jigdo /home/steve/test1.jigdo \
117 -jigdo-template /home/steve/test1.template \
118 -jigdo-min-file-size 16384 \
119 -jigdo-ignore "README*" \
120 -jigdo-force-md5 "/pool/" \
121 -jigdo-map Debian=/mirror/debian \
122 -md5-list /home/steve/md5.list \
123 /mirror/jigdo-test
124
125 If the -jigdo-* options are not used, the normal genisoimage execution path is
126 not affected at all. The above invocation will create 3 output files (.iso,
127 .jigdo and .template). Multiple -jigdo-ignore and -jigdo-map options are
128 accepted, for multiple ignore and map patterns.
129
130 Use the -md5-list option to specify the location of a list of files and their
131 md5sums in normal md5sum format. genisoimage will then compare the checksum of
132 each file it is asked to write against the checksum of that file in the list.
133 It will abort on any mismatches. The MD5 list file must list all the files that
134 are expected to be found and listed in the output .jigdo file. The
135 -jigdo-force-md5 option specifies a path where all files are expected to have
136 an MD5 entry (e.g. /pool/). Then if any files do not have a match, they must
137 have been corrupted and genisoimage will abort.
138
139 ----------------------------------------------------------------------
140
141 How JTE works
142
143 I hooked all the places in genisoimage where it will normally write image data.
144 All the normal data write calls (directory entries etc.) I simply copy through
145 and build into the template file. Any file data entries are instead passed
146 through with information about the original file. If that file is large enough
147 (see -jigdo-min-file-size above), I grab the filename and the MD5 of the file's
148 data. If that MD5, size and length match an entry in the md5-list, I can just
149 write a file match record into the template file (and then the jigdo file)
150 instead of the file data itself.
151
152 ----------------------------------------------------------------------
153
154 How to use jigit-mkimage
155
156 jigit-mkimage is a faster, more minimal version of "jigdo-file make-image",
157 written in portable C. It takes a few options:
158
159 ┌─────────┬─────────────────────────────────────────────────────────────┐
160 │-f <MD5 │Specify a file containing MD5sums for files we should attempt to │
161 │file> │use when rebuilding the image │
162 ├─────────┼─────────────────────────────────────────────────────────────┤
163 │-j <jigdo │Specify the input jigdo file │
164 │file> │ │
165 ├─────────┼─────────────────────────────────────────────────────────────┤
166 │-t │ │
167 │<template │Specify the input template file │
168 │file> │ │
169 ├─────────┼─────────────────────────────────────────────────────────────┤
170 │-m <item= │Map <item> to <path> to find the files in the mirror │
171 │path> │ │
172 ├─────────┼─────────────────────────────────────────────────────────────┤
173 │-M │Don't attempt to build the image; just verify that all the │
174 │<Missing │components needed are available. If some are missing, list them in │
175 │file> │the specified file. │
176 ├─────────┼─────────────────────────────────────────────────────────────┤
177 │-v │Make the output logging more verbose. │
178 ├─────────┼─────────────────────────────────────────────────────────────┤
179 │-l <log │Specify a logfile. If not specified, will log to stderr just like │
180 │file> │genisoimage │
181 ├─────────┼─────────────────────────────────────────────────────────────┤
182 │ │Don't bother checking md5sums of the input files, or of the output │
183 │-q │image. │
184 │ │WARNING: this may lead to corrupt images, but is much faster. │
185 ├─────────┼─────────────────────────────────────────────────────────────┤
186 │-s <start │Specify where to start in the image (in bytes). If not specified, │
187 │offset> │will start at the beginning (offset 0). Added for iso-image.pl use │
188 ├─────────┼─────────────────────────────────────────────────────────────┤
189 │-e <end │Specify where to end in the image (in bytes). If not specified, │
190 │offset> │will run all the way to the end of the image. Added for │
191 │ │iso-image.pl use │
192 ├─────────┼─────────────────────────────────────────────────────────────┤
193 │ │Don't attempt to reassemble the image; simply parse the image │
194 │-z │descriptor in the template file and print the image size. Added for│
195 │ │iso-image.pl use │
196 └─────────┴─────────────────────────────────────────────────────────────┘
197
198 Specifying a start or end offset implies -q - it's difficult to check MD5 sums
199 if the full image is not generated!
200
201 ----------------------------------------------------------------------
202
203 (Dead) experiments
204 ------------------
205
206 I had extra plans for JTE that never really came to fruition due to a lack of
207 time and energy... :-/ Check git history if you're interested.
208
209 * iso-image.pl - on-the-fly rebuild of ISO images for HTTP
210
211 iso-image.pl was a small perl wrapper script written to drive mkimage and turn
212 it into a CGI. It would parse the incoming request (including byte-ranges) and
213 call jigit-mkimage to actually generate the image pieces wanted.
214
215 This code worked, but was always too slow for production use. Each CGI request
216 needed to index into the ISO image independently, leading to lots and lots of
217 overlapping calls to decompress the template data.
218
219 * jigdoofus - a better way to do on-the-fly assembly
220
221 I started on a new project, creating a FUSE-based filesystem that would rebuild
222 ISOs on the fly. I decided to use a database backend and a caching system to
223 solve the problem of the repetitive decompression that stopped iso-image.pl. I
224 made some progress, but ran out of steam. Code is still in the "jigdoofus"
225 branch in git in case anybody ever finds it useful.
226
227 * jigit - a friendly wrapper for jigit-mkimage
228
229 Similarly to the jigdo-lite script in the jigdo package, I wanted to provide a
230 nicer user experience for easy downloading of Debian and Ubuntu CD images. It
231 worked, but never really gained much traction. It needed much more effort to
232 make things reliable for production use.
233
234 ----------------------------------------------------------------------
235
236 External integration
237 --------------------
238
239 * debian-cd
240
241 The debian-cd package in Debian is what we use to generate installer CDs and
242 DVDs. It has supported JTE since 2005, and we still use it every day.
243
244 * cdrkit/genisoimage
245
246 genisoimage in Debian shipped with integrated JTE code for a long time, but is
247 basically dead upstream. Not recommended for use any more.
248
249 * xorriso
250
251 xorriso uses libjte to generate jigdo and template files, and has worked this
252 way since 2010.
253
254 ----------------------------------------------------------------------
255
256 What's left to do?
257 ------------------
258
259 1. Testing! :-) This is where you lot come in! Please play with this some more
260 and let me know if you have any problems, especially with data corruption.
261 2. More documentation.
262
263
264
265
266 ----------------------------------------------------------------------
267
268 How to use JTE
269
270 To use the jigdo creation code, specify the location of the output .jigdo and
271 .template files alongside the ISO image. You can also specify the minimum size
272 beneath which files will just be dropped into the binary template file data
273 rather than listed as separate files to be found on the mirror, and exclude
274 patterns to ignore certain files in the same way. And paths in the original
275 filesystem can be mapped onto more global namespaces using the [Servers]
276 section in the .jigdo file. For example:
277
278 genisoimage -J -r -o /home/steve/test1.iso \
279 -jigdo-jigdo /home/steve/test1.jigdo \
280 -jigdo-template /home/steve/test1.template \
281 -jigdo-min-file-size 16384 \
282 -jigdo-ignore "README*" \
283 -jigdo-force-md5 "/pool/" \
284 -jigdo-map Debian=/mirror/debian \
285 -md5-list /home/steve/md5.list \
286 /mirror/jigdo-test
287
288 If the -jigdo-* options are not used, the normal genisoimage execution path is
289 not affected at all. The above invocation will create 3 output files (.iso,
290 .jigdo and .template). Multiple -jigdo-ignore and -jigdo-map options are
291 accepted, for multiple ignore and map patterns.
292
293 Use the -md5-list option to specify the location of a list of files and their
294 md5sums in normal md5sum format. genisoimage will then compare the checksum of
295 each file it is asked to write against the checksum of that file in the list.
296 It will abort on any mismatches. The MD5 list file must list all the files that
297 are expected to be found and listed in the output .jigdo file. The
298 -jigdo-force-md5 option specifies a path where all files are expected to have
299 an MD5 entry (e.g. /pool/). Then if any files do not have a match, they must
300 have been corrupted and genisoimage will abort.
301
302 ----------------------------------------------------------------------
303
304 How JTE works
305
306 I hooked all the places in genisoimage where it will normally write image data.
307 All the normal data write calls (directory entries etc.) I simply copy through
308 and build into the template file. Any file data entries are instead passed
309 through with information about the original file. If that file is large enough
310 (see -jigdo-min-file-size above), I grab the filename and the MD5 of the file's
311 data. If that MD5, size and length match an entry in the md5-list, I can just
312 write a file match record into the template file (and then the jigdo file)
313 instead of the file data itself.
314
315 ----------------------------------------------------------------------
316
317 How to use jigit-mkimage
318
319 jigit-mkimage is a faster, more minimal version of "jigdo-file make-image",
320 written in portable C. It takes a few options:
321
322 ┌─────────┬───────────────────────────────────────────────────────────────────┐
323 │-f <MD5 │Specify a file containing MD5sums for files we should attempt to │
324 │file> │use when rebuilding the image │
325 ├─────────┼───────────────────────────────────────────────────────────────────┤
326 │-j <jigdo│Specify the input jigdo file │
327 │file> │ │
328 ├─────────┼───────────────────────────────────────────────────────────────────┤
329 │-t │ │
330 │<template│Specify the input template file │
331 │file> │ │
332 ├─────────┼───────────────────────────────────────────────────────────────────┤
333 │-m <item=│Map <item> to <path> to find the files in the mirror │
334 │path> │ │
335 ├─────────┼───────────────────────────────────────────────────────────────────┤
336 │-M │Don't attempt to build the image; just verify that all the │
337 │<Missing │components needed are available. If some are missing, list them in │
338 │file> │the specified file. │
339 ├─────────┼───────────────────────────────────────────────────────────────────┤
340 │-v │Make the output logging more verbose. │
341 ├─────────┼───────────────────────────────────────────────────────────────────┤
342 │-l <log │Specify a logfile. If not specified, will log to stderr just like │
343 │file> │genisoimage │
344 ├─────────┼───────────────────────────────────────────────────────────────────┤
345 │ │Don't bother checking md5sums of the input files, or of the output │
346 │-q │image. │
347 │ │WARNING: this may lead to corrupt images, but is much faster. │
348 ├─────────┼───────────────────────────────────────────────────────────────────┤
349 │-s <start│Specify where to start in the image (in bytes). If not specified, │
350 │offset> │will start at the beginning (offset 0). Added for iso-image.pl use │
351 ├─────────┼───────────────────────────────────────────────────────────────────┤
352 │-e <end │Specify where to end in the image (in bytes). If not specified, │
353 │offset> │will run all the way to the end of the image. Added for │
354 │ │iso-image.pl use │
355 ├─────────┼───────────────────────────────────────────────────────────────────┤
356 │ │Don't attempt to reassemble the image; simply parse the image │
357 │-z │descriptor in the template file and print the image size. Added for│
358 │ │iso-image.pl use │
359 └─────────┴───────────────────────────────────────────────────────────────────┘
360
361 Specifying a start or end offset implies -q - it's difficult to check MD5 sums
362 if the full image is not generated!
363
364 ----------------------------------------------------------------------
365
366 (Dead) experiments
367
368 I had extra plans for JTE that never really came to fruition due to a lack of
369 time and energy... :-/ Check git history if you're interested.
370
371 iso-image.pl - on-the-fly rebuild of ISO images for HTTP
372
373 iso-image.pl was a small perl wrapper script written to drive mkimage and turn
374 it into a CGI. It would parse the incoming request (including byte-ranges) and
375 call jigit-mkimage to actually generate the image pieces wanted.
376
377 This code worked, but was always too slow for production use. Each CGI request
378 needed to index into the ISO image independently, leading to lots and lots of
379 overlapping calls to decompress the template data.
380
381 jigdoofus - a better way to do on-the-fly assembly
382
383 I started on a new project, creating a FUSE-based filesystem that would rebuild
384 ISOs on the fly. I decided to use a database backend and a caching system to
385 solve the problem of the repetitive decompression that stopped iso-image.pl. I
386 made some progress, but ran out of steam. Code is still in the "jigdoofus"
387 branch in git in case anybody ever finds it useful.
388
389 jigit - a friendly wrapper for jigit-mkimage
390
391 Similarly to the jigdo-lite script in the jigdo package, I wanted to provide a
392 nicer user experience for easy downloading of Debian and Ubuntu CD images. It
393 worked, but never really gained much traction. It needed much more effort to
394 make things reliable for production use.
395
396 ----------------------------------------------------------------------
397
398 External integration
399
400 debian-cd
401
402 The debian-cd package in Debian is what we use to generate installer CDs and
403 DVDs. It has supported JTE since 2005, and we still use it every day.
404
405 cdrkit/genisoimage
406
407 genisoimage in Debian shipped with integrated JTE code for a long time, but is
408 basically dead upstream. Not recommended for use any more.
409
410 xorriso
411
412 xorriso uses libjte to generate jigdo and template files, and has worked this
413 way since 2010.
414
415 ----------------------------------------------------------------------
416
417 What's left to do?
418
419 1. Testing! :-) This is where you lot come in! Please play with this some more
420 and let me know if you have any problems, especially with data corruption.
421 2. More documentation.
422