| 1 |
README for JTE 1.22 |
| 2 |
|
| 3 |
Steve McIntyre <[email protected]> |
| 4 |
20 November 2019 |
| 5 |
|
| 6 |
License - GPL v2+. See the file COPYING for more details. |
| 7 |
|
| 8 |
JTE - Jigdo Template Export |
| 9 |
=========================== |
| 10 |
|
| 11 |
Introduction - jigdo and JTE |
| 12 |
---------------------------- |
| 13 |
|
| 14 |
Jigdo is a useful tool to help in the distribution of large files like CD and |
| 15 |
DVD images. See Richard Atterer's site for more details. Debian CDs and DVD ISO |
| 16 |
images are published on the web in jigdo format to allow end users to download |
| 17 |
them more efficiently. |
| 18 |
|
| 19 |
Jigdo is generic and powerful - it can be used for any large files that are |
| 20 |
made up of smaller files. However, to be this generic is costly. Creating jigdo |
| 21 |
files from ISO images is quite inefficient - to work out which files are |
| 22 |
included in the ISO image, jigdo has to calculate and compare checksums of |
| 23 |
every possible file and every extent in the image. Essentially it has to |
| 24 |
brute-force the image. It can take a long time to do this for a large image |
| 25 |
(imagine a 4.5GB DVD image or a 30+GB Blu-Ray image). |
| 26 |
|
| 27 |
I first started looking for ways to improve this back in 2004: |
| 28 |
|
| 29 |
1. Modify jigdo so it knew about the internals of ISO images and could |
| 30 |
efficiently scan them (bad, not very generic for jigdo) |
| 31 |
2. Write a helper tool to dump extra information for jigdo to use alongside |
| 32 |
the ISO image (I had a helper tool written, but modifying jigdo to use this |
| 33 |
looked HARD) |
| 34 |
3. Patch mkisofs/genisoimage to write .jigdo and .template files alongside the |
| 35 |
ISO image |
| 36 |
|
| 37 |
I completed the third of these options, and called it JTE (or Jigdo Template |
| 38 |
Export). The code worked fine, and ran in a very small fraction of the time |
| 39 |
taken to run genisoimage and jigdo separately. The output .jigdo and .template |
| 40 |
files worked correctly, i.e. jigdo-file and the wrapper script jigdo-mirror |
| 41 |
accept them and would generate an ISO image that exactly matches the original. |
| 42 |
|
| 43 |
Debian used that code for a number of years within genisoimage, but we've since |
| 44 |
switched over to using xorriso instead for our image building instead. It has a |
| 45 |
lot of useful features that we want compared to genisoimage, not least a |
| 46 |
friendly and engaged author in Thomas Scmitt! |
| 47 |
|
| 48 |
Thomas and I and George Danchev worked together to package up my JTE code into |
| 49 |
libjte such that xorriso could use it effectively. Xorriso has been capable of |
| 50 |
generating jigdo files since 2010. |
| 51 |
|
| 52 |
In late 2019, I took over maintenance of the jigdo upstream code and added |
| 53 |
support for a new (v2) jigdo data format, using SHA256 instead of MD5 |
| 54 |
internally. See my jigdo page for more details about that. I have also updated |
| 55 |
the JTE codebase to support this new format, of course. |
| 56 |
|
| 57 |
As genisoimage is effectively dead at this point, I took the decision to not |
| 58 |
add the jigdo v2 support into the genisoimage codebase. If you need to generate |
| 59 |
jigdo v2 format, either use jigdo itself or xorriso if you'd like the |
| 60 |
performance benefit of the libjte integration. |
| 61 |
|
| 62 |
JTE includes a few tools: |
| 63 |
|
| 64 |
• jigit-mkimage, a simple and very fast tool to reconstruct image files from |
| 65 |
.jigdo and .template files. It doesn't have any logic to cope with |
| 66 |
downloading missing files, but will list the missing files that are needed. |
| 67 |
It is also much faster for people (like me!) who already have full local |
| 68 |
mirrors. |
| 69 |
• parallel-sums is a simple extra utility to generate checksums quickly and |
| 70 |
efficiently, reading file data only once and calculating checksums using |
| 71 |
multiple algorithms in parallel using threads. |
| 72 |
• jigsum, jigsum-sha256 and rsyncsum are checksum tools which will output |
| 73 |
checksums in jigdo's base64-like format rather than the normal hexadecimal |
| 74 |
format. Useful for debugging jigdo issues. |
| 75 |
• jigdump is a tool to dump the contents of a jigdo template or .iso.tmp |
| 76 |
file. Useful for debugging jigdo issues. |
| 77 |
• mkjigsnap is a utility to help with maintaining the "snapshots" that jigdo |
| 78 |
needs if you're going to be keeping data around for users in the long term. |
| 79 |
We use this on some Debian systems. |
| 80 |
|
| 81 |
Why the "jigit" name? The packages and source are named jigit to match the name |
| 82 |
of a long-dead wrapper script. |
| 83 |
|
| 84 |
---------------------------------------------------------------------- |
| 85 |
|
| 86 |
Download |
| 87 |
-------- |
| 88 |
|
| 89 |
The jigit source package (and hence the various binary packages it builds) is |
| 90 |
included in the main Debian archive, so your best bet is to get binary packages |
| 91 |
from there. Check for the current version(s) using tracker.debian.org). |
| 92 |
|
| 93 |
Source and backported versions are in the download area [1] alongside |
| 94 |
the current ChangeLog. All the files for download are PGP-signed for |
| 95 |
safety. You can find my keys online if you need them [2]. |
| 96 |
|
| 97 |
jigit is maintained in git [3]. |
| 98 |
|
| 99 |
[1] https://www.einval.com/~steve/software/JTE/download/ |
| 100 |
[2] https://www.einval.com/~steve/pgp/ |
| 101 |
[3] https://git.einval.com/cgi-bin/gitweb.cgi?p=jigit.git. |
| 102 |
|
| 103 |
---------------------------------------------------------------------- |
| 104 |
|
| 105 |
How to use JTE |
| 106 |
|
| 107 |
To use the jigdo creation code, specify the location of the output .jigdo and |
| 108 |
.template files alongside the ISO image. You can also specify the minimum size |
| 109 |
beneath which files will just be dropped into the binary template file data |
| 110 |
rather than listed as separate files to be found on the mirror, and exclude |
| 111 |
patterns to ignore certain files in the same way. And paths in the original |
| 112 |
filesystem can be mapped onto more global namespaces using the [Servers] |
| 113 |
section in the .jigdo file. For example: |
| 114 |
|
| 115 |
genisoimage -J -r -o /home/steve/test1.iso \ |
| 116 |
-jigdo-jigdo /home/steve/test1.jigdo \ |
| 117 |
-jigdo-template /home/steve/test1.template \ |
| 118 |
-jigdo-min-file-size 16384 \ |
| 119 |
-jigdo-ignore "README*" \ |
| 120 |
-jigdo-force-md5 "/pool/" \ |
| 121 |
-jigdo-map Debian=/mirror/debian \ |
| 122 |
-md5-list /home/steve/md5.list \ |
| 123 |
/mirror/jigdo-test |
| 124 |
|
| 125 |
If the -jigdo-* options are not used, the normal genisoimage execution path is |
| 126 |
not affected at all. The above invocation will create 3 output files (.iso, |
| 127 |
.jigdo and .template). Multiple -jigdo-ignore and -jigdo-map options are |
| 128 |
accepted, for multiple ignore and map patterns. |
| 129 |
|
| 130 |
Use the -md5-list option to specify the location of a list of files and their |
| 131 |
md5sums in normal md5sum format. genisoimage will then compare the checksum of |
| 132 |
each file it is asked to write against the checksum of that file in the list. |
| 133 |
It will abort on any mismatches. The MD5 list file must list all the files that |
| 134 |
are expected to be found and listed in the output .jigdo file. The |
| 135 |
-jigdo-force-md5 option specifies a path where all files are expected to have |
| 136 |
an MD5 entry (e.g. /pool/). Then if any files do not have a match, they must |
| 137 |
have been corrupted and genisoimage will abort. |
| 138 |
|
| 139 |
---------------------------------------------------------------------- |
| 140 |
|
| 141 |
How JTE works |
| 142 |
|
| 143 |
I hooked all the places in genisoimage where it will normally write image data. |
| 144 |
All the normal data write calls (directory entries etc.) I simply copy through |
| 145 |
and build into the template file. Any file data entries are instead passed |
| 146 |
through with information about the original file. If that file is large enough |
| 147 |
(see -jigdo-min-file-size above), I grab the filename and the MD5 of the file's |
| 148 |
data. If that MD5, size and length match an entry in the md5-list, I can just |
| 149 |
write a file match record into the template file (and then the jigdo file) |
| 150 |
instead of the file data itself. |
| 151 |
|
| 152 |
---------------------------------------------------------------------- |
| 153 |
|
| 154 |
How to use jigit-mkimage |
| 155 |
|
| 156 |
jigit-mkimage is a faster, more minimal version of "jigdo-file make-image", |
| 157 |
written in portable C. It takes a few options: |
| 158 |
|
| 159 |
┌─────────┬─────────────────────────────────────────────────────────────┐ |
| 160 |
│-f <MD5 │Specify a file containing MD5sums for files we should attempt to │ |
| 161 |
│file> │use when rebuilding the image │ |
| 162 |
├─────────┼─────────────────────────────────────────────────────────────┤ |
| 163 |
│-j <jigdo │Specify the input jigdo file │ |
| 164 |
│file> │ │ |
| 165 |
├─────────┼─────────────────────────────────────────────────────────────┤ |
| 166 |
│-t │ │ |
| 167 |
│<template │Specify the input template file │ |
| 168 |
│file> │ │ |
| 169 |
├─────────┼─────────────────────────────────────────────────────────────┤ |
| 170 |
│-m <item= │Map <item> to <path> to find the files in the mirror │ |
| 171 |
│path> │ │ |
| 172 |
├─────────┼─────────────────────────────────────────────────────────────┤ |
| 173 |
│-M │Don't attempt to build the image; just verify that all the │ |
| 174 |
│<Missing │components needed are available. If some are missing, list them in │ |
| 175 |
│file> │the specified file. │ |
| 176 |
├─────────┼─────────────────────────────────────────────────────────────┤ |
| 177 |
│-v │Make the output logging more verbose. │ |
| 178 |
├─────────┼─────────────────────────────────────────────────────────────┤ |
| 179 |
│-l <log │Specify a logfile. If not specified, will log to stderr just like │ |
| 180 |
│file> │genisoimage │ |
| 181 |
├─────────┼─────────────────────────────────────────────────────────────┤ |
| 182 |
│ │Don't bother checking md5sums of the input files, or of the output │ |
| 183 |
│-q │image. │ |
| 184 |
│ │WARNING: this may lead to corrupt images, but is much faster. │ |
| 185 |
├─────────┼─────────────────────────────────────────────────────────────┤ |
| 186 |
│-s <start │Specify where to start in the image (in bytes). If not specified, │ |
| 187 |
│offset> │will start at the beginning (offset 0). Added for iso-image.pl use │ |
| 188 |
├─────────┼─────────────────────────────────────────────────────────────┤ |
| 189 |
│-e <end │Specify where to end in the image (in bytes). If not specified, │ |
| 190 |
│offset> │will run all the way to the end of the image. Added for │ |
| 191 |
│ │iso-image.pl use │ |
| 192 |
├─────────┼─────────────────────────────────────────────────────────────┤ |
| 193 |
│ │Don't attempt to reassemble the image; simply parse the image │ |
| 194 |
│-z │descriptor in the template file and print the image size. Added for│ |
| 195 |
│ │iso-image.pl use │ |
| 196 |
└─────────┴─────────────────────────────────────────────────────────────┘ |
| 197 |
|
| 198 |
Specifying a start or end offset implies -q - it's difficult to check MD5 sums |
| 199 |
if the full image is not generated! |
| 200 |
|
| 201 |
---------------------------------------------------------------------- |
| 202 |
|
| 203 |
(Dead) experiments |
| 204 |
------------------ |
| 205 |
|
| 206 |
I had extra plans for JTE that never really came to fruition due to a lack of |
| 207 |
time and energy... :-/ Check git history if you're interested. |
| 208 |
|
| 209 |
* iso-image.pl - on-the-fly rebuild of ISO images for HTTP |
| 210 |
|
| 211 |
iso-image.pl was a small perl wrapper script written to drive mkimage and turn |
| 212 |
it into a CGI. It would parse the incoming request (including byte-ranges) and |
| 213 |
call jigit-mkimage to actually generate the image pieces wanted. |
| 214 |
|
| 215 |
This code worked, but was always too slow for production use. Each CGI request |
| 216 |
needed to index into the ISO image independently, leading to lots and lots of |
| 217 |
overlapping calls to decompress the template data. |
| 218 |
|
| 219 |
* jigdoofus - a better way to do on-the-fly assembly |
| 220 |
|
| 221 |
I started on a new project, creating a FUSE-based filesystem that would rebuild |
| 222 |
ISOs on the fly. I decided to use a database backend and a caching system to |
| 223 |
solve the problem of the repetitive decompression that stopped iso-image.pl. I |
| 224 |
made some progress, but ran out of steam. Code is still in the "jigdoofus" |
| 225 |
branch in git in case anybody ever finds it useful. |
| 226 |
|
| 227 |
* jigit - a friendly wrapper for jigit-mkimage |
| 228 |
|
| 229 |
Similarly to the jigdo-lite script in the jigdo package, I wanted to provide a |
| 230 |
nicer user experience for easy downloading of Debian and Ubuntu CD images. It |
| 231 |
worked, but never really gained much traction. It needed much more effort to |
| 232 |
make things reliable for production use. |
| 233 |
|
| 234 |
---------------------------------------------------------------------- |
| 235 |
|
| 236 |
External integration |
| 237 |
-------------------- |
| 238 |
|
| 239 |
* debian-cd |
| 240 |
|
| 241 |
The debian-cd package in Debian is what we use to generate installer CDs and |
| 242 |
DVDs. It has supported JTE since 2005, and we still use it every day. |
| 243 |
|
| 244 |
* cdrkit/genisoimage |
| 245 |
|
| 246 |
genisoimage in Debian shipped with integrated JTE code for a long time, but is |
| 247 |
basically dead upstream. Not recommended for use any more. |
| 248 |
|
| 249 |
* xorriso |
| 250 |
|
| 251 |
xorriso uses libjte to generate jigdo and template files, and has worked this |
| 252 |
way since 2010. |
| 253 |
|
| 254 |
---------------------------------------------------------------------- |
| 255 |
|
| 256 |
What's left to do? |
| 257 |
------------------ |
| 258 |
|
| 259 |
1. Testing! :-) This is where you lot come in! Please play with this some more |
| 260 |
and let me know if you have any problems, especially with data corruption. |
| 261 |
2. More documentation. |
| 262 |
|
| 263 |
|
| 264 |
|
| 265 |
|
| 266 |
---------------------------------------------------------------------- |
| 267 |
|
| 268 |
How to use JTE |
| 269 |
|
| 270 |
To use the jigdo creation code, specify the location of the output .jigdo and |
| 271 |
.template files alongside the ISO image. You can also specify the minimum size |
| 272 |
beneath which files will just be dropped into the binary template file data |
| 273 |
rather than listed as separate files to be found on the mirror, and exclude |
| 274 |
patterns to ignore certain files in the same way. And paths in the original |
| 275 |
filesystem can be mapped onto more global namespaces using the [Servers] |
| 276 |
section in the .jigdo file. For example: |
| 277 |
|
| 278 |
genisoimage -J -r -o /home/steve/test1.iso \ |
| 279 |
-jigdo-jigdo /home/steve/test1.jigdo \ |
| 280 |
-jigdo-template /home/steve/test1.template \ |
| 281 |
-jigdo-min-file-size 16384 \ |
| 282 |
-jigdo-ignore "README*" \ |
| 283 |
-jigdo-force-md5 "/pool/" \ |
| 284 |
-jigdo-map Debian=/mirror/debian \ |
| 285 |
-md5-list /home/steve/md5.list \ |
| 286 |
/mirror/jigdo-test |
| 287 |
|
| 288 |
If the -jigdo-* options are not used, the normal genisoimage execution path is |
| 289 |
not affected at all. The above invocation will create 3 output files (.iso, |
| 290 |
.jigdo and .template). Multiple -jigdo-ignore and -jigdo-map options are |
| 291 |
accepted, for multiple ignore and map patterns. |
| 292 |
|
| 293 |
Use the -md5-list option to specify the location of a list of files and their |
| 294 |
md5sums in normal md5sum format. genisoimage will then compare the checksum of |
| 295 |
each file it is asked to write against the checksum of that file in the list. |
| 296 |
It will abort on any mismatches. The MD5 list file must list all the files that |
| 297 |
are expected to be found and listed in the output .jigdo file. The |
| 298 |
-jigdo-force-md5 option specifies a path where all files are expected to have |
| 299 |
an MD5 entry (e.g. /pool/). Then if any files do not have a match, they must |
| 300 |
have been corrupted and genisoimage will abort. |
| 301 |
|
| 302 |
---------------------------------------------------------------------- |
| 303 |
|
| 304 |
How JTE works |
| 305 |
|
| 306 |
I hooked all the places in genisoimage where it will normally write image data. |
| 307 |
All the normal data write calls (directory entries etc.) I simply copy through |
| 308 |
and build into the template file. Any file data entries are instead passed |
| 309 |
through with information about the original file. If that file is large enough |
| 310 |
(see -jigdo-min-file-size above), I grab the filename and the MD5 of the file's |
| 311 |
data. If that MD5, size and length match an entry in the md5-list, I can just |
| 312 |
write a file match record into the template file (and then the jigdo file) |
| 313 |
instead of the file data itself. |
| 314 |
|
| 315 |
---------------------------------------------------------------------- |
| 316 |
|
| 317 |
How to use jigit-mkimage |
| 318 |
|
| 319 |
jigit-mkimage is a faster, more minimal version of "jigdo-file make-image", |
| 320 |
written in portable C. It takes a few options: |
| 321 |
|
| 322 |
┌─────────┬───────────────────────────────────────────────────────────────────┐ |
| 323 |
│-f <MD5 │Specify a file containing MD5sums for files we should attempt to │ |
| 324 |
│file> │use when rebuilding the image │ |
| 325 |
├─────────┼───────────────────────────────────────────────────────────────────┤ |
| 326 |
│-j <jigdo│Specify the input jigdo file │ |
| 327 |
│file> │ │ |
| 328 |
├─────────┼───────────────────────────────────────────────────────────────────┤ |
| 329 |
│-t │ │ |
| 330 |
│<template│Specify the input template file │ |
| 331 |
│file> │ │ |
| 332 |
├─────────┼───────────────────────────────────────────────────────────────────┤ |
| 333 |
│-m <item=│Map <item> to <path> to find the files in the mirror │ |
| 334 |
│path> │ │ |
| 335 |
├─────────┼───────────────────────────────────────────────────────────────────┤ |
| 336 |
│-M │Don't attempt to build the image; just verify that all the │ |
| 337 |
│<Missing │components needed are available. If some are missing, list them in │ |
| 338 |
│file> │the specified file. │ |
| 339 |
├─────────┼───────────────────────────────────────────────────────────────────┤ |
| 340 |
│-v │Make the output logging more verbose. │ |
| 341 |
├─────────┼───────────────────────────────────────────────────────────────────┤ |
| 342 |
│-l <log │Specify a logfile. If not specified, will log to stderr just like │ |
| 343 |
│file> │genisoimage │ |
| 344 |
├─────────┼───────────────────────────────────────────────────────────────────┤ |
| 345 |
│ │Don't bother checking md5sums of the input files, or of the output │ |
| 346 |
│-q │image. │ |
| 347 |
│ │WARNING: this may lead to corrupt images, but is much faster. │ |
| 348 |
├─────────┼───────────────────────────────────────────────────────────────────┤ |
| 349 |
│-s <start│Specify where to start in the image (in bytes). If not specified, │ |
| 350 |
│offset> │will start at the beginning (offset 0). Added for iso-image.pl use │ |
| 351 |
├─────────┼───────────────────────────────────────────────────────────────────┤ |
| 352 |
│-e <end │Specify where to end in the image (in bytes). If not specified, │ |
| 353 |
│offset> │will run all the way to the end of the image. Added for │ |
| 354 |
│ │iso-image.pl use │ |
| 355 |
├─────────┼───────────────────────────────────────────────────────────────────┤ |
| 356 |
│ │Don't attempt to reassemble the image; simply parse the image │ |
| 357 |
│-z │descriptor in the template file and print the image size. Added for│ |
| 358 |
│ │iso-image.pl use │ |
| 359 |
└─────────┴───────────────────────────────────────────────────────────────────┘ |
| 360 |
|
| 361 |
Specifying a start or end offset implies -q - it's difficult to check MD5 sums |
| 362 |
if the full image is not generated! |
| 363 |
|
| 364 |
---------------------------------------------------------------------- |
| 365 |
|
| 366 |
(Dead) experiments |
| 367 |
|
| 368 |
I had extra plans for JTE that never really came to fruition due to a lack of |
| 369 |
time and energy... :-/ Check git history if you're interested. |
| 370 |
|
| 371 |
iso-image.pl - on-the-fly rebuild of ISO images for HTTP |
| 372 |
|
| 373 |
iso-image.pl was a small perl wrapper script written to drive mkimage and turn |
| 374 |
it into a CGI. It would parse the incoming request (including byte-ranges) and |
| 375 |
call jigit-mkimage to actually generate the image pieces wanted. |
| 376 |
|
| 377 |
This code worked, but was always too slow for production use. Each CGI request |
| 378 |
needed to index into the ISO image independently, leading to lots and lots of |
| 379 |
overlapping calls to decompress the template data. |
| 380 |
|
| 381 |
jigdoofus - a better way to do on-the-fly assembly |
| 382 |
|
| 383 |
I started on a new project, creating a FUSE-based filesystem that would rebuild |
| 384 |
ISOs on the fly. I decided to use a database backend and a caching system to |
| 385 |
solve the problem of the repetitive decompression that stopped iso-image.pl. I |
| 386 |
made some progress, but ran out of steam. Code is still in the "jigdoofus" |
| 387 |
branch in git in case anybody ever finds it useful. |
| 388 |
|
| 389 |
jigit - a friendly wrapper for jigit-mkimage |
| 390 |
|
| 391 |
Similarly to the jigdo-lite script in the jigdo package, I wanted to provide a |
| 392 |
nicer user experience for easy downloading of Debian and Ubuntu CD images. It |
| 393 |
worked, but never really gained much traction. It needed much more effort to |
| 394 |
make things reliable for production use. |
| 395 |
|
| 396 |
---------------------------------------------------------------------- |
| 397 |
|
| 398 |
External integration |
| 399 |
|
| 400 |
debian-cd |
| 401 |
|
| 402 |
The debian-cd package in Debian is what we use to generate installer CDs and |
| 403 |
DVDs. It has supported JTE since 2005, and we still use it every day. |
| 404 |
|
| 405 |
cdrkit/genisoimage |
| 406 |
|
| 407 |
genisoimage in Debian shipped with integrated JTE code for a long time, but is |
| 408 |
basically dead upstream. Not recommended for use any more. |
| 409 |
|
| 410 |
xorriso |
| 411 |
|
| 412 |
xorriso uses libjte to generate jigdo and template files, and has worked this |
| 413 |
way since 2010. |
| 414 |
|
| 415 |
---------------------------------------------------------------------- |
| 416 |
|
| 417 |
What's left to do? |
| 418 |
|
| 419 |
1. Testing! :-) This is where you lot come in! Please play with this some more |
| 420 |
and let me know if you have any problems, especially with data corruption. |
| 421 |
2. More documentation. |
| 422 |
|