Debian Codegraph @

doc/html/x67.html

raw
1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN""http://www.w3.org/TR/html4/loose.dtd">
2 <HTML
3 ><HEAD
4 ><TITLE
5 >new debdelta-upgrade service</TITLE
6 ><META
7 NAME="GENERATOR"
8 CONTENT="Modular DocBook HTML Stylesheet Version 1.79"><LINK
9 REL="HOME"
10 HREF="index.html"><LINK
11 REL="PREVIOUS"
12 TITLE="a delta"
13 HREF="x56.html"><LINK
14 REL="NEXT"
15 TITLE="Goals, tricks, ideas and issues"
16 HREF="x190.html"></HEAD
17 ><BODY
18 CLASS="section"
19 BGCOLOR="#FFFFFF"
20 TEXT="#000000"
21 LINK="#0000FF"
22 VLINK="#840084"
23 ALINK="#0000FF"
24 ><DIV
25 CLASS="NAVHEADER"
26 ><TABLE
27 SUMMARY="Header navigation table"
28 WIDTH="100%"
29 BORDER="0"
30 CELLPADDING="0"
31 CELLSPACING="0"
32 ><TR
33 ><TH
34 COLSPAN="3"
35 ALIGN="center"
36 ></TH
37 ></TR
38 ><TR
39 ><TD
40 WIDTH="10%"
41 ALIGN="left"
42 VALIGN="bottom"
43 ><A
44 HREF="x56.html"
45 ACCESSKEY="P"
46 >Prev</A
47 ></TD
48 ><TD
49 WIDTH="80%"
50 ALIGN="center"
51 VALIGN="bottom"
52 ></TD
53 ><TD
54 WIDTH="10%"
55 ALIGN="right"
56 VALIGN="bottom"
57 ><A
58 HREF="x190.html"
59 ACCESSKEY="N"
60 >Next</A
61 ></TD
62 ></TR
63 ></TABLE
64 ><HR
65 ALIGN="LEFT"
66 WIDTH="100%"></DIV
67 ><DIV
68 CLASS="section"
69 ><H1
70 CLASS="section"
71 ><A
72 NAME="AEN67"
73 >3. new debdelta-upgrade service</A
74 ></H1
75 ><P
76 >&#13; Using the delta-upgrading framework, people may upgrade their
77 Debian box using <B
78 CLASS="command"
79 >debdelta-upgrade</B
80 > (that
81 downloads package 'deltas').
82
83 In 2007 I had set up a server in my home, that was active up to 2017.
84 Since 2017, the generation of deltas is performed inside a dedicated host inside the Debian infrastructure.
85
86 This section is an introduction to the framework that is behind
87 'debdelta-upgrade', and is also used by 'cupt'.
88 In the following, I will simplify (in places, quite a lot).
89 </P
90 ><DIV
91 CLASS="section"
92 ><H2
93 CLASS="section"
94 ><A
95 NAME="AEN71"
96 >3.1. The framework</A
97 ></H2
98 ><P
99 >&#13; The framework is so organized: the
100 program 'debdeltas_server' creates all the deltas; whereas endusers use the
101 client 'debdelta-upgrade' to download the deltas and apply them to
102 produce the debs needed to upgrade their boxes.
103 The server generates any delta that may be needed for upgrades
104 in all main releases (currently, in 2024: <SPAN
105 CLASS="emphasis"
106 ><I
107 CLASS="emphasis"
108 >bookworm, bookworm-backports, bookworm-proposed-updates, bookworm-updates,
109 bullseye, bullseye-backports, bullseye-proposed-updates, bullseye-updates, buster, buster-backports,
110 buster-proposed-updates, buster-updates</I
111 ></SPAN
112 >) as well as the security channel (<SPAN
113 CLASS="emphasis"
114 ><I
115 CLASS="emphasis"
116 >bookworm-security, bullseye-security,
117 buster, trixie-security</I
118 ></SPAN
119 >); and also deltas to upgrade from any main release to its
120 <SPAN
121 CLASS="emphasis"
122 ><I
123 CLASS="emphasis"
124 >-backports</I
125 ></SPAN
126 > or <SPAN
127 CLASS="emphasis"
128 ><I
129 CLASS="emphasis"
130 >-security</I
131 ></SPAN
132 > channel.
133 This is performed for architectures <SPAN
134 CLASS="emphasis"
135 ><I
136 CLASS="emphasis"
137 >all, i386 and amd64</I
138 ></SPAN
139 >.
140 </P
141 ></DIV
142 ><DIV
143 CLASS="section"
144 ><H2
145 CLASS="section"
146 ><A
147 NAME="AEN79"
148 >3.2. The goals</A
149 ></H2
150 ><P
151 >There are two ultimate goals in designing this framework:
152 <P
153 ></P
154 ><OL
155 TYPE="1"
156 ><LI
157 ><P
158 >&#13; SMALL) reduce the size of downloads
159 (fit for people that pay-by-megabyte);
160 </P
161 ></LI
162 ><LI
163 ><P
164 >&#13; FAST) speed up the upgrade.
165 </P
166 ></LI
167 ></OL
168 >
169
170 The two goals are unfortunately only marginally compatible. An
171 example: bsdiff can produce very small deltas, but is quite slow (in
172 particular with very large files); so currently (2009 on) I use 'xdelta3'
173 as the backend diffing tool for 'debdeltas' in my server.
174
175 Another example is in debs that contain archives ( .gz, , tar.gz
176 etc etc): I have methods and code to peek inside them, so
177 the delta become smaller, but the applying gets slower.
178 </P
179 ></DIV
180 ><DIV
181 CLASS="section"
182 ><H2
183 CLASS="section"
184 ><A
185 NAME="AEN87"
186 >3.3. The repository structure</A
187 ></H2
188 ><P
189 >&#13; The repository of deltas is just a HTTP archive; it is similar to the pool of packages; that is, if
190 <TT
191 CLASS="filename"
192 >foobar_1_all.deb</TT
193 > is stored in
194 <TT
195 CLASS="filename"
196 >pool/main/f/foobar/</TT
197 > in the repository of debs, then the
198 delta to upgrade it will be stored in <TT
199 CLASS="filename"
200 >pool/main/f/foobar/foobar_1_2_all.debdelta</TT
201 >
202 in the repository of deltas. Contrary to the repository of debs, a repository of deltas
203 has no indexes, see <A
204 HREF="x67.html#no_indexes"
205 >Section 3.7.2</A
206 >. The delta repository is in
207 <TT
208 CLASS="filename"
209 >http://debdeltas.debian.net/debian-deltas</TT
210 >.
211 </P
212 ></DIV
213 ><DIV
214 CLASS="section"
215 ><H2
216 CLASS="section"
217 ><A
218 NAME="delta_creation"
219 >3.4. The repository creation</A
220 ></H2
221 ><P
222 >&#13; Suppose that the unstable archive, on 1st Mar, contains
223 <TT
224 CLASS="filename"
225 >foobar_1_all.deb</TT
226 > (and it is in
227 <TT
228 CLASS="filename"
229 >pool/main/f/foobar/</TT
230 > ) ; then on 2nd Mar,
231 <TT
232 CLASS="filename"
233 >foobar_2_all.deb</TT
234 > is uploaded; but this
235 has a flaw (e.g. FTBFS) and so on 3rd Mar
236 <TT
237 CLASS="filename"
238 >foobar_3_all.deb</TT
239 > is uploaded.
240
241 On 2nd Mar, the delta server generates
242 <TT
243 CLASS="filename"
244 >pool/main/f/foobar/foobar_1_2_all.debdelta</TT
245 >
246 On 3rd Mar, the server generates both
247 <TT
248 CLASS="filename"
249 >pool/main/f/foobar/foobar_1_3_all.debdelta</TT
250 >
251 <TT
252 CLASS="filename"
253 >pool/main/f/foobar/foobar_2_3_all.debdelta</TT
254 >.
255
256 So, if the end-user Ann upgrades the system on both 2nd and 3rd Mar,
257 then she uses both foobar_1_2_all.debdelta (on 2nd) and
258 <TT
259 CLASS="filename"
260 >foobar_2_3_all.debdelta</TT
261 > (on 3rd Mar). If the end-user Boe has not
262 upgraded the system on 2nd Mar, , and he upgrades on 3rd Mar, then on
263 3rd Mar he uses <TT
264 CLASS="filename"
265 >foobar_1_3_all.debdelta</TT
266 >.
267 </P
268 ></DIV
269 ><DIV
270 CLASS="section"
271 ><H2
272 CLASS="section"
273 ><A
274 NAME="AEN107"
275 >3.5. size limit</A
276 ></H2
277 ><P
278 >&#13; Note that currently the server rejects deltas that exceed 70% of the deb
279 size: indeed the size gain would be too small, and the time would be
280 wasted, if you sum the time to download the delta and the time to apply
281 it (OK, these are run as much as possible in parallel, yet ....).
282 </P
283 ><P
284 >&#13; Also, the server does not generate delta for packages that are smaller than 10KB.
285 </P
286 ></DIV
287 ><DIV
288 CLASS="section"
289 ><H2
290 CLASS="section"
291 ><A
292 NAME="AEN111"
293 >3.6. /etc/debdelta/sources.conf</A
294 ></H2
295 ><P
296 >&#13; Consider a package that is currently installed. It is characterized by
297 <SPAN
298 CLASS="emphasis"
299 ><I
300 CLASS="emphasis"
301 > name installed_version architecture</I
302 ></SPAN
303 >
304 (unfortunately there is no way to tell from which archive it came
305 from, but this does not seem to be a problem currently)
306
307 Suppose now that a newer version is available somewhere in an archive,
308 and that the user wishes to upgrade to that version.
309
310 The archive Release file contain these info:
311 <SPAN
312 CLASS="QUOTE"
313 >"Origin , Label , Site, Archive"</SPAN
314 >.
315 (Note that Archive is called Suite in the Release file).
316 Example for the security archive:
317 <PRE
318 CLASS="programlisting"
319 >&#13; Origin=Debian
320 Label=Debian-Security
321 Archive=stable
322 Site=security.debian.org
323 </PRE
324 >
325 The file <TT
326 CLASS="filename"
327 >/etc/debdelta/sources.conf</TT
328 >
329 , given the above info, determines
330 the host that should contain the delta for upgrading the package. This
331 information is called "delta_uri" in that file.
332
333 The complete URL for the delta is built adding to the delta_uri a
334 directory path that mimicks the "pool" structure used in Debian
335 archives, and appending to it a filename of the form
336 <TT
337 CLASS="filename"
338 >name_oldversion_newversion_architecture.debdelta</TT
339 >.
340
341 All this is implemented in the example script contrib/findurl.py .
342
343 If the delta is not available at that URL, and
344 <TT
345 CLASS="filename"
346 >name_oldversion_newversion_architecture.debdelta-too-big</TT
347 >
348 is available, then the delta is too big to be useful.
349
350 If neither is present, then, either the delta has not yet been
351 generated, or it will never be generated... but this is difficult to
352 know.
353 </P
354 ></DIV
355 ><DIV
356 CLASS="section"
357 ><H2
358 CLASS="section"
359 ><A
360 NAME="AEN120"
361 >3.7. indexes</A
362 ></H2
363 ><DIV
364 CLASS="section"
365 ><H3
366 CLASS="section"
367 ><A
368 NAME="AEN122"
369 >3.7.1. indexes of debs in APT</A
370 ></H3
371 ><P
372 >&#13; Let's start examining the situation for debs and APT.
373 Using indexes for debs is a no-brainer decision: indeed, the client
374 (i.e. the end user) does not know the list of available debs in the
375 server, and, even knowing the current list, cannot foresee the future
376 changes.
377 So indexes provide needed informations: the packages' descriptions,
378 versions, dependencies, etc etc; these info are used by apt and the
379 other frontends.
380 </P
381 ></DIV
382 ><DIV
383 CLASS="section"
384 ><H3
385 CLASS="section"
386 ><A
387 NAME="no_indexes"
388 >3.7.2. no indexes of deltas in debdelta</A
389 ></H3
390 ><P
391 >&#13; If you then think of deltas, you realize that all requirements above
392 fall. Firstly there is no description and no dependencies for deltas.
393 <A
394 NAME="AEN128"
395 HREF="#FTN.AEN128"
396 ><SPAN
397 CLASS="footnote"
398 >[1]</SPAN
399 ></A
400 >
401
402 Of course 'debdelta-upgrade' needs some information to determine if a delta
403 exists, and to download it; but these information are already available:
404 <PRE
405 CLASS="programlisting"
406 >&#13; the name of the package P
407 the old version O
408 the new version N
409 the architecture A
410 </PRE
411 >
412 Once these are known, the URL of the file F can be algorithmically
413 determined as
414 <TT
415 CLASS="filename"
416 >URI/POOL/P_O_N_A.debdelta</TT
417 >
418 where URI is determined from
419 <TT
420 CLASS="filename"
421 >/etc/debdelta/sources.conf</TT
422 >
423 and POOL is the directory in the pool of the package P .
424 This algorithm is also implemented (quite verbosely) in
425 contrib/findurl.py in the sources of debdelta.
426
427 This is the reason why currently there is no "index of deltas", and
428 nonetheless 'debdelta-upgrade' works fine (and "cupt" as well).
429
430 Adding an index of file would only increase downloads (time and size)
431 and increase disk usage; with negligeable benefit, if any.
432 </P
433 ></DIV
434 ></DIV
435 ><DIV
436 CLASS="section"
437 ><H2
438 CLASS="section"
439 ><A
440 NAME="no_incremental"
441 >3.8. no incremental deltas</A
442 ></H2
443 ><P
444 >&#13; Let me add another point that may be unclear. There are no incremental
445 deltas (and IMHO never will be).
446 </P
447 ><DIV
448 CLASS="section"
449 ><H3
450 CLASS="section"
451 ><A
452 NAME="AEN136"
453 >3.8.1. What "incremental" would be, and why it is not</A
454 ></H3
455 ><P
456 >&#13; Please recall <A
457 HREF="x67.html#delta_creation"
458 >Section 3.4</A
459 >.
460
461 What <SPAN
462 CLASS="emphasis"
463 ><I
464 CLASS="emphasis"
465 >does not happen</I
466 ></SPAN
467 > currently is what follows:
468 on 3rd Mar , Boe decides to upgrade, and invokes 'debdelta-upgrade';
469 then 'debdelta-upgrade' finds <TT
470 CLASS="filename"
471 >foobar_1_2_all.debdelta</TT
472 > and
473 <TT
474 CLASS="filename"
475 >foobar_2_3_all.debdelta</TT
476 > , it uses the foremost to generate
477 <TT
478 CLASS="filename"
479 >foobar_2_all.deb</TT
480 >, and in turn it uses this and the second delta to
481 <TT
482 CLASS="filename"
483 >generate foobar_3_all.deb</TT
484 > .
485
486 This is not implemented, and it will not, for the following reasons.
487 <P
488 ></P
489 ><UL
490 ><LI
491 ><P
492 > The delta size is, on average, 40% of the size of the deb (and this
493 is getting worse, for different reasons, see <A
494 HREF="x285.html#getting_worse"
495 >Section 5.2</A
496 >); so two deltas are 80% of the
497 target deb, and this too much.
498 </P
499 ></LI
500 ><LI
501 ><P
502 > It takes time to apply a delta; applying two deltas to produce one
503 deb takes too much time.</P
504 ></LI
505 ><LI
506 ><P
507 > The server does generate the direct delta
508 <TT
509 CLASS="filename"
510 >foobar_1_3_all.debdelta</TT
511 >
512 :-) so why making things complex when they are easy? :-)</P
513 ></LI
514 ><LI
515 ><P
516 > Note also that incremental deltas would
517 need some index system to be implemented... indeed, Boe
518 would have no way to know on 3rd Mar that the intermediate
519 version of foobar between "1" and "3" is "2"; but since
520 incremental deltas do not exist, then there is no need to
521 have indexes). </P
522 ></LI
523 ></UL
524 >
525 </P
526 ></DIV
527 ></DIV
528 ><DIV
529 CLASS="section"
530 ><H2
531 CLASS="section"
532 ><A
533 NAME="repo_howto"
534 >3.9. Repository howto</A
535 ></H2
536 ><P
537 >There are (at least) three ways to manage a repository, and run a server that creates the deltas
538 </P
539 ><DIV
540 CLASS="section"
541 ><H3
542 CLASS="section"
543 ><A
544 NAME="AEN159"
545 >3.9.1. debdeltas_server</A
546 ></H3
547 ><P
548 >&#13; This is what is currently used.
549 </P
550 ></DIV
551 ><DIV
552 CLASS="section"
553 ><H3
554 CLASS="section"
555 ><A
556 NAME="AEN162"
557 >3.9.2. debmirror --debmarshal</A
558 ></H3
559 ><P
560 >&#13; This is what I used up to 2017 use. It is implemented in the script
561 <TT
562 CLASS="filename"
563 >/usr/share/debdelta/debmirror-marshal-deltas</TT
564 >
565 (a simpler version, much primitive but more readable , is
566 <TT
567 CLASS="filename"
568 >/usr/share/debdelta/debmirror-delta-security</TT
569 >)
570
571 I used the complex script that creates deltas for amd64 and
572 i386, and for lenny squeeze sid experimental ; and the simpler one for
573 lenny-security.
574
575 Let me start outlining how the simple script generate deltas . It is a 3 steps
576 process.
577
578 Lets say that $secdebmir is the directory containg the mirror of the
579 repository security.debian.org.
580
581 <P
582 ></P
583 ><OL
584 TYPE="1"
585 ><LI
586 ><PRE
587 CLASS="programlisting"
588 >&#13; --- 1st step
589 #make copy of current stable-security lists of packages
590 olddists=${TMPDIR:-/tmp}/oldsecdists-`date +'%F_%H-%M-%S'`
591 mkdir $olddists
592 cp -a $secdebmir/dists $olddists
593 </PRE
594 ></LI
595 ><LI
596 ><P
597 >&#13; --- 2nd step
598 call 'debmirror' to update the mirror ; note that I apply a patch to
599 debmirror so that old debs are not deleted , but moved to a /old_deb
600 directory
601 </P
602 ></LI
603 ><LI
604 ><P
605 >&#13; --- 3rd step
606 call 'debdeltas' to generate deltas , from the state of packages in
607 $olddists to the current state in $secdebmir , and also wrt what is in
608 stable.
609 Note that, for any package that was deleted from the archive, then
610 'debdeltas' will go fishing for it inside /old_deb .
611 </P
612 ></LI
613 ></OL
614 >
615 The more complex script uses the new <SPAN
616 CLASS="emphasis"
617 ><I
618 CLASS="emphasis"
619 >debmirror --debmarshal</I
620 ></SPAN
621 >
622 so it keeps 40 old snapshots of the deb archives, and it generates deltas of the current
623 package version (the "new" version) to the versions in snapshots -10,-20,-30,-40.
624 </P
625 ></DIV
626 ><DIV
627 CLASS="section"
628 ><H3
629 CLASS="section"
630 ><A
631 NAME="AEN175"
632 >3.9.3. hooks and repository of old_debs</A
633 ></H3
634 ><P
635 >&#13;
636 I wrote the scheleton for some commands.
637 <P
638 ><B
639 CLASS="command"
640 >debdelta_repo</B
641 > [--add name version arch filename disttoken]</P
642 >
643 This first one is to be called by the archive management tool (e.g. DAK) when a new package enters
644 in a part of the archive (lets say,
645 package="foobar" version="2" arch="all" and filename="pool/main/f/foobar/foobar_2_all.deb" just entered
646 disttoken="testing/main/amd64"). That command will add that to a delta queue, so
647 appropriate deltas will be generated; this command returns almost immediately.
648 <P
649 ><B
650 CLASS="command"
651 >debdelta_repo</B
652 > [--delta]</P
653 >
654 This does create all the deltas.
655 <P
656 ><B
657 CLASS="command"
658 >debdelta_repo</B
659 > [--sos filename]</P
660 >
661 This will be called by DAK when (before) it does delete a package from the archive;
662 this command will save that old deb somewhere (indeed it may be needed to generate deltas sometimes in the future).
663 (It will be up to some piece of <SPAN
664 CLASS="emphasis"
665 ><I
666 CLASS="emphasis"
667 >debdelta_repo</I
668 ></SPAN
669 > code to manage the repository of old debs, and
670 delete excess copies).
671 </P
672 ><P
673 ><SPAN
674 CLASS="emphasis"
675 ><I
676 CLASS="emphasis"
677 >TODO that scheleton does not handle 'security', where some old versions of the packages are in
678 a different DISTTOKEN</I
679 ></SPAN
680 ></P
681 ></DIV
682 ></DIV
683 ></DIV
684 ><H3
685 CLASS="FOOTNOTES"
686 >Notes</H3
687 ><TABLE
688 BORDER="0"
689 CLASS="FOOTNOTES"
690 WIDTH="100%"
691 ><TR
692 ><TD
693 ALIGN="LEFT"
694 VALIGN="TOP"
695 WIDTH="5%"
696 ><A
697 NAME="FTN.AEN128"
698 HREF="x67.html#AEN128"
699 ><SPAN
700 CLASS="footnote"
701 >[1]</SPAN
702 ></A
703 ></TD
704 ><TD
705 ALIGN="LEFT"
706 VALIGN="TOP"
707 WIDTH="95%"
708 ><P
709 >deltas have a "info" section, but that is, as to say, standalone</P
710 ></TD
711 ></TR
712 ></TABLE
713 ><DIV
714 CLASS="NAVFOOTER"
715 ><HR
716 ALIGN="LEFT"
717 WIDTH="100%"><TABLE
718 SUMMARY="Footer navigation table"
719 WIDTH="100%"
720 BORDER="0"
721 CELLPADDING="0"
722 CELLSPACING="0"
723 ><TR
724 ><TD
725 WIDTH="33%"
726 ALIGN="left"
727 VALIGN="top"
728 ><A
729 HREF="x56.html"
730 ACCESSKEY="P"
731 >Prev</A
732 ></TD
733 ><TD
734 WIDTH="34%"
735 ALIGN="center"
736 VALIGN="top"
737 ><A
738 HREF="index.html"
739 ACCESSKEY="H"
740 >Home</A
741 ></TD
742 ><TD
743 WIDTH="33%"
744 ALIGN="right"
745 VALIGN="top"
746 ><A
747 HREF="x190.html"
748 ACCESSKEY="N"
749 >Next</A
750 ></TD
751 ></TR
752 ><TR
753 ><TD
754 WIDTH="33%"
755 ALIGN="left"
756 VALIGN="top"
757 >a delta</TD
758 ><TD
759 WIDTH="34%"
760 ALIGN="center"
761 VALIGN="top"
762 >&nbsp;</TD
763 ><TD
764 WIDTH="33%"
765 ALIGN="right"
766 VALIGN="top"
767 >Goals, tricks, ideas and issues</TD
768 ></TR
769 ></TABLE
770 ></DIV
771 ></BODY
772 ></HTML
773 >