Debian Codegraph @

doc/html/x190.html

raw
1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN""http://www.w3.org/TR/html4/loose.dtd">
2 <HTML
3 ><HEAD
4 ><TITLE
5 >Goals, tricks, ideas and issues</TITLE
6 ><META
7 NAME="GENERATOR"
8 CONTENT="Modular DocBook HTML Stylesheet Version 1.79"><LINK
9 REL="HOME"
10 HREF="index.html"><LINK
11 REL="PREVIOUS"
12 TITLE="new debdelta-upgrade service"
13 HREF="x67.html"><LINK
14 REL="NEXT"
15 TITLE="Todo"
16 HREF="x285.html"></HEAD
17 ><BODY
18 CLASS="section"
19 BGCOLOR="#FFFFFF"
20 TEXT="#000000"
21 LINK="#0000FF"
22 VLINK="#840084"
23 ALINK="#0000FF"
24 ><DIV
25 CLASS="NAVHEADER"
26 ><TABLE
27 SUMMARY="Header navigation table"
28 WIDTH="100%"
29 BORDER="0"
30 CELLPADDING="0"
31 CELLSPACING="0"
32 ><TR
33 ><TH
34 COLSPAN="3"
35 ALIGN="center"
36 ></TH
37 ></TR
38 ><TR
39 ><TD
40 WIDTH="10%"
41 ALIGN="left"
42 VALIGN="bottom"
43 ><A
44 HREF="x67.html"
45 ACCESSKEY="P"
46 >Prev</A
47 ></TD
48 ><TD
49 WIDTH="80%"
50 ALIGN="center"
51 VALIGN="bottom"
52 ></TD
53 ><TD
54 WIDTH="10%"
55 ALIGN="right"
56 VALIGN="bottom"
57 ><A
58 HREF="x285.html"
59 ACCESSKEY="N"
60 >Next</A
61 ></TD
62 ></TR
63 ></TABLE
64 ><HR
65 ALIGN="LEFT"
66 WIDTH="100%"></DIV
67 ><DIV
68 CLASS="section"
69 ><H1
70 CLASS="section"
71 ><A
72 NAME="AEN190"
73 >4. Goals, tricks, ideas and issues</A
74 ></H1
75 ><DIV
76 CLASS="section"
77 ><H2
78 CLASS="section"
79 ><A
80 NAME="AEN192"
81 >4.1. exact patching</A
82 ></H2
83 ><P
84 >&#13; When <B
85 CLASS="command"
86 >debpatch</B
87 > or <B
88 CLASS="command"
89 >debdelta-upgrade</B
90 >
91 recreates a .deb, it will be identical to the desired
92 one (so it may be possible to check it using the
93 <A
94 HREF="http://wiki.debian.org/SecureApt"
95 TARGET="_top"
96 >&#13; security features in APT</A
97 >
98 <A
99 NAME="AEN198"
100 HREF="#FTN.AEN198"
101 ><SPAN
102 CLASS="footnote"
103 >[1]</SPAN
104 ></A
105 >). See though <A
106 HREF="x190.html#long_time"
107 >Section 4.5</A
108 >.
109 </P
110 ></DIV
111 ><DIV
112 CLASS="section"
113 ><H2
114 CLASS="section"
115 ><A
116 NAME="AEN203"
117 >4.2. exact recompression</A
118 ></H2
119 ><P
120 >&#13; Suppose a .deb has inside a huge file
121 /usr/share/doc/foobar/document.info.gz
122 and this starts with a RCS tag ... then each time it
123 is released, the file will be different even though
124 just few bytes were changed. Another examples are manpages that start with the header
125 containing the version of the command.
126 So , to get good compression of the difference, I had
127 to be able to gunzip those files, diff them,
128 and gzip back them <SPAN
129 CLASS="emphasis"
130 ><I
131 CLASS="emphasis"
132 >exactly identical</I
133 ></SPAN
134 > (but possibly for headers
135 <A
136 NAME="AEN207"
137 HREF="#FTN.AEN207"
138 ><SPAN
139 CLASS="footnote"
140 >[2]</SPAN
141 ></A
142 >)
143 For this reason, I studied gzip formats, and I wrote in debdelta
144 some python code that does the trick (90% of the times...).
145 <A
146 NAME="AEN209"
147 HREF="#FTN.AEN209"
148 ><SPAN
149 CLASS="footnote"
150 >[3]</SPAN
151 ></A
152 >
153 </P
154 ></DIV
155 ><DIV
156 CLASS="section"
157 ><H2
158 CLASS="section"
159 ><A
160 NAME="AEN212"
161 >4.3. speed</A
162 ></H2
163 ><DIV
164 CLASS="section"
165 ><H3
166 CLASS="section"
167 ><A
168 NAME="AEN214"
169 >4.3.1. some (old) numbers</A
170 ></H3
171 ><P
172 >&#13; Warning: this section is referred to experiments done in 2006, and the backend for
173 delta encoding was 'xdelta'.
174
175 On a desktop with CPU Athlon64 3000 and a average hard disk,
176 <PRE
177 CLASS="programlisting"
178 >&#13; $ debdelta mozilla-browser_1.7.8-1sarge3_i386.deb \
179 mozilla-browser_1.7.8-1sarge6_i386.deb /tmp/m-b.debdelta
180 </PRE
181 >
182 processes the 10Mb of mozilla-browser in ~11sec,
183 that is a speed of ~900kB per second.
184
185 Then debpatch applies the above delta in 16sec,
186 at a speed of ~600kB per second.
187
188 Numbers drop in a old PC, or in a notebook (like mine, that has a
189 Athlon 1600MHz and slow disks), where data are chewed at ~200kB per
190 second. Still, since I have a ADSL line that downloads at
191 max 80kB per second, I have a benefit downloading deltas.
192
193 In a theoretical example, indeed, to download a 80MB package, it would
194 take 1000seconds; whereas to download a delta that is 20% of 80MB it
195 takes 200seconds, and then 80MB / (200kB/sec) = 400seconds to apply
196 it, for a total of 600seconds. So I may get a "virtual speed" of 80MB /
197 600sec = 130kB/sec .
198
199 Note that delta downloading and delta patching is done in parallel:
200 if 4 packages as above have to be downloaded, then the total
201 time for downloading of full debs would be 4000seconds, while the time
202 for parallel-download-patch-apply-patch may be as low as 1400seconds.
203 </P
204 ><P
205 >&#13; This is a real example of running 'debdelta-upgrade' :
206 <PRE
207 CLASS="programlisting"
208 >&#13; Looking for a delta for libc6 from 2.3.6-9 to 2.3.6-11
209 Looking for a delta for udev from 0.092-2 to 0.093-1
210 Patching done, time: 22sec, speed: 204kB/sec, result: libc6_2.3.6-11_i386.deb
211 Patching done, time: 4sec, speed: 57kB/sec, result: udev_0.093-1_i386.deb
212 Delta-upgrade download time 28sec speed 21.6k/sec
213 total time: 53sec; virtual speed: 93.9k/sec.
214 </PRE
215 >
216 (Note that the "virtual speed" of 93.9k/sec , while less than the
217 130kB/sec of the theoretical example above, is still more than the
218 80kB that my ADSL line would allow).
219
220 Of course the above is even better for people with fast disks and/or
221 slow modems.
222
223 Actually, an apt delta method may do a smart decision of how many
224 deltas to download, and in which order, to optimize the result, (given
225 the deltas size, the packages size, the downloading speed and the
226 patching speed).
227 </P
228 ></DIV
229 ><DIV
230 CLASS="section"
231 ><H3
232 CLASS="section"
233 ><A
234 NAME="AEN220"
235 >4.3.2. speeding up</A
236 ></H3
237 ><P
238 >&#13; The problem is that the process of applying a delta to create a new
239 deb is currently slow, even on very fast machines.
240
241 One way to overcome is to "parallelize as much as possible".
242
243 The best strategy that I can imagine is to keep both the CPU,
244 the hard disk, and the Internet connection, always maxed up.
245
246 This is why 'debdelta-upgrade' has two threads, the "downloading
247 thread" and the "patching thread". The downloading thread downloads
248 deltas (ordered by increasing size), and as soon as they are
249 downloaded, it queues them to be applied in the "patching thread";
250 whereas as soon as all available deltas are downloaded it starts
251 downloading some debs, and goes on for as long as the deltas are being
252 applied in the "patching thread".
253
254 Summarizing, the downloading thread keeps Internet busy while the
255 patching thread keeps the CPU and HDD busy.
256 </P
257 ><P
258 >&#13; Another speedup strategy is embedded inside the deltas
259 themselves: since bsdiff is a memory hog, when the backend is
260 bsdiff, I have to divide the data in chunks; this may lower the
261 compression ratio, but the good point is that the HDD accesses
262 and the calls to bsdiff can run "in parallel". With newer
263 xdelta3, xdelta3 can read the original data from a pipe, so the
264 data are not divided in chunks, but rather continously piped
265 into xdelta3; so xdelta3 runs at the same time as when the data
266 are read from HDD.
267 </P
268 ></DIV
269 ><DIV
270 CLASS="section"
271 ><H3
272 CLASS="section"
273 ><A
274 NAME="AEN224"
275 >4.3.3. the 10kb trick</A
276 ></H3
277 ><P
278 >currently, roughly half of the generated deltas<A
279 NAME="AEN227"
280 HREF="#FTN.AEN227"
281 ><SPAN
282 CLASS="footnote"
283 >[4]</SPAN
284 ></A
285 > are less than 10KB.
286 <B
287 CLASS="command"
288 >debdelta-upgrade</B
289 > downloads deltas in two passes,
290 <P
291 ></P
292 ><OL
293 TYPE="1"
294 ><LI
295 ><P
296 >in the first pass it tries to download the first 10KB of a delta;
297 if it gets a complete delta, it immediatly pipes it in the "patching thread queue", otherwise if it gets
298 only a partial download, it adds it to the download queue; if it gets HTTP404, it
299 possibly checks for the "toobig" timestamp, and it possibly warns the user.
300 </P
301 ></LI
302 ><LI
303 ><P
304 >in the second pass, it downloads the rest of the deltas, and queues them for patching</P
305 ></LI
306 ></OL
307 >
308 Why this complex method? because the first 10KBs of a delta contain the info, and those may be used
309 to actually decide not to download the rest of the delta (if a TODO predictor
310 decides that it is not worthwhile...<A
311 HREF="x190.html#predictor"
312 >Section 4.3.4</A
313 >).
314 </P
315 ></DIV
316 ><DIV
317 CLASS="section"
318 ><H3
319 CLASS="section"
320 ><A
321 NAME="predictor"
322 >4.3.4. the choice, the predictor</A
323 ></H3
324 ><P
325 >&#13; Which deltas should be downloaded, VS which debs?
326
327 Currently there is a rule-of-thumb: the server immediately deletes any
328 delta that exceeds 70% of the original deb , and it replaces it with
329 an empty file ending in ".debdelta-too-big". In such cases,
330 "debdelta-upgrade" will download the deb instead.
331
332 See the explanation of "debdelta-upgrade --deb-policy" in the man page
333 for more info and customization on which debs get downloaded.
334 </P
335 ><P
336 >&#13; Some time ago I tried to do devise a better way to understand when to
337 download a delta w.r.t. a deb. The code is in the "Predictor" class
338 .... but I could not reliably predict the final speed of patching, so
339 currently it is not used.
340 </P
341 ></DIV
342 ><DIV
343 CLASS="section"
344 ><H3
345 CLASS="section"
346 ><A
347 NAME="AEN240"
348 >4.3.5. State of the art</A
349 ></H3
350 ><P
351 >&#13; All in all, I still cannot obtain high speeds: so people that have a fast
352 ADSL Internet connection usually are better
353 downloading all the debs, and ignoring "debdelta-upgrade" alltogether.
354 Anyway, the best way to know is to try "debdelta-upgrade -v" and
355 read the final statistics. See
356 <A
357 HREF="x190.html#format_unzipped"
358 >Section 4.7</A
359 >
360 and <A
361 HREF="x190.html#format_preunpacked"
362 >Section 4.8</A
363 > for recent developments.
364 </P
365 ></DIV
366 ></DIV
367 ><DIV
368 CLASS="section"
369 ><H2
370 CLASS="section"
371 ><A
372 NAME="the_enemy_within"
373 >4.4. better deb compression is a worse delta</A
374 ></H2
375 ><P
376 >&#13; 'xdelta3' can reconstruct data at high speed: on nowadays processors, it can process up to 2MB per second;
377 but, when applying a delta, 'xdelta3' works on <SPAN
378 CLASS="emphasis"
379 ><I
380 CLASS="emphasis"
381 >uncompressed data</I
382 ></SPAN
383 >.
384 So if the data is then compressed at a ratio 1/3, then the resulting speed on <SPAN
385 CLASS="emphasis"
386 ><I
387 CLASS="emphasis"
388 >compressed data</I
389 ></SPAN
390 >
391 is 700KB/sec. Moreover, time is needed to actually compress the data.
392 </P
393 ><P
394 >&#13; In recent years, 'dpkg' has transitioned from 'data.tar.gz' to 'data.tar.bz2' to 'data.tar.lzma';
395 each method is better at compressing, but is also slower than the previous one; since it is better at
396 compressing, it also defeats the ability of 'debdelta' to produce small deltas (wrt the original deb, of course),
397 and indeed statistics show that deltas are getting larger; since it is slower, it slows down the applying of
398 deltas as well.</P
399 ></DIV
400 ><DIV
401 CLASS="section"
402 ><H2
403 CLASS="section"
404 ><A
405 NAME="long_time"
406 >4.5. long time recovery</A
407 ></H2
408 ><P
409 >As aforementioned, deltas can rebuild the deb identically to the byte. But the patch.sh script
410 calls the standard tools 'tail','head','zgip','bzip2','lzma', etc etc to rebuild a delta; so
411 if the argument calling or output of any of those tools changes, than a delta may become unusable.
412 As long as deltas are used for the debdelta-upgrade service, this is no big deal: if such a tool changes,
413 then we can adjust the deltas to it, and there is just some days disruption of the service
414 <A
415 NAME="AEN254"
416 HREF="#FTN.AEN254"
417 ><SPAN
418 CLASS="footnote"
419 >[5]</SPAN
420 ></A
421 >
422 (and people will download debs instead of deltas .... as we used to).</P
423 ><P
424 >If anybody wants instead to use debdelta to archive debs for long time, (as the archive.debian.org service
425 was doing), then we should make sure that , at any moment in future, deltas can be applied.
426 A possible solution would be that deltas should contain, in the info files, the versions of all tools that
427 are needed for applying. A second solution is that debdelta should keep a standard set of those tools inside the package.
428 </P
429 ></DIV
430 ><DIV
431 CLASS="section"
432 ><H2
433 CLASS="section"
434 ><A
435 NAME="AEN257"
436 >4.6. streaming</A
437 ></H2
438 ><P
439 >&#13; Let me summarize. When 'debdelta-upgrade' (or 'debpatch') recreates a
440 deb, one step is reassembling the data.tar part inside it; this part
441 moreover is compressed (gzip, bzip2 or lately lzma). This
442 'reassembling and compressing' takes time (both for CPU and for HD),
443 and is moreover quite useless, since, in short time, 'apt' will call
444 'dpkg -i' that decompresses and reopens the data.tar in the deb.
445 </P
446 ><P
447 >&#13; It is then reasonable to collapse this two parts, and this would
448 possibly speed up the upgrade a bit. A first step is
449 <SPAN
450 CLASS="emphasis"
451 ><I
452 CLASS="emphasis"
453 >'--format=unzipped'</I
454 ></SPAN
455 > <A
456 HREF="x190.html#format_unzipped"
457 >Section 4.7</A
458 >
459 , a next step may be <SPAN
460 CLASS="emphasis"
461 ><I
462 CLASS="emphasis"
463 >'--format=preunpacked'</I
464 ></SPAN
465 >
466 <A
467 HREF="x190.html#format_preunpacked"
468 >Section 4.8</A
469 >.
470 </P
471 ></DIV
472 ><DIV
473 CLASS="section"
474 ><H2
475 CLASS="section"
476 ><A
477 NAME="format_unzipped"
478 >4.7. --format=unzipped</A
479 ></H2
480 ><P
481 >The recently introduced
482 new <CODE
483 CLASS="varname"
484 >--format=unzipped</CODE
485 >
486 may speed up package upgrades. If you call
487 'debdelta-upgrade' with the option '--format=unzipped' , then in the
488 recreated deb the data.tar part will not be compressed.
489 This may speedup the 'debdelta-upgrade' + 'apt-get upgrade' process. Indeed, writing
490 to hard disk is fast (let's say 5MB/sec, but usually much more); whereas
491 compressing random data with 'bzip2 -9' or 'lzma -9' is much slower
492 (let's say 2.0MB/sec and 1.5 MB/sec) ; and moreover the compressed data
493 is then decompressed by dpkg when installing; so avoiding the
494 compress/decompress should be a win/win (unless you run out of disk
495 space...). Indeed I see that the creation of deltas is much faster;
496 but I still do not have enough data collected....
497 </P
498 ></DIV
499 ><DIV
500 CLASS="section"
501 ><H2
502 CLASS="section"
503 ><A
504 NAME="format_preunpacked"
505 >4.8. --format=preunpacked</A
506 ></H2
507 ><P
508 >&#13; Here is another idea. When 'debdelta-upgrade' is called in upgrading a
509 package 'foobar' it currently creates 'foobar_2.deb'. By an
510 appropriate cmdline switch <SPAN
511 CLASS="emphasis"
512 ><I
513 CLASS="emphasis"
514 >'--format=preunpacked'</I
515 ></SPAN
516 >,
517 instead of creating a 'foobar_2.deb' , it
518 directly saves all of its file to the filesystem, and it
519 adds an extension to all the file names, making sure that no file name
520 conflicts (=overwrite) with a preexisting file on the filesystem
521 ; then it creates a file 'foobar_2.deb_preunpacked' , that is
522 a deb package were 'data.tar.xxx' is replaced with
523 'data_list', just a text file specifying the contents of 'data.tar.xxx'
524 and where regular files were temporarily unpacked.
525 </P
526 ><P
527 >&#13; Note that the above idea overlaps a lot with
528 <A
529 HREF="http://wiki.debian.org/SummerOfCode2010/StreamingPackageInstall"
530 TARGET="_top"
531 >&#13; the SummerOfCode2010 StreamingPackageInstall</A
532 >
533 </P
534 ><P
535 >
536 <B
537 CLASS="command"
538 >debdelta-upgrade --format=preunpacked</B
539 > is now implemented as a proof-of-concept
540 (it does not really write temporary files to HD yet).
541
542
543
544
545
546 The format of <SPAN
547 CLASS="emphasis"
548 ><I
549 CLASS="emphasis"
550 >data_list</I
551 ></SPAN
552 > is
553 <PRE
554 CLASS="programlisting"
555 >&#13;Files:
556 TYPE MODE USER GROUP MTIME
557 NAME_FILE_WAS_UNPACKED_TO (if regular file)
558 ORIGINAL_FILENAME
559 LINK_NAME (if link)
560 [repeat]
561 </PRE
562 >
563 Example of data_list
564 <PRE
565 CLASS="programlisting"
566 >&#13;Files:
567 d 0755 root root 1304626623
568
569 ./etc
570
571 - 0644 root root 1304626594
572 /./etc/gnashrc_1956_debdelta_preunpacked
573 ./etc/gnashrc
574 l 0777 root root 1304626629
575
576 ./usr/share/man/man1/gtk-gnash.1.gz
577 gnash.1.gz
578 </PRE
579 >
580 </P
581 ><P
582 > PROS: (1) may be faster; (2) if you need to
583 upgrade a 100MB package, you do not need to save both the deb
584 and (while 'dpkg --unpack') the whole new deb data : so there is
585 less risk of running our of disk space.
586 </P
587 ><P
588 > CONS: (1)
589 you cannot install that "preunpacked deb" twice (so dpkg should
590 probably remove it once it has installed it); (2) you cannot
591 move it to another host; (3) when "apt-get clean", all temporary
592 files have to be removed as well.
593 </P
594 ><P
595 > <SPAN
596 CLASS="emphasis"
597 ><I
598 CLASS="emphasis"
599 > So it may be a
600 good idea to use ".deb_preunpacked" as extension for them. And
601 I would recommend using '--format=unzipped' for essential
602 packages such as the kernel.</I
603 ></SPAN
604 >
605 </P
606 ><P
607 >&#13; If you like the idea, someone should help in changing 'dpkg' so that it would be
608 able to install starting from 'foobar_2.deb_preunpacked'. And change APT
609 so that it would interact with 'debdelta' to create the
610 'foobar_2.deb_unpacked' files, and pass them to dpkg (and clean them properly).
611 </P
612 ></DIV
613 ></DIV
614 ><H3
615 CLASS="FOOTNOTES"
616 >Notes</H3
617 ><TABLE
618 BORDER="0"
619 CLASS="FOOTNOTES"
620 WIDTH="100%"
621 ><TR
622 ><TD
623 ALIGN="LEFT"
624 VALIGN="TOP"
625 WIDTH="5%"
626 ><A
627 NAME="FTN.AEN198"
628 HREF="x190.html#AEN198"
629 ><SPAN
630 CLASS="footnote"
631 >[1]</SPAN
632 ></A
633 ></TD
634 ><TD
635 ALIGN="LEFT"
636 VALIGN="TOP"
637 WIDTH="95%"
638 ><P
639 >note though that <B
640 CLASS="command"
641 >debdelta-upgrade</B
642 > saves the
643 recontructed debs in <TT
644 CLASS="filename"
645 >/var/cache/apt/archives</TT
646 >, and APT does not check
647 them there, AFAICT</P
648 ></TD
649 ></TR
650 ><TR
651 ><TD
652 ALIGN="LEFT"
653 VALIGN="TOP"
654 WIDTH="5%"
655 ><A
656 NAME="FTN.AEN207"
657 HREF="x190.html#AEN207"
658 ><SPAN
659 CLASS="footnote"
660 >[2]</SPAN
661 ></A
662 ></TD
663 ><TD
664 ALIGN="LEFT"
665 VALIGN="TOP"
666 WIDTH="95%"
667 ><P
668 >the re-gzipped files are identical but for headers,
669 (indeed gzip headers contain sometimes a timestamp ); but this is not a problem
670 since the reconstructed gzipeed file is then piped again into 'xdelta3' or 'bsdiff' to rebuild the 'data.tar',
671 so the header is fixed at that stage
672 </P
673 ></TD
674 ></TR
675 ><TR
676 ><TD
677 ALIGN="LEFT"
678 VALIGN="TOP"
679 WIDTH="5%"
680 ><A
681 NAME="FTN.AEN209"
682 HREF="x190.html#AEN209"
683 ><SPAN
684 CLASS="footnote"
685 >[3]</SPAN
686 ></A
687 ></TD
688 ><TD
689 ALIGN="LEFT"
690 VALIGN="TOP"
691 WIDTH="95%"
692 ><P
693 >This is implemented in the python routine <CODE
694 CLASS="function"
695 >delta_gzipped_files</CODE
696 >.
697 </P
698 ></TD
699 ></TR
700 ><TR
701 ><TD
702 ALIGN="LEFT"
703 VALIGN="TOP"
704 WIDTH="5%"
705 ><A
706 NAME="FTN.AEN227"
707 HREF="x190.html#AEN227"
708 ><SPAN
709 CLASS="footnote"
710 >[4]</SPAN
711 ></A
712 ></TD
713 ><TD
714 ALIGN="LEFT"
715 VALIGN="TOP"
716 WIDTH="95%"
717 ><P
718 >that is, discarding those that
719 are more than 70% of the corresponding deb</P
720 ></TD
721 ></TR
722 ><TR
723 ><TD
724 ALIGN="LEFT"
725 VALIGN="TOP"
726 WIDTH="5%"
727 ><A
728 NAME="FTN.AEN254"
729 HREF="x190.html#AEN254"
730 ><SPAN
731 CLASS="footnote"
732 >[5]</SPAN
733 ></A
734 ></TD
735 ><TD
736 ALIGN="LEFT"
737 VALIGN="TOP"
738 WIDTH="95%"
739 ><P
740 >this actually already happened some years ago, with libzip</P
741 ></TD
742 ></TR
743 ></TABLE
744 ><DIV
745 CLASS="NAVFOOTER"
746 ><HR
747 ALIGN="LEFT"
748 WIDTH="100%"><TABLE
749 SUMMARY="Footer navigation table"
750 WIDTH="100%"
751 BORDER="0"
752 CELLPADDING="0"
753 CELLSPACING="0"
754 ><TR
755 ><TD
756 WIDTH="33%"
757 ALIGN="left"
758 VALIGN="top"
759 ><A
760 HREF="x67.html"
761 ACCESSKEY="P"
762 >Prev</A
763 ></TD
764 ><TD
765 WIDTH="34%"
766 ALIGN="center"
767 VALIGN="top"
768 ><A
769 HREF="index.html"
770 ACCESSKEY="H"
771 >Home</A
772 ></TD
773 ><TD
774 WIDTH="33%"
775 ALIGN="right"
776 VALIGN="top"
777 ><A
778 HREF="x285.html"
779 ACCESSKEY="N"
780 >Next</A
781 ></TD
782 ></TR
783 ><TR
784 ><TD
785 WIDTH="33%"
786 ALIGN="left"
787 VALIGN="top"
788 >new debdelta-upgrade service</TD
789 ><TD
790 WIDTH="34%"
791 ALIGN="center"
792 VALIGN="top"
793 >&nbsp;</TD
794 ><TD
795 WIDTH="33%"
796 ALIGN="right"
797 VALIGN="top"
798 >Todo</TD
799 ></TR
800 ></TABLE
801 ></DIV
802 ></BODY
803 ></HTML
804 >