Debian Codegraph @

doc/README

raw
1 Apt-Cacher-NG User Manual
2 =========================
3
4 Apt-Cacher NG is a caching proxy for software packages which are downloaded
5 by Unix/Linux system distribution mechanisms from mirror servers accessible
6 via HTTP.
7
8 This manual provides an overview of Apt-Cacher-NG's features and a walk
9 through the required configuration steps for server administrators and
10 users of the proxy.
11
12 Chapter 1: Introduction
13 -----------------------
14
15 apt-cacher-ng attempts to achieve the same goals as related proxies
16 - it acts as a proxy which is used by clients in the local network
17 to share the data that has been downloaded. It monitors the state of
18 packages and is capable of merging downloads of the same packages
19 from different locations (real or simulated).
20
21 The program reuses many ideas behind the other famous proxy, its
22 predecessor apt-cacher 1.x (which has been written in Perl). In
23 contrast to apt-cacher, different aspects have been declared as
24 primary targets during the development of apt-cacher-ng:
25
26 - lightweight implementation - allow use on systems with low
27 memory and processing resources
28
29 - internal (native) threading - avoiding process fork'ing wherever
30 possible, avoiding kludges for pseudo-thread synchronization,
31 avoiding relying on special file system features for internal
32 operations where possible
33
34 - real (effective) support of HTTP pipelining, using an internal
35 client with native stream control (having the nice side effect:
36 reduction of resource overhead and minimization of possible
37 points of failure)
38
39 - avoiding featuritis where they cause too much bloat and the
40 functionality can be provided by native OS features
41
42 - reliable but efficient content merging in the local package
43 pool, avoiding delivering of wrong data.
44
45 As with apt-cacher, explicit tracking of dynamically changed
46 and unchanged files is established, and the use in non-Debian
47 environment is supported.
48
49 Long story: Not all goals have been achieved. The initial plan of
50 using background databases to merge any download from any arbitrary
51 location has been dropped because of complexity and performance
52 considerations, reliable heuristics could not be found either.
53 Instead, a semi-automated solution has been created which used
54 machine-parsable files with mirror information, like the one
55 available for Debian mirrors in Debian's CVS repository.
56
57 Chapter 2: Running apt-cacher-ng
58 --------------------------------
59
60 Run "build/apt-cacher-ng -c conf" when configured where conf is the
61 configuration directory. See section 4.2 for details on possible and
62 required contents of this directory.
63
64 Most options from the configuration file can also be passed through
65 command line parameters. Just append them with the same format as in
66 the configuration file but without separating spaces inside, e.g.
67
68 Port:4855 ForeGround=1
69
70 For convenience, the colon can also be replaced with the equals sign
71 and letter case does not matter, so this is also possible:
72
73 port=4855 foreground:1.
74
75 Chapter 3: Basic Configuration
76 ------------------------------
77
78 3.1 Server Configuration
79
80 Unlike some rumors on the internet claim, there should be no need
81 for exhausting configuration work to just test apt-cacher-ng and run
82 it with default parameters. It's actually designed to bootstrap most
83 of its working environment without additional help.
84
85 The package setup scripts used by distributions should already
86 prepare working initial settings for apt-cacher-ng. Check the
87 file `/etc/apt-cacher-ng/acng.conf' file where most settings are
88 explained. For the beginning they should not be changed, the only
89 interesting setting present there is the TCP port. See Advanced
90 Server Configuration for details.
91
92 There is also a daily cron job which executes some maintenance work.
93 Additional automated control commands can be added by administrator.
94
95 3.2 Client Configuration
96
97 From the client side, apt-cacher-ng can be used as a drop-in
98 replacement for apt-cacher. The same rules apply, e.g. Debian/Ubuntu
99 users should EITHER:
100
101 - Specify the caching machine as HTTP Proxy for your download
102 client. This can be usually done (temporarily) with the
103 environment variable `http_proxy' or (permanently) in some
104 configuration file or dialog. For APT, this means putting a line
105 like the following into a file like /etc/apt/apt.conf.d/02proxy:
106
107 Acquire::http::proxy "http://CacheServerIp:3142";
108
109 OR:
110
111 - Replace all mirror hostnames with cachinghost/hostname in
112 sources.list, so
113
114 deb http://ftp.uni-kl.de/debian etch main
115
116 should now become:
117
118 deb http://192.168.0.17:3142/ftp.uni-kl.de/debian etch main
119
120 (assuming that CacheServerIp is 192.168.0.17 and the service port
121 is 3142) .
122
123 These both methods have their own pros and cons. The last method can
124 be used with clients which cannot configure an additional http proxy
125 for some reason. The disadvantages are the worse progress visibility
126 (all downloads seem to come from the same machine) and some resource
127 usage limits might be hit (i.e. maximum number of simultaneous
128 downloads from the "same" machine). It might also require to modify
129 many different URLs and some of those URLs might be hardcoded in
130 software and not be accessible to the user.
131
132 The former method is more convenient since it usually means less
133 configuration work; however, it implies that all relevant client
134 programs respect the proxy setting and use it from this central
135 location.
136
137 Mixing the configuration methods is usually possible but not within
138 the same client program instance. Doing that (going with proxy mode
139 AND use rewritten URLs) will probably confuse the server: in best-
140 case, the connection will be slower because of a little transport
141 loop on the server side, and in the worst-case, the target URL will
142 eventually become not resolvable and the downloads will just fail.
143
144 Using SSL/TLS transport (i.e. https urls) is also possible with some
145 restrictions, see section 8.2 for details.
146
147 Chapter 4: Advanced Server Configuration
148 ----------------------------------------
149
150 4.1 Vocabulary
151
152 This chapter introduces some terminology which is needed to
153 understand the functionality of apt-cacher-ng; it's recommended to
154 understand it before continuing with the advanced configuration.
155
156 - "Backend": a text file consisting of a list of mirror URLs, one
157 per line (a more complex RFC822-like format is also supported).
158 Used for URL remapping; see section 4.3.
159
160 - "Volatile files": files with changing contents on the server
161 while keeping the same URL (i.e. the same filename and
162 location). For example, metadata pertaining to package files
163 stored in a remote archive is classified as 'volatile'. They
164 are usually 'index files' known as Packages, Sources, Release,
165 Pdiff and the like, and maybe also associated extra data (like
166 translations and thumbnail icons).
167
168 - "Package files": files that contain software packages and other
169 "solid" data: DEBs, source files for their creation (.tar.gz,
170 .diff, .dsc), various metadata which is not subject to change
171 after first appearance on the server.
172
173 - "Configuration line": one single line in the configuration file.
174 Some examples in this chapter may be displayed as wrapped lines
175 but should be stored as a single line in the configuration. Some
176 variables (like `RequestAppendix') support multiline data which
177 needs to be separated with \n delimiter.
178
179 4.2 Configuration file types
180
181 By default, the /etc/apt-cacher-ng directory (or the one specified
182 with program options) contains all config files, HTML page
183 templates, the stylesheet and other text-based support files used by
184 apt-cacher-ng. The contents may vary depending on the installation
185 of apt-cacher-ng, refer to the package documentation for Linux
186 Distribution packages.
187
188 There are a few certain file types distinguished by apt-cacher-ng:
189
190 1. Main configuration files:
191
192 *.conf files are assumed to contain configuration directives
193 in the form of "key: value" pairs. The package comes with a
194 commented example configuration file. apt-cacher-ng reads all
195 files matching *.conf in alphabetical order and merges the
196 contents. For options documentation, see commented example file
197 shipped with apt-cacher-ng (conf/ directory in original source).
198
199 For security reasons, the files can be made readable only to
200 the daemon and administorator accounts, e.g. when they contain
201 passwords or other sensitive data.
202
203 2. URL lists and remote repository list files. The file names
204 are arbitrary, no special suffix is required. They are read
205 and included during processing of configuration files and can
206 contain data in one of the following formats:
207
208 - simple text files with one URL per line (the URL should
209 point to the base directory of the repository, e.g.
210 "http://ftp.de.debian.org/debian/"). A URL must start with
211 http:// and should end with a slash
212
213 - an RFC822-like format, with lines like 'Site: <hostname>'
214 and 'Archive-http: /base/directory/of/repository/'. Optional
215 fields are also used in this remapping descriptions to add
216 more possible variants (Alias, Aliases, X-Archive-http:) of
217 the URLs to the lookup list
218
219 3. Various support files used for the configuration web interface,
220 named like *.css and *.html.
221
222 4. *.default files are used in some rare cases as replacement for
223 list files having the same name without .default suffix.
224
225 5. *.hooks files specify custom actions which can be executed upon
226 connection/disconnection (see section 4.3.2 for details).
227
228 Except from .conf files, most files listed above can be moved to
229 another "support" directory and the daemon will look for them
230 therein if they are not present in the primary configuration
231 directory. This feature is intended to help keeping pure
232 configuration data and mostly static data in different locations.
233 The directory path is specified at build time and can be overriden
234 with the SupportDir directive (and if used, this should be set as
235 early as possible).
236
237 4.3 Repositories and URL mapping
238
239 With the most simple configuration, apt-cacher-ng will act almost
240 like an ordinary HTTP proxy with improved caching behaviour. When
241 files are requested, they are downloaded from a remote location
242 specified in client's request and are stored in a unique way.
243
244 However, for some use cases it can be beneficial to specify
245 additional rules to achieve further improvements, e.g. in order to
246 detect and prevent avoidable downloads, to reduce space requirements
247 for the cache directory or simply hide real download locations from
248 the APT clients.
249
250 These modifications are generally achieved by two strategies,
251 `Merging' and `Redirection', which are configured in a context of a
252 specified cache `Repository'. The configuration for them is created
253 using one or multiple Remap-... configuration directives (see
254 below).
255
256 Merging:
257
258 "Merging" of incoming requests can be done if some subdirectories of
259 different remote servers are considered equal where the last part of
260 the remote file path leads to the same file content. When specified,
261 the internal cache content is shared and the live download stream
262 is shared. The configuration work consists of setting an "equality
263 list" containing a set of URLs representing the base directories
264 (like `http://ftp.debian.org/debian' and `http://ftp.uni-
265 kl.de/pub/linux/debian').
266
267 Redirection:
268
269 With redirection, client requests cause a download from a remote
270 location which is different from what clients requested and believe
271 to receive from. Redirection is an optional feature; if used, it's
272 configured by one or multiple URL(s) pointing to target servers. The
273 URL(s) must include a directory spec which matches the directory
274 level of the URLs in the `Merging' URL(s), for example all ending
275 with /ubuntu/ for usual Ubuntu mirror URLs. If redirection is not
276 used (i.e. the target URL list is empty) the original URL from
277 client's request is used to get the data.
278
279 Repository:
280
281 A (cache) repository is the internal identifier which declares the
282 scope in which `Merging/Redirection' specs are applied. It also
283 represents the name of an internal cache subdirectory.
284
285 4.3.1 Writing Remap-... configuration
286
287 When use cases for merging/redirection are identified and a
288 repository name is chosen, these components are written into
289 configuration directives starting with Remap- which follow the
290 simple syntax:
291
292 Remap-`RepositoryName': `MergingURLs' ; `TargetURLs' ;
293 `OptionalFlags'
294
295 The repository name is a symbolic name which should be chosen
296 carefully and should not be changed afterwards, otherwise the
297 data might become inaccessible for clients until the files are
298 extracted and reimported semi-manually. Internally, this string
299 shares the namespace with host names and/or top directory names
300 of other URLs. Name collisions can cause nasty side effects and
301 should be avoided. Recommended names are made up from alphanumeric
302 or URL-friendly characters. Also, a repository name should not be
303 associated to a real hostname. Examples for good names: `archlinux-
304 repo', `debianlocal'. Examples for bad names: `fedora.example.com',
305 `_very&weird'.
306
307 The `TargetURLs' part is optional (see `Redirection' description
308 above). If multiple targets are specified, the order of servers here
309 defines their order of preference (see also the `NetworkTimeout'
310 option and additional notes below).
311
312 Both URL lists simply contain URLs separated by spaces. The
313 strings must be properly URL-encoded. Since all URLs are assumed
314 to belong to http:// protocol and point to a remote directory, the
315 `http://' protocol prefix and trailing slashes are optional. There
316 is no hard limit to the number of URLs. However, for readability
317 reasons it's recommended to put them into separate list files (see
318 section 4.2) and specify the particular list files with tags like
319 `file:urlsDebian.list' instead of writing them into a single line.
320 Raw URLs and `file:...' lists can be mixed.
321
322 Fully configured Remap lines can look like:
323
324 Example I:
325
326 Remap-
327 debrep: ftp.de.debian.org/debian http://ftp.at.debian.org/debian
328
329 for the use case: small home network, clients have de... or at...
330 servers in their sources.list files and use acng as HTTP proxy. Now
331 the files are still downloaded from at... or de... mirrors depending
332 on the user request, but already cached data is served to both,
333 at... and de... users.
334
335 Example II:
336
337 Remap-
338 ubuntu: file:ubumir.lst ; 192.168.17.23/pu ca.archive.ubuntu.com/ubuntu
339
340 for the use case: small home network, clients have various Ubuntu
341 mirrors (which are listed in ubumir.lst) in their sources.list files
342 and use acng as HTTP proxy. All requests are redirected to a mirror
343 in the /pu directory of some local machine. When that machine is
344 down, Canadian public server is used instead.
345
346 4.3.2 Special tricks and additional notes
347
348 There are some implementation details (partially explained above)
349 and some configuration options related to repository settings which
350 should be mentioned explicitly.
351
352 The internal cache directory tree follows the URL requests from
353 the clients unless modified by Remapping rules. For proxy-
354 style configuration on the user side, it is always the hostname
355 of the requested URL. But if clients access the apt-cacher-
356 ng server like a regular mirror (not using APT's proxy config)
357 then it's just passed as regular directory name. And at this
358 point, it's possible to use Remapping constructs to access
359 random remote locations while the client assumes to download
360 from a subdirectory of apt-cacher-ng (as http server). This is
361 configured by simply using /some/directory/string/ instead of
362 URLs in the `Merging' list to let your clients download from
363 http://acngserver/some/directory/string/... paths.
364
365 If multiple Remap- lines for the same `Repository' are specified,
366 the contents of both URL lists are merged.
367
368 On some restricted networks, it may be needed to enforce the use
369 of predefined mirrors. If the `ForceManaged' option is set, only
370 requests to URL matched in some Remap-... config is allowed.
371
372 Sometimes, it may be needed to execute a system command before
373 connection to certain machines is established. This is possible by
374 associating commands with a repository declaration, i.e. by storing
375 a file named like `repositoryname'.hooks in the main configuration
376 directory. It can contain PreUp, Down and DownTimeout settings.
377 PreUp/Down are executed by the system shell and it's up to the
378 administrator to make sure that no malicious code is contained there
379 and that the execution of these commands does not cause significant
380 delays for other apt-cacher-ng users. See package documentation for
381 an exemplary hooks file.
382
383 If the Redirection part contains multiple URLs, the server prefers
384 to use them in the order of appearance. On success, the first
385 target is used all the time, and so this should be the preferred
386 mirror (note: "success" means getting a started download or a non-
387 critical failure in this context. A "404 File not found" status
388 is not considered critical since client's apt can expect and use
389 it to check the existence of remote files and then change its own
390 behaviour accordingly).
391
392 And finally, there is an optional third field in the `Remap'
393 directives which can contain extra flags to modify downloading
394 behavior in the scope of that particular cache repository.
395
396 - `keyfile=...' The meaning of this setting is: if any real
397 download error (status code 400 and higher) happens on a file
398 which path ends with the specified string then the target server
399 is blacklisted (considered faulty) immediately and this download
400 (and subsequent ones requested by this client connection) are
401 retried from other servers (see `TargetURLs' description above).
402 Can be used multiple times to define a list. See below for
403 documented example.
404
405 - `deltasrc=URL' Configures the base URL used to download
406 .debdelta files. The path hierarchy below this URL should
407 correspond to the source URLs and file paths in the cache. Only
408 one URL can be specified at the moment. It is used for explicit
409 mirroring operations, see section 8.14 for details.
410
411 - `proxy=proxyspec' Configures an alternative proxy URL which
412 overrides the global proxy setting in the context of this
413 repository. Can be set empty to disable proxy usage.
414
415 Config example:
416
417 Remap-debrep: file:deb_mirror*.gz ; file:backends_debian ;
418 keyfile=Release keyfile=.deb
419
420 If the first mirror from backends_debian goes wild and returns 404
421 responses for everything then the next candidate will be used.
422 However, while this feature can improve redundancy for certain
423 installations it needs to be used with care! Some file types are
424 allowed to be missing and apt interprets their absence to change its
425 behavior as needed. keyfile= should only match files which have an
426 essential role and which disappearance is undoubtful indication of a
427 broken server.
428
429 Chapter 5: Security
430 -------------------
431
432 Like many data storing daemons with predictable filenames, apt-
433 cacher-ng is vulnerable to symlink attacks and similar malicious
434 actions. Therefore, the user must make sure that the cache and log
435 directories are writable only to the user account under which apt-
436 cacher-ng is running.
437
438 General network access control can be established with operating
439 system's facilities in kernel space (see below) or in user space
440 (e.g. with inetd, see section 5.3). There is also experimental
441 built-in access filter feature which uses the TCP Wrapper library
442 for filtering (i.e. uses the same mechanism as tcpd). See section
443 5.2 for details.
444
445 As to the program's internal security, apt-cacher-ng has been
446 developed with concern about a certain grade of attacks from
447 internal users as well as from malicious external hosts. However,
448 no guarantees can be made about the security of the program. It is
449 recommended to run apt-cacher-ng under a dedicated system account
450 which has no write access to any files outside of the cache and log
451 directories. Refer to the manuals of the administration utilities
452 of the system distribution (like start-stop-daemon) to create the
453 required configuration.
454
455 If relaxed permissions are required, e.g. to make files group-
456 writeable, this can be established through the appropriate use
457 of umask command in the startup scripts of apt-cacher-ng (see
458 `/etc/default/apt-cacher-ng', for example) and the sticky bit on the
459 cache directories (see `chmod(1)' manpage for details). However,
460 write permissions should be assigned very carefully because that
461 may make the server vulnerable to symlink attacks and like-minded
462 activities.
463
464 The management web interface which allows execution of certain
465 operations can be protected by HTTP credentials (username/password).
466 The trust levels for most actions depend on their purpose. Those
467 which should never be able to destroy important local files can be
468 triggered by anyone if no password is set. And on the other hand,
469 some operations are considered dangerous and can only be accessed
470 when the admin password is configured and was entered correctly.
471
472 5.1 Access control by IP interface
473
474 A simple control method for incoming connections is listening only
475 to network interfaces which are inside a secure perimeter, e.g.
476 belong to the local network. See section 8.10 for details on this
477 configuration parameter. The default setting is listening to all
478 interfaces.
479
480 5.2 Access control with libwrap
481
482 If the access control with operating system's ip filters is not
483 possible, the embedded filtering might be using instead. It is
484 turned on if the feature is included at build time and the `UseWrap'
485 option is set to non-zero. If `UseWrap' is not set at all, it might
486 be enabled implicitly if the hosts.allow or hosts.deny files have
487 rules for apt-cacher-ng.
488
489 For more information about host access list creation, refer to
490 hosts_access(5) manual page (or manual pages for hosts.allow and
491 hosts.deny).
492
493 The libwrap filtering method has a little drawback compared to
494 alternatives. The host checks are called in single-threaded context
495 and can add some lag for other incoming connections.
496
497 5.3 Access control with inetd
498
499 In some situations, access filtering by client IP or hostname might
500 be not supported directly or there are other reasons to use inetd
501 to wrap access to apt-cacher inetd. For this case, an inetd daemon
502 is shipped with the package which makes the use of tcpd possible.
503 Installation is done in following steps:
504
505 1. compile the inetd bridge tool "in.acng", if not already done
506 (check `/usr/lib/apt-cacher-ng').
507
508 2. Edit apt-cacher-ng's configuration (acng.conf, for example), and
509 set a path for a new file in a writable directory, like this:
510
511 `SocketPath:/var/run/apt-cacher-ng/socket'
512
513 3. Edit /etc/inetd.conf and add following line with appropriate path
514 names and TCP port:
515
516 3143 stream tcp nowait user /usr/sbin/tcpd
517 /usr/local/sbin/in.acng /var/run/apt-cacher-ng/socket
518
519 4. Edit hosts.allow and other files to configure ACLs for port 3143.
520 See tcpd(8) and related manpages for further details.
521
522 5. Configure clients to use the alternative port (3143 in the
523 example above).
524
525 5.4 Access control with iptables
526
527 Looking at performance, the most efficient way to estables access
528 control by source IP is the use of system's native mechanisms. For
529 the Linux kernel, it can be created with iptables rules like in the
530 following example, usually stored in system's firewall setup script
531 (or any init script like `/etc/rc.local'):
532
533 iptables -A INPUT -p tcp --dport 3142 --source 127.0.0.0/8 -j ACCEPT
534 iptables -A INPUT -p tcp --dport 3142 --source 192.168.0.0/16 -j ACCEPT
535 iptables -A INPUT -p tcp --dport 3142 -j REJECT --reject-with tcp-reset
536
537 5.5 Target port filter
538
539 In the default configuration, apt-cacher-ng restricts the choice
540 of the target port on the remote mirror to the default HTTP port
541 (80). This is intended to prevent creation of HTTP tunnels and
542 similar malicious user activities.
543
544 Cache administrator can define an own list of allowed ports by
545 setting them as (space separated) list with the AllowUserPorts
546 configuration directive.
547
548 Alternatively, access to custom TCP ports can be established through
549 custom remapping specification.
550
551 Chapter 6: Distribution specific instructions
552 ---------------------------------------------
553
554 6.1 Debian and Ubuntu
555
556 Use as is. Report bugs using reportbug (Debian) or to Launchpad
557 (Ubuntu).
558
559 6.2 OpenSUSE
560
561 Server can be used as is with limited expiration (see below, and see
562 `INSTALL' file for compilation hints). The merging mode (multiple
563 servers mapped into the same repository) is not preconfigured in the
564 example configuration. This is object to research, competent support
565 is required.
566
567 Clients can configure apt-cacher-ng as central proxy in Yast
568 ("Network devices"/"Proxy" tab). If this is not desirable then each
569 sofware source can be edited to be redirected through the server.
570 This can be done in the Software Installer view of Yast2, see menu
571 Configuration/Repositories. To edit them quickly, switch to URL
572 editing mode and insert `host:port/' (of the apt-cacher-ng server)
573 between `http://' and the source server name.
574
575 6.3 Fedora Core
576
577 Attempts to add apt-cacher-ng support ended up in pain and the
578 author lost any motivation in further research on this subject.
579
580 6.4 Arch Linux
581
582 Mostly usable. The mirror structure design has been identified by
583 trial-and-error and the regular expressions might need some tuning
584 by experts.
585
586 The installer seems to have no way to specify a dedicated proxy but
587 it's possible to edit the source URL and insert the `host:port/'
588 part into it. The pacman mirror list can be modified the same way.
589
590 Expiration code should work in the basic mode (index data is
591 extracted from *.db.tar.gz files). File checksum checking mode
592 might also work (untested). The example configuration contains a
593 preconfigured list of mirrors which can be rebuilt with the Makefile
594 if needed. The preferred backend server can be specified like with
595 other distributions (see above for details).
596
597 6.5 Sourceforge mirror network
598
599 Not a Linux distro but commonly used by those to download certain
600 files. Therefore most mirrors can get unified access cache sharing
601 the files in the same cache repository. Some .exe files there are
602 never expired.
603
604 6.6 Cygwin mirrors
605
606 While not being a pure Linux distro it's mostly GNU and has a nice
607 mirror setup. Proxy server can be used as is with limited expiration
608 (see below). Apt-cacher-ng can also be compiled and used on Windows
609 machines in Cygwin environment (see `INSTALL' file for details).
610
611 Clients need to specify the server as HTTP proxy in the setup.exe
612 wizard, only HTTP mirrors should be selected in the mirror list.
613
614 6.7 Limited expiration
615
616 The expiration code for Non-Debian/Ubuntu repositories is quite
617 limited due to lack of man-power or know-how. For some distros,
618 the support is limited to checks of the filename and no further
619 validation is supported. Therefore, the extra validation of path
620 location or file contents should NOT be turned on when running
621 expiration with data from that distros in the cache, because good
622 data may be deleted in this case.
623
624 Chapter 7: Maintenance
625 ----------------------
626
627 There are few optional tasks that need to be executed by the
628 administrator from time to time or during the initial configuration.
629
630 7.1 Cache cleanup
631
632 If a package is no longer downloadable by APT clients then its
633 files are also not referenced in any index file and can be
634 removed. This rule also applies to most volatile files from the
635 distribution metadata. For example, Debian's Release file references
636 some Packages and Sources files or Diff-Index file, and those do
637 reference most other non-volatile files (binary packages, source
638 packages, index diffs, ...).
639
640 7.1.1 Manual expiration
641
642 To run the cleanup action manually visit the report page in a
643 browser and trigger the `Expiration' operation there.
644
645 There are different flags configuring the parameters of this
646 tracking described below. Usually just the filename is sufficient
647 to consider a file in the cache as a valid (downloadable) file.
648 This is ok in most cases but sometimes leads to false positives,
649 i.e. when another repository in the cache refers to a file with the
650 same name but the reference to the original location is gone. On
651 the other hand there can be cases where the assignment to different
652 repositories happened by mistake and administrator would like to
653 merge repositories later on.
654
655 For most files the checksum values are also provided in the index
656 files and so the file contents can be validated as well. This
657 requires reading of the whole cache archive to generate local
658 checksums. It should also not be done when apt-cacher-ng is being
659 used (file locking is not used here).
660
661 Usually it's necessary to bring various index files
662 (Release,Sources,Packages,Index) in sync with the repository. This
663 is necessary because apt works around the whole file download by
664 fetching small patches for the original file, and this mode of
665 operation is not supported yet by apt-cacher-ng (and might still
666 be unreliable). When this synchronization fails, the index files
667 might be incomplete or obsolete or damaged, and they might no longer
668 contain references to some files in the cache. Abortion of the
669 cleanup process is advisable in this case.
670
671 There is also a precaution mechanism designed to prevent the
672 destruction of cache contents when some volatile index files have
673 been lost temporarily. The results of cache examination are stored
674 in a list with the date when the particular files became orphaned.
675 The removals are only executed after few days (configurable, see
676 configuration file) unless they are removed from this list in the
677 meantime.
678
679 Parameters of `Expiration':
680
681 Skip header checks
682
683 By default, header description file of every package is opened
684 and checked for bad data and for obvious inconsistencies (like
685 local file being larger than specified by server). Which means
686 opening reading a few kilobytes from disk for almost every file
687 in the cache, and slightly degrades performance of the process.
688 This option skips that basic checks.
689
690 Stop cleanup on errors during index update step
691
692 Index files update is done first, on errors the expiration will
693 be interrupted.
694
695 Validate by file name AND file directory
696
697 This option can be used to remove distribution stages. Example:
698 to remove "oldstable" one just needs to delete the "Release"
699 files in the cache and run Expiration with this option two
700 times. There are some issues with this mode operation, see above
701 for details.
702
703 Validate by file name AND file contents (through checksum)
704
705 Checking file size and contents where possible against the
706 metadata in the index files. Note: future calls of Expiration
707 process without this option will discard the results of this
708 check and forget about corrupted files. Therefore, an action
709 on this files needs to be done ASAP, like truncating them (see
710 below) or removing via the removal operation (using the checkbox
711 and the Delete button, see process output) or via the "Delete
712 all unreferenced files" operation on the main control page.
713
714 Force the download of index files
715
716 Sometimes it may be needed to redownload all index files,
717 explicitly replacing the cached versions. This flag enables this
718 behaviour.
719
720 Purge unreferenced files after scan
721
722 Avoid the use of the orphan list and delete files instead.
723 This option is dangerous and should not be used unless when
724 absolutely no mistakes/problems can happen. Instead, it's
725 possible to view the orphan list later and delete then (see
726 control web interface).
727
728 Truncate damaged files immediately
729
730 If a file has been identified as damaged, it will be truncated
731 (file size reset to 0). Setting this option is a good compromise
732 for debugging purposes compared to the simple deletion since it
733 will keep the header files on the disk, for further analysis of
734 the problem's cause.
735
736 More verbosity
737
738 Shows more information, e.g. each scanned file when used with
739 some of the other options. This might result in a very large
740 HTML page, making the watching HTML browser very slow.
741
742 In additional to the default scan run, there are some "Direct
743 Action" buttons in the Web frontend. It's possible to see the
744 temporary list of files that have been identified as orphaned
745 (unreferenced), and it's possible to delete all files from that list
746 immediately. To be used carefully!
747
748 7.1.2 Automated cache cleanup
749
750 A program called `acngtool' is shipped with the package. Among other
751 functionality, this tool effectively implements a HTTP client which
752 operates like a human who runs the expiration manually (see above).
753 If needed, it would read the admin credentials and unix socket file
754 path from the local configuration files. On Debian installations it
755 is called by the file `/etc/cron.daily/apt-cacher-ng' so it should
756 run automatically as a daily `cron' task. The results are usually
757 not reported unless an error occurs, in which case some hints are
758 written to the standard error output (i.e. sent in cron mails).
759
760 The operator script can take some options from the environment,
761 see below. The default operation mode is calling the expiration
762 operation with default parameters and with credentials from local
763 system's apt-cacher-ng installation. However, this can be changed
764 with `ACNGREQ' variable.
765
766 `ACNGIP=10.0.1.3'
767
768 The network address for remote connection may be guessed
769 incorrectly by the operator script. This variable can specify
770 an explicit target to connect to, e.g. the same IP as the one
771 used by the clients (unless this network connection is somehow
772 restricted in the local setup).
773
774 `HOSTNAME=localOrPublicName'
775
776 When an error occurs, the operator script most likely adds an
777 URL to be opened for further investigation. The host name of
778 in this URL can be customized, i.e. can be set to a public
779 domain name representing the server as accessible from the
780 administrator's machine.
781
782 `ACNGREQ=cgiparameters'
783
784 Override the auto-detected command parameters with a custom
785 query string. This is the part of a command URL from the
786 management interface after the ? sign.
787
788 7.1.3 Keeping latest versions of expired package files
789
790 Sometimes it makes sense to keep a couple of versions of (Debian)
791 packages even after they have been removed from remote source. It
792 is possible to set an exceptional rule for package files which
793 follow the naming and versioning scheme of .deb-packages. This extra
794 handling is configured by the `KeepExtraVersions' options which
795 tells how many of the top-latest versions shall be kept. The cache
796 system needs the dpkg program and sufficient CPU power (depending on
797 the option value).
798
799 7.2 Removal of distribution releases
800
801 Sometimes it's needed to remove all files from a distribution, i.e.
802 when a new release became Stable and older package files are still
803 lying around. In perfect conditions the reference tracking described
804 above should take care of it and remove them soon.
805
806 However, this solution will fail if the release files are still
807 available on the server AND apt-cacher-ng learned their real
808 location (i.e. the code name instead of not the release state name)
809 and so they are refreshed during regular expiration.
810
811 After all, if the old release is no longer used by local cache users
812 then the extra disk usage can become problematic. This problem will
813 go away after many months when the old release files are finally
814 deleted on the servers, then the package expiration will start
815 complaining for some days (the expiration delay) and only then the
816 finally unreferenced files will be removed.
817
818 To speed up this process, the local administrator can remove the
819 index files of the old distribution release from the archive. This
820 can be done by using the Expiration task in the administrator
821 portal. On the end of the cache analysis, a helper form can be
822 expanded where index file can be selected and the destruction of
823 those cache contents can be ordered.
824
825 Chapter 8: HOWTOs and FAQ
826 -------------------------
827
828 8.1 Package import
829
830 Already existing packages can be imported into apt-cacher-ng's cache
831 pool instead of downloading them. There are some restrictions:
832
833 1. Don't try to import incomplete files. They will be refused since
834 their contents cannot be checked against the archive metadata.
835
836 2. If possible, don't import symbolic links. Even if doing so,
837 they should not point to other files inside of the cache and
838 especially not to other files under the `_import' directory.
839
840 HOWTO:
841
842 1. Make sure that apt-cacher-ng has valid index files in the cache.
843 This is the tricky part. To get them right, a client needs to
844 download them through apt-cacher-ng once. Therefore:
845
846 1. Configure the server and one client before doing the import.
847 See above for instructions.
848
849 2. Run "apt-get update" on client(s) once to teach ACNG about
850 remote locations of (volatile) index files. In some cases
851 this is not sufficient. See the note on APT below for a
852 workaround.
853
854 2. Store copies of your .debs, .orig.tar.gz, ... somewhere in the
855 `_import' subdirectory in the cache, ie. in `/var/cache/apt-
856 cacher-ng/_import/'. The files may be links or symlinks, it does
857 not matter. When done, apt-cacher will move those files to its
858 own internal locations. Example:
859
860 cd /var/cache
861 mkdir apt-cacher-ng/_import
862 cp -laf apt-proxy apt-cacher /var/cache/apt-cacher-ng/_import
863 chown -R apt-cacher-ng:apt-cacher-ng apt-cacher-ng/_import
864
865 3. Visit the report page and trigger the import action there. Check
866 the results, look for (red) error messages.
867
868 4. Check the `_import' directory again. All files that could be
869 identified as referenced by archive metadata should no longer
870 be there if they have been successfully moved. If some files
871 have been left behind, check whether the client can use them,
872 i.e. with "apt-cache policy ..." and/or checking checksums
873 with md5sum/sha1sum tools. Probably they are no longer needed
874 by anyone and therefore apt-cacher-ng just left them behind.
875 If no, follow the instructions in 1 or do similar things for
876 your distribution and retry the import operation. Setting the
877 verbosity flag (see checkbox on the command-and-control page)
878 can also help to discover the reason for the refusal to import
879 the particular files.
880
881 NOTE: APT is pretty efficient on avoiding unneccessary downloads
882 which can make a proxy blind to some relevant files. ACNG makes some
883 attempts to guess the remote locations of missed (not downloaded)
884 files but these heuristics may fail, especially on non-Debian
885 systems. When some files are permanently ignored, check the process
886 output for messages about the update of Packages/Sources files. When
887 some relevant package sources are missing there, there is a brute-
888 force method for Debian/Ubuntu users to force their download to the
889 client side. To do that, run:
890
891 rm /var/cache/apt/*cache.bin
892 rm /var/lib/apt/lists/*Packages
893 rm /var/lib/apt/lists/*Sources
894
895 on the client to purge APT's internal cache, and then rerun "apt-get
896 update" there.
897
898 8.2 Access to SSL/TLS remotes (HTTPS)
899
900 It is possible to have encrypted access to remote sites via HTTPS
901 protocol with recent versions of apt-cacher-ng if the OpenSSL
902 support was enabled at compile time. However this leads certain side
903 effects and complications; due to the nature of the HTTPS connection
904 model, it is not possible to act as an intermediate server (e.g.
905 caching proxy) by the same rules as with HTTP:
906
907 - SSL ensures strict verification of remote host name. Therefore,
908 you cannot add an URL to sources.list and make them point to
909 "hostname of acng server": the download client would expect
910 the remote site to offer the certificate of "hostname of acng
911 server" but the certificate of the remote server would be used.
912
913 - This restriction also applies if the LAN admin somehow manages
914 to trick the client that it connects to the remote site but
915 the real connection is silently rerouted to the local server:
916 since the proxy in the middle cannot fake the real certificate
917 it would need to create a custom one but that manipulation
918 usually cannot go unnoticed by the client unless some local hack
919 disables or manipulates the certificate verification. All parts
920 of this should be considered a crude hack and won't be described
921 here in detail. In fact, apt-cacher-ng does not implement local
922 SSL server functionality (as of version 0.8.1), partly because
923 of these considerations.
924
925 - If the client assumes that it needs to access a HTTPS remote
926 server through a HTTP proxy, it usually goes for the HTTP
927 tunneling (see the Wikipedia article for details). This is
928 a well-known and widely-used method but it comes with some
929 disadvantages: the proxy has no control of the data flowing
930 through it so the proxy cannot cache the data in any useful way;
931 in environments with additional security restrictions, it's also
932 hard to identify malicious users that abuse the service for
933 illegitimate purposes.
934
935 Considering these difficulties, there are three (and a half) methods
936 to use SSL.
937
938 - First, the "half method" - not using the proxy at all,
939 configuring each client to not use the HTTP proxy for HTTPS
940 urls. This will obviously disable central caching and requires
941 the client has separated configuration options to set this. For
942 Debian based distros, this can be done by adding a configuration
943 like this: `Acquire::https::proxy "DIRECT";' to apt.conf or
944 one of the apt.conf.d files. See section 8.4 for further
945 information.
946
947 - The "backend configuration method": if the clients access the
948 remote sites through URLs remapped on the server, the cacher
949 admin can add https URLs to backend lists instead of http urls.
950 Data will be cached just like usual.
951
952 - The "laissez-faire method": in acng.conf (or related) configure
953 the `PassThroughPattern' option to contain a regex like `.*'
954 and configure the clients to use apt-cacher-ng as HTTP proxy
955 and let the clients connect to https URLs "as usual". Some
956 limited access control can be achieved through adjustment of the
957 regular expression (.* permits access to any host and any port,
958 including 443 for https). Data is not cached on the server.
959
960 - The "tell-me-what-you-need method": on the client side, modify
961 the access URLs and change https to http and put the string
962 "HTTPS///" between http:// and the host name. With this trick,
963 the user client will access the proxy like going for a usual
964 HTTP download and the proxy will access the remote URL with
965 the https protocol. Caching (and file merging to repositories)
966 will work and there is still enough flexibility for the users.
967 The disadvantages of this method are basically the same as with
968 the access URLs rewriting (see Section 3.2) but the method is
969 still the preferred one by the apt-cacher-ng author. For apt's
970 sources.list, the modification may look like the following
971 example.
972
973 deb http://HTTPS///get.docker.com/ubuntu docker main
974 # If apt-cacher-ng is configured as proxy in APT, this makes it
975 # switch internally to https://get.docker.com/ubuntu
976 deb http://acnghost:3142/HTTPS///get.docker.com/ubuntu docker main
977 # Basically the same just with access to apt-cacher-ng through
978 # URL rewritting instead of setting http proxy.
979
980 8.3 JIGDO usage
981
982 It's possible to use apt-cacher-ng source with the jigdo-lite
983 utility. There are some limitations, though:
984
985 - since many mirrors do not distribute the jigdo files (or even
986 nothing from cdimage.debian.org at all), there is a high chance
987 to be redirected to a such mirror when using the backend-mapped
988 configuration. I.e. when user follows the official documentation
989 and edits wgetOpts in the jigdo configuration, it will fail in
990 many cases.
991
992 - apt-cacher-ng does not support .template files properly. They
993 might be cached but will be expired (removed from cache), sooner
994 or later.
995
996 But it's possible to feed jigdo-lite with the package contents from
997 your mirror. To do that, first start jigdo-lite as usual, something
998 like:
999
1000 `jigdo-lite http://cdimage.debian.org/.../...-DVD-1.jigdo'
1001
1002 When asked about Debian mirror, enter something like:
1003
1004 `http://proxy.host:3142/ftp.de.debian.org/debian/'
1005
1006 i.e. construct the same URL as present in usual apt-cacher-ng's
1007 user's sources.list.
1008
1009 That's all, jigdo-lite will fetch the package files using apt-
1010 cacher-ng proxy.
1011
1012 8.4 Avoid use of apt-cacher-ng for certain hosts
1013
1014 Sometimes clients might need to access some remote side directly
1015 to do some non-file-transfer oriented work but still passing the
1016 data through configured apt-cacher-ng proxy. Such remote hosts
1017 can be marked for direct access in apt configuration, e.g. in
1018 `/etc/apt/apt.conf':
1019
1020 Acquire::HTTP::Proxy::archive.example.org "DIRECT";
1021 //or Acquire::HTTP::Proxy::archive.example.org "other.proxy:port"
1022
1023 8.5 Avoid caching for certain domains or certain file types
1024
1025 Sometimes clients to download through apt-cacher-ng but the data
1026 shall not be stored on the harddisk of the server. To get it, use
1027 the DontCache directive (see examples for details) to define such
1028 files.
1029
1030 8.6 How to make big download series faster
1031
1032 Symptom: A common situation is a periodic download of hundreds of
1033 files through apt-cacher-ng where just a half is present in the
1034 cache. Although caching works fine, there are visible delays on some
1035 files during the download.
1036
1037 Possible cause and relief: the download from the real mirror gets
1038 interrupted while apt-cacher-ng delivers a set of files from the
1039 internal cache. While the connection is suspended, it times out and
1040 needs to be recreated when a miss occurs, i.e. apt-cacher-ng has to
1041 fetch more from the remote mirror. A workaround to this behaviour
1042 is simple, provided that the remote mirror can handle long request
1043 queues: set the pipelining depth to a very high value in apt.conf
1044 file or one of its replacement files in /etc/apt/apt.conf.d/. With
1045 something like:
1046
1047 `Acquire::http { Pipeline-Depth "200"; } '
1048
1049 there is a higher chance to get the server connection "preheated"
1050 before a stall occurs.
1051
1052 8.7 How to import DVDs or ISO images
1053
1054 First, it should be clear what is needed to be done. In order to
1055 integrate the packages from a DVD or ISO image, read on in section
1056 8.8.
1057
1058 The situation with ISO files import is complicated. They are not
1059 supported by the cache and there is also no expiration mode for
1060 them. The feature might be considered for addition in some future
1061 release of apt-cacher-ng.
1062
1063 What is possible now is publishing a directory with ISO files using
1064 its web server mode, see `LocalDirs' config option for details.
1065
1066 8.8 How to integrate DVDs or ISO image data
1067
1068 Integrating package files from DVD or ISO images is not much
1069 different to the usual import operation, see above for instructions.
1070
1071 One possible way to get files into the `_import' directory is simply
1072 mounting it there:
1073
1074 mount -o loop /dev/cdrom /var/cache/apt-cacher-ng/_import
1075
1076 After running the import operation, the disk can be umounted and
1077 removed.
1078
1079 A possible variation is import via symlinks. This can make sense
1080 where the space consumption must be reduced and the ISO image should
1081 stay on the server for a long time. To achive this, the image should
1082 be mounted at some mount point outside of the `_import' directory;
1083 the mounted state should be made permanent, maybe via an /etc/fstab
1084 entry with the loop option; then a symbolic link tree pointing to
1085 the mountpoint location should be created in the `_import' directory
1086 (something like `cp -as /mnt/image_jessie_01/pool /var/cache/apt-
1087 cacher-ng/_import/'). The subsequent "import" operation should pick
1088 up the symlinks and continue using them as links istead of file
1089 copies.
1090
1091 8.9 How to execute commands before and after going online?
1092
1093 It is possible to configure custom commands which are executed
1094 before the internet connection attempt and after a certain period
1095 after closing the connection. The commands are bound to a remapping
1096 configuration and the config file is named after the name of that
1097 remapping config, like `debrep.hooks' for `Remap-debrep'. See
1098 section 4.3.2, `conf/*.hooks' and `/usr/share/doc/apt-cacher-
1099 ng/examples/*.hooks' files for details.
1100
1101 8.10 Listen to only specific interfaces or IP protocols
1102
1103 Unless configured explicitely, the server listens to any interface
1104 with IPv4 or IPv6 protocol. To disable some of this, use the
1105 `BindAddress' option. It should contain a list of IP adresseses
1106 associated with particular network interfaces, separated by space.
1107 When option is set then the server won't listen to addresses or
1108 protocols not included there.
1109
1110 To limit to specific IP protocol, the address should only be present
1111 in the protocol specific syntax (like 192.0.43.10) will limit the
1112 use to the specific protocol.
1113
1114 The usual wildcard addresses can also be used to match all
1115 interfaces configured for the specific protocol, like 0.0.0.0 for
1116 IPv4.
1117
1118 8.11 How to avoid use of IPv4 (or IPv6) where possible?
1119
1120 Usually, outgoing hosts are accessed by the protocol and with the
1121 target IP reported as the first candidate by operating system
1122 facilities (getaddrinfo). It is possible to change this behavior,
1123 i.e. to skip IPv6 or IPv4 versions or try IPv6 connection first
1124 and then use IPv4 as alternative (or vice versa). See option
1125 ConnectProto in configuration examples.
1126
1127 8.12 Use the proxy without storing all data twice
1128
1129 There is a general use case where the data storing behavior of APT
1130 is not so fortunate. Imagine an old laptop with a slow and small
1131 harddisk but a modern network connection (i.e. Cardbus-attached WLAN
1132 card). But there is not enough space for APT to store the downloaded
1133 packages on the local disk, or not enough to perform the upgrade
1134 afterwards.
1135
1136 A plausible workaround in this case are moving contents of
1137 /var/cache/apt/archives directory to a mounted NFS share and
1138 replacing the original directory with a symlink (or bind-mount to
1139 the mentioned share). However, this solution would transfer all data
1140 at least three times over network. Another plausible workaround
1141 might be the use of curlftpfs which would embedd a remote FTP share
1142 which then can be specified as file:// URL in sources.list. However,
1143 this solution won't work with a local HTTP proxy like apt-cacher-
1144 ng (and httpfs http://sourceforge.net/projects/httpfs/ is not an
1145 alternative because it works only with a single file per mount).
1146
1147 As real alternative, apt-cacher-ng comes with an own implementation
1148 of a http file system called `acngfs'. It makes some assumptions of
1149 proxy's behaviour in order to emulate a real directory structure.
1150 Directories can be entered but not browsed (i.e. content listing
1151 is disallowed because of HTTP protocol limitations). Anyhow, this
1152 solution is good enough for APT. When it's checking the contents of
1153 the data source located on acngfs share, it reads the file contents
1154 of just the files required for the update which makes the apt-
1155 cacher-ng server download them on-the-fly.
1156
1157 And finally, angfs usage can be optimized for local access. This
1158 works best if the proxy daemons runs on the same machine as acngfs
1159 and there are hundreds of packages to update while filesystem access
1160 costs are negligible. Here the cache directory can be specified in
1161 acngfs parameters, and then it gets files directly from the cache if
1162 they are completely downloaded and don't have volatile contents.
1163
1164 8.13 Optional semi-automatic use of proxy
1165
1166 Apt-Cacher NG daemon has some optional operations modes regading the
1167 use of external proxy (configured with "Proxy" setting). The default
1168 mode means use of that proxy for all remote connections.
1169
1170 However, in some environments like when running on a portable
1171 system, the "upstream" proxy server may not reachable when running
1172 outside of home LAN area. For this situation, there are automated
1173 ways to detect it and to switch to direct internet access without
1174 additional configuration work.
1175
1176 One way is to use a (short) timeout value (OptProxyTimeout setting)
1177 which simply makes a failed connection attempt after the timeout
1178 being interpret as broken proxy. Then the proxy is not used
1179 in context of this connection until a new connection is to be
1180 established. If OptProxyCheckInterval is set (see below) then this
1181 change is effective for the configured time span.
1182
1183 Another way is through a custom (shell) command - when it returns
1184 successfully, the proxy is used, otherwise not and the command
1185 will be rerun only after a specified period. The command may
1186 check IP routes or a specific router address in the MAC pool
1187 or similar traces of the current environment. This setting
1188 (OptProxyCheckCommand) is also affected by another option
1189 (OptProxyCheckInterval) that specifies how often this check command
1190 shall be rerun. In the meantime, the configured proxy is considered
1191 faulty.
1192
1193 Hint: this special modes can be combined with non-caching behavior
1194 (see above) and another ACNG proxy in the home LAN to get smart
1195 caching for laptop users.
1196
1197 8.14 Partial Mirroring
1198
1199 It is possible to create a partial local mirror of a remote package
1200 repository. The method to do this is usually known as pre-caching. A
1201 such mirror would contain all files available to apt through `apt-
1202 cacher-ng', making the cache server suitable for pure off-line use.
1203
1204 The config uses index files in the local cache in order to declare
1205 which remote files shall be mirrored. Choice of relevant files
1206 decides which branch, which architecture or which source tree is to
1207 be mirrored. For convenience, it's possible to use glob expressions
1208 to create semi-dynamic list. The format is shell-like and relative
1209 to cache directory, a shell running in the cache directory can be
1210 helpful to verify the correctness.
1211
1212 Example:
1213
1214 PrecacheFor: debrep/dists/unstable/*/binary-amd64/Packages*
1215
1216 PrecacheFor: emacs.naquadah.org/unstable/*
1217
1218 Assuming that debrep repository is configured with proper remapping
1219 setup (see above), this would download all Debian packages listed
1220 for amd64 architecture in the unstable branch.
1221
1222 There is also support for faster file update using deltas, see
1223 Debdelta for details. The delta_uri URL mentioned there needs to be
1224 added as deltasrc option, see section 4.3.2 for details.
1225
1226 The operation is triggered using the web interface, various options
1227 or estimation mode can also be configured there. The CGI URL
1228 generated by the browser can be called with other clients to repeat
1229 this job, for example in a daily executed script. For regular tools
1230 like wget or curl, remember the need of quotation and secrecy of
1231 user/password data - command calls might expose them to local users.
1232 Alternatively, the tool `acngtool' (see above) can be used in order
1233 to run such commands even over a secured socket which prevents
1234 potential exposure of the credentials.
1235
1236 Chapter 9: Troubleshooting
1237 --------------------------
1238
1239 9.1 Debugging
1240
1241 Preliminary meanings of Debug option settings are:
1242
1243 - 0: No debug printing
1244
1245 - 1: Log file buffers are flushed faster
1246
1247 - 2: Some additional information appears within usual
1248 transfer/error logs
1249
1250 - 4: extra debug information is written to apt-cacher.err (also
1251 enables lots of additional trace points when apt-cacher-ng
1252 binary is built with debug configuration, see section 9.6 for
1253 details)
1254
1255 To combine that settings, add them (i.e. 7 enables all messages and
1256 log flushing)
1257
1258 Getting HTTP headers from apt-get works like this:
1259
1260 apt-get update -o Debug::Acquire::Http=true
1261
1262 9.2 Problem: keeps delivering damaged files
1263
1264 Even in this millennium, sometimes damaged files are downloaded from
1265 the server and are stored in the cache. Sometimes lazy maintainers
1266 of 3rd party archives replace package files with the same name
1267 but different contents. Sometimes the server's file system gets
1268 corrupted without detection by the OS.
1269
1270 Anyhow, there might be cases where cached data becomes invalid.
1271 Volatile files might be replaced by fixed version on some future
1272 download but static package files are never changed upon completion
1273 and even incomplete downloads are resumed and keep bad data
1274 downloaded before.
1275
1276 Usually the damage is only discovered by the client later. The
1277 particular file can be located in the cache and replaced manually.
1278 And if there are many of them, a mass file check might be needed to
1279 clean the mess. Fortunately, there are helpers in cache maintenance
1280 interface to automate this process.
1281
1282 To start, visit the web control interface and check the options of
1283 Expiration task. Enable the check for explicit paths and the check
1284 of data contents, then start the expiration. With this parameters,
1285 complete files with incorrect checksum are detected. The default
1286 action for such files is adding them to a list of damage files.
1287 After that, the "Delete damaged files" button in the main web page
1288 can be used to remove them (or the Show button to display them
1289 first). Alternatively, the checkboxes appearing aside of each damage
1290 detection can be used together with the control buttons which appear
1291 at the end of the report. And another way of dealing with them is
1292 truncating (setting to zero size). This can be done on-the-fly and
1293 is enabled by the expiration parameters, or with the appropriate
1294 command button in the web interface.
1295
1296 NOTE: several index files and related support files can create false
1297 positives, i.e. as incomplete or bad files. This usually happens
1298 because their volatile contents has changed but the file was not
1299 downloaded for a while and another version of it was used instead
1300 (like bzip2-compressed instead of gzip-compressed or uncompressed).
1301 The default code attempts to detect files with good reasons to stay
1302 in the cache and does not mark them as damaged.
1303
1304 9.3 Problem: regular expiration action reproducibly aborts
1305
1306 A quick investigation of action logs should help identifying the
1307 problem. A typical one is a mirror listed somewhere which is not
1308 reachable when expiration runs.
1309
1310 Unfortunately there is no simple and safe way to solve this. One
1311 method is setting the ExAbortOnProblems configuration variable, but
1312 this can destroy the whole cache if a bigger problem with index
1313 file occurs and this state remains unnoticed for many days until
1314 ExTreshold period (see configuration) is over.
1315
1316 Another way is listing the index files of the faulty mirrors to
1317 a special file. It needs to be stored as "ignore_list" in the
1318 configuration directory and store one path name per line with paths
1319 relative to the cache directory, as seen in the error messages.
1320
1321 9.4 Problem: cacher suddenly terminates, log reports IO errors
1322
1323 For simplicity and memory saving reasons, apt-cacher-ng assumes that
1324 some files can be opened exclusively for reading and they don't
1325 suddenly become unreadable. Unfortunatelly, in some conditions and
1326 in case of IO errors, this doesn't work and the file systems sends a
1327 fatal signal and eventually terminates the program.
1328
1329 To track this down to the likely reasons, it's possible to execute a
1330 custom script in the moment of the reception of the fatal signal. To
1331 do that, add something like this to the configuration:
1332
1333 BusAction = ls -l /proc/$PPID/ | mail -s SIGBUS! root
1334
1335 This command would send a list of opened files with paths to the
1336 "root" mail user, and the root shall check the state of those files
1337 (maybe run them through md5sum or similar).
1338
1339 NOTE: Beware of the security implications of this configuration
1340 option. It runs regular shell code in context of the daemon user in
1341 a blocking way, so it may be vulnerable to symlink attacks and it
1342 will delay the automatic restart of the daemon through systemd.
1343
1344 9.5 Problem: download fails with 503 ... status message
1345
1346 Code 503 usually represents an internal failure which could not be
1347 described correctly by other HTTP status codes. In the most cases
1348 it's caused by file system errors or incorrect cache directory
1349 setup, like files or directories with incorrect owner, missing
1350 write/read permissions for the effective user account or other
1351 system related exceptions like running out of disk space.
1352
1353 The log file `apt-cacher.err' located in the `LogDir' directory
1354 should document more details. In case it doesn't, setting the
1355 `Debug' config option to a higher value might reveal more
1356 information.
1357
1358 Fixing permission problems shouldn't be a real challenge for system
1359 administrators. Usually, a command set like this should do the trick
1360 on Debian/Ubuntu systems, assuming that all group users should
1361 receive write access to the cache files:
1362
1363 chown -R apt-cacher-ng:apt-cacher-ng /var/cache/apt-cacher-ng
1364 chmod -R a+rX,g+rw,u+rw /var/cache/apt-cacher-ng
1365
1366 9.6 Problem: `apt-get' freezes when downloading files
1367
1368 Solution: First, check:
1369
1370 - Free disk space and inode usage (`df', `df -i')
1371
1372 - Internet connection to the remote sites (browse them via HTTP,
1373 e.g. visiting http://ftp.your.mirror)
1374
1375 If nothing helps then you may have hit a spooky problem which
1376 is hard to track down. If you like, help the author on problem
1377 identification. To do that, do:
1378
1379 su -
1380 # enter root password
1381 cd /tmp
1382 apt-get source apt-cacher-ng
1383 apt-get build-dep apt-cacher-ng
1384 cd apt-cacher-ng-*
1385 ./build.sh DEBUG
1386 /etc/init.d/apt-cacher-ng stop
1387 builddir/apt-cacher-ng -c /etc/apt-cacher-ng logdir=/tmp foreground=1 debug=7
1388 # (let apt-get run now, on timeouts just wait >> 20 seconds)
1389 # stop the daemon with Ctrl-C
1390 /etc/init.d/apt-cacher-ng start
1391 # compress /tmp/apt-cacher.err and send it to author
1392 chown -R apt-cacher-ng:apt-cacher-ng /var/cache/apt-cacher-ng
1393
1394 The value of debug can be varied to have different verbosity (see
1395 section 9.1 for more information about Debug levels).
1396
1397 9.7 `apt-get' reports corrupted bzip2 data
1398
1399 Symptoms: apt-get fails to run through "update" no matter what you
1400 do. And you may have get a message like this one.
1401
1402 99% [6 Packages bzip2 0] [Waiting for headers] [Waiting for headers]
1403 bzip2: Data integrity error when decompressing.
1404 Input file = (stdin), output file = (stdout)
1405
1406 It is possible that the compressed file(s) have become corrupted.
1407 You can use the -tvv option to test integrity of such files.
1408
1409 You can use the `bzip2recover' program to attempt to recover
1410 data from undamaged sections of corrupted files.
1411
1412 Err http://debian.netcologne.de unstable/main Packages
1413 Sub-process bzip2 returned an error code (2)
1414
1415 - This might be one of Apt's problem with insufficient handling
1416 of errors, i.e. passing incomplete files to bzip2 on premature
1417 connection termination. Retry the update and it might work.
1418
1419 - Another issue is more severe: old versions of apt-cacher-ng had
1420 a bug which could cause data corruption while resuming downloads
1421 however this problem appears only in unusual conditions. To
1422 make sure there are no broken files in your repository, run the
1423 Expiration task with content verification enabled, and also
1424 "immediate deletion" (or delete later after checking the list).
1425 See section 7.1.1 for details.
1426
1427 9.8 Problem: `apt-cacher-ng' refuses to start with "Address already in
1428 use"
1429
1430 Another service is already listening on the port which apt-cacher-ng
1431 is configured to use. This might be the apt-cacher daemon which used
1432 the same port number by default. To identify the daemon behind that
1433 process, use the fuser utility, executing it as root for IPv4 and
1434 IPv6 protocol versions. Example:
1435
1436 fuser -4 -v -n tcp 3142
1437 fuser -6 -v -n tcp 3142
1438 USER PID ACCESS COMMAND
1439 3142/tcp: xwwwfsd 17914 F.... xwwwfsd
1440
1441 (where 3142 is the port number from the apt-cacher-ng configuration
1442 file). To resolve the collision, reconfigure the other daemon or
1443 apt-cacher-ng to use another free port (and reconfigure the clients
1444 to use the new apt-cacher-ng port accordingly).
1445
1446 Chapter 10: Known Bugs and Limitations
1447 --------------------------------------
1448
1449 - Only the "Basic" type of HTTP proxy authentication is supported
1450 at the moment
1451
1452 - Only HTTP HEAD and GET commands are supported properly.
1453
1454 - CONNECT (for https/SSL/TLS tunnels) is supported but needs to be
1455 enabled by local administrators, see section 8.2 for details
1456
1457 - POST support is restricted to apt-listbugs actions and is
1458 actually implemented as simple data forwarding with minor header
1459 manipulation. If additional proxy is configured, the client will
1460 be redirected to the HTTPS version of bugs.debian.org service,
1461 i.e. switching to the CONNECT method (see above)
1462
1463 - Transparent proxy mode not implemented yet
1464
1465 - The metadata update in maintenance jobs (like expiration task)
1466 attempts to guess the best source of update based on original
1467 source information and current backend configuration. In case
1468 of repeating failures, please run a "forced" download of index
1469 files through the web interface
1470
1471 - See TODO file in apt-cacher-ng source for more related
1472 information
1473
1474 Chapter 11: Contact
1475 -------------------
1476
1477 The planned features are listed in the file TODO. Don't hesitate to
1478 contact the author if you really need something not found there and
1479 you can explain the severity of your request.
1480
1481 There are also a public mailing list and request trackers available
1482 on the Alioth project page.
1483
1484 And last, but not least: feel free to express your gratitude by
1485 donating a small amount of money via PayPal (to [email protected]).
1486
1487 [Eduard Bloch, Sun, 19 Apr 2015 10:25:49 +0200]