| 1 |
Apt-Cacher-NG User Manual |
| 2 |
========================= |
| 3 |
|
| 4 |
Apt-Cacher NG is a caching proxy for software packages which are downloaded |
| 5 |
by Unix/Linux system distribution mechanisms from mirror servers accessible |
| 6 |
via HTTP. |
| 7 |
|
| 8 |
This manual provides an overview of Apt-Cacher-NG's features and a walk |
| 9 |
through the required configuration steps for server administrators and |
| 10 |
users of the proxy. |
| 11 |
|
| 12 |
Chapter 1: Introduction |
| 13 |
----------------------- |
| 14 |
|
| 15 |
apt-cacher-ng attempts to achieve the same goals as related proxies |
| 16 |
- it acts as a proxy which is used by clients in the local network |
| 17 |
to share the data that has been downloaded. It monitors the state of |
| 18 |
packages and is capable of merging downloads of the same packages |
| 19 |
from different locations (real or simulated). |
| 20 |
|
| 21 |
The program reuses many ideas behind the other famous proxy, its |
| 22 |
predecessor apt-cacher 1.x (which has been written in Perl). In |
| 23 |
contrast to apt-cacher, different aspects have been declared as |
| 24 |
primary targets during the development of apt-cacher-ng: |
| 25 |
|
| 26 |
- lightweight implementation - allow use on systems with low |
| 27 |
memory and processing resources |
| 28 |
|
| 29 |
- internal (native) threading - avoiding process fork'ing wherever |
| 30 |
possible, avoiding kludges for pseudo-thread synchronization, |
| 31 |
avoiding relying on special file system features for internal |
| 32 |
operations where possible |
| 33 |
|
| 34 |
- real (effective) support of HTTP pipelining, using an internal |
| 35 |
client with native stream control (having the nice side effect: |
| 36 |
reduction of resource overhead and minimization of possible |
| 37 |
points of failure) |
| 38 |
|
| 39 |
- avoiding featuritis where they cause too much bloat and the |
| 40 |
functionality can be provided by native OS features |
| 41 |
|
| 42 |
- reliable but efficient content merging in the local package |
| 43 |
pool, avoiding delivering of wrong data. |
| 44 |
|
| 45 |
As with apt-cacher, explicit tracking of dynamically changed |
| 46 |
and unchanged files is established, and the use in non-Debian |
| 47 |
environment is supported. |
| 48 |
|
| 49 |
Long story: Not all goals have been achieved. The initial plan of |
| 50 |
using background databases to merge any download from any arbitrary |
| 51 |
location has been dropped because of complexity and performance |
| 52 |
considerations, reliable heuristics could not be found either. |
| 53 |
Instead, a semi-automated solution has been created which used |
| 54 |
machine-parsable files with mirror information, like the one |
| 55 |
available for Debian mirrors in Debian's CVS repository. |
| 56 |
|
| 57 |
Chapter 2: Running apt-cacher-ng |
| 58 |
-------------------------------- |
| 59 |
|
| 60 |
Run "build/apt-cacher-ng -c conf" when configured where conf is the |
| 61 |
configuration directory. See section 4.2 for details on possible and |
| 62 |
required contents of this directory. |
| 63 |
|
| 64 |
Most options from the configuration file can also be passed through |
| 65 |
command line parameters. Just append them with the same format as in |
| 66 |
the configuration file but without separating spaces inside, e.g. |
| 67 |
|
| 68 |
Port:4855 ForeGround=1 |
| 69 |
|
| 70 |
For convenience, the colon can also be replaced with the equals sign |
| 71 |
and letter case does not matter, so this is also possible: |
| 72 |
|
| 73 |
port=4855 foreground:1. |
| 74 |
|
| 75 |
Chapter 3: Basic Configuration |
| 76 |
------------------------------ |
| 77 |
|
| 78 |
3.1 Server Configuration |
| 79 |
|
| 80 |
Unlike some rumors on the internet claim, there should be no need |
| 81 |
for exhausting configuration work to just test apt-cacher-ng and run |
| 82 |
it with default parameters. It's actually designed to bootstrap most |
| 83 |
of its working environment without additional help. |
| 84 |
|
| 85 |
The package setup scripts used by distributions should already |
| 86 |
prepare working initial settings for apt-cacher-ng. Check the |
| 87 |
file `/etc/apt-cacher-ng/acng.conf' file where most settings are |
| 88 |
explained. For the beginning they should not be changed, the only |
| 89 |
interesting setting present there is the TCP port. See Advanced |
| 90 |
Server Configuration for details. |
| 91 |
|
| 92 |
There is also a daily cron job which executes some maintenance work. |
| 93 |
Additional automated control commands can be added by administrator. |
| 94 |
|
| 95 |
3.2 Client Configuration |
| 96 |
|
| 97 |
From the client side, apt-cacher-ng can be used as a drop-in |
| 98 |
replacement for apt-cacher. The same rules apply, e.g. Debian/Ubuntu |
| 99 |
users should EITHER: |
| 100 |
|
| 101 |
- Specify the caching machine as HTTP Proxy for your download |
| 102 |
client. This can be usually done (temporarily) with the |
| 103 |
environment variable `http_proxy' or (permanently) in some |
| 104 |
configuration file or dialog. For APT, this means putting a line |
| 105 |
like the following into a file like /etc/apt/apt.conf.d/02proxy: |
| 106 |
|
| 107 |
Acquire::http::proxy "http://CacheServerIp:3142"; |
| 108 |
|
| 109 |
OR: |
| 110 |
|
| 111 |
- Replace all mirror hostnames with cachinghost/hostname in |
| 112 |
sources.list, so |
| 113 |
|
| 114 |
deb http://ftp.uni-kl.de/debian etch main |
| 115 |
|
| 116 |
should now become: |
| 117 |
|
| 118 |
deb http://192.168.0.17:3142/ftp.uni-kl.de/debian etch main |
| 119 |
|
| 120 |
(assuming that CacheServerIp is 192.168.0.17 and the service port |
| 121 |
is 3142) . |
| 122 |
|
| 123 |
These both methods have their own pros and cons. The last method can |
| 124 |
be used with clients which cannot configure an additional http proxy |
| 125 |
for some reason. The disadvantages are the worse progress visibility |
| 126 |
(all downloads seem to come from the same machine) and some resource |
| 127 |
usage limits might be hit (i.e. maximum number of simultaneous |
| 128 |
downloads from the "same" machine). It might also require to modify |
| 129 |
many different URLs and some of those URLs might be hardcoded in |
| 130 |
software and not be accessible to the user. |
| 131 |
|
| 132 |
The former method is more convenient since it usually means less |
| 133 |
configuration work; however, it implies that all relevant client |
| 134 |
programs respect the proxy setting and use it from this central |
| 135 |
location. |
| 136 |
|
| 137 |
Mixing the configuration methods is usually possible but not within |
| 138 |
the same client program instance. Doing that (going with proxy mode |
| 139 |
AND use rewritten URLs) will probably confuse the server: in best- |
| 140 |
case, the connection will be slower because of a little transport |
| 141 |
loop on the server side, and in the worst-case, the target URL will |
| 142 |
eventually become not resolvable and the downloads will just fail. |
| 143 |
|
| 144 |
Using SSL/TLS transport (i.e. https urls) is also possible with some |
| 145 |
restrictions, see section 8.2 for details. |
| 146 |
|
| 147 |
Chapter 4: Advanced Server Configuration |
| 148 |
---------------------------------------- |
| 149 |
|
| 150 |
4.1 Vocabulary |
| 151 |
|
| 152 |
This chapter introduces some terminology which is needed to |
| 153 |
understand the functionality of apt-cacher-ng; it's recommended to |
| 154 |
understand it before continuing with the advanced configuration. |
| 155 |
|
| 156 |
- "Backend": a text file consisting of a list of mirror URLs, one |
| 157 |
per line (a more complex RFC822-like format is also supported). |
| 158 |
Used for URL remapping; see section 4.3. |
| 159 |
|
| 160 |
- "Volatile files": files with changing contents on the server |
| 161 |
while keeping the same URL (i.e. the same filename and |
| 162 |
location). For example, metadata pertaining to package files |
| 163 |
stored in a remote archive is classified as 'volatile'. They |
| 164 |
are usually 'index files' known as Packages, Sources, Release, |
| 165 |
Pdiff and the like, and maybe also associated extra data (like |
| 166 |
translations and thumbnail icons). |
| 167 |
|
| 168 |
- "Package files": files that contain software packages and other |
| 169 |
"solid" data: DEBs, source files for their creation (.tar.gz, |
| 170 |
.diff, .dsc), various metadata which is not subject to change |
| 171 |
after first appearance on the server. |
| 172 |
|
| 173 |
- "Configuration line": one single line in the configuration file. |
| 174 |
Some examples in this chapter may be displayed as wrapped lines |
| 175 |
but should be stored as a single line in the configuration. Some |
| 176 |
variables (like `RequestAppendix') support multiline data which |
| 177 |
needs to be separated with \n delimiter. |
| 178 |
|
| 179 |
4.2 Configuration file types |
| 180 |
|
| 181 |
By default, the /etc/apt-cacher-ng directory (or the one specified |
| 182 |
with program options) contains all config files, HTML page |
| 183 |
templates, the stylesheet and other text-based support files used by |
| 184 |
apt-cacher-ng. The contents may vary depending on the installation |
| 185 |
of apt-cacher-ng, refer to the package documentation for Linux |
| 186 |
Distribution packages. |
| 187 |
|
| 188 |
There are a few certain file types distinguished by apt-cacher-ng: |
| 189 |
|
| 190 |
1. Main configuration files: |
| 191 |
|
| 192 |
*.conf files are assumed to contain configuration directives |
| 193 |
in the form of "key: value" pairs. The package comes with a |
| 194 |
commented example configuration file. apt-cacher-ng reads all |
| 195 |
files matching *.conf in alphabetical order and merges the |
| 196 |
contents. For options documentation, see commented example file |
| 197 |
shipped with apt-cacher-ng (conf/ directory in original source). |
| 198 |
|
| 199 |
For security reasons, the files can be made readable only to |
| 200 |
the daemon and administorator accounts, e.g. when they contain |
| 201 |
passwords or other sensitive data. |
| 202 |
|
| 203 |
2. URL lists and remote repository list files. The file names |
| 204 |
are arbitrary, no special suffix is required. They are read |
| 205 |
and included during processing of configuration files and can |
| 206 |
contain data in one of the following formats: |
| 207 |
|
| 208 |
- simple text files with one URL per line (the URL should |
| 209 |
point to the base directory of the repository, e.g. |
| 210 |
"http://ftp.de.debian.org/debian/"). A URL must start with |
| 211 |
http:// and should end with a slash |
| 212 |
|
| 213 |
- an RFC822-like format, with lines like 'Site: <hostname>' |
| 214 |
and 'Archive-http: /base/directory/of/repository/'. Optional |
| 215 |
fields are also used in this remapping descriptions to add |
| 216 |
more possible variants (Alias, Aliases, X-Archive-http:) of |
| 217 |
the URLs to the lookup list |
| 218 |
|
| 219 |
3. Various support files used for the configuration web interface, |
| 220 |
named like *.css and *.html. |
| 221 |
|
| 222 |
4. *.default files are used in some rare cases as replacement for |
| 223 |
list files having the same name without .default suffix. |
| 224 |
|
| 225 |
5. *.hooks files specify custom actions which can be executed upon |
| 226 |
connection/disconnection (see section 4.3.2 for details). |
| 227 |
|
| 228 |
Except from .conf files, most files listed above can be moved to |
| 229 |
another "support" directory and the daemon will look for them |
| 230 |
therein if they are not present in the primary configuration |
| 231 |
directory. This feature is intended to help keeping pure |
| 232 |
configuration data and mostly static data in different locations. |
| 233 |
The directory path is specified at build time and can be overriden |
| 234 |
with the SupportDir directive (and if used, this should be set as |
| 235 |
early as possible). |
| 236 |
|
| 237 |
4.3 Repositories and URL mapping |
| 238 |
|
| 239 |
With the most simple configuration, apt-cacher-ng will act almost |
| 240 |
like an ordinary HTTP proxy with improved caching behaviour. When |
| 241 |
files are requested, they are downloaded from a remote location |
| 242 |
specified in client's request and are stored in a unique way. |
| 243 |
|
| 244 |
However, for some use cases it can be beneficial to specify |
| 245 |
additional rules to achieve further improvements, e.g. in order to |
| 246 |
detect and prevent avoidable downloads, to reduce space requirements |
| 247 |
for the cache directory or simply hide real download locations from |
| 248 |
the APT clients. |
| 249 |
|
| 250 |
These modifications are generally achieved by two strategies, |
| 251 |
`Merging' and `Redirection', which are configured in a context of a |
| 252 |
specified cache `Repository'. The configuration for them is created |
| 253 |
using one or multiple Remap-... configuration directives (see |
| 254 |
below). |
| 255 |
|
| 256 |
Merging: |
| 257 |
|
| 258 |
"Merging" of incoming requests can be done if some subdirectories of |
| 259 |
different remote servers are considered equal where the last part of |
| 260 |
the remote file path leads to the same file content. When specified, |
| 261 |
the internal cache content is shared and the live download stream |
| 262 |
is shared. The configuration work consists of setting an "equality |
| 263 |
list" containing a set of URLs representing the base directories |
| 264 |
(like `http://ftp.debian.org/debian' and `http://ftp.uni- |
| 265 |
kl.de/pub/linux/debian'). |
| 266 |
|
| 267 |
Redirection: |
| 268 |
|
| 269 |
With redirection, client requests cause a download from a remote |
| 270 |
location which is different from what clients requested and believe |
| 271 |
to receive from. Redirection is an optional feature; if used, it's |
| 272 |
configured by one or multiple URL(s) pointing to target servers. The |
| 273 |
URL(s) must include a directory spec which matches the directory |
| 274 |
level of the URLs in the `Merging' URL(s), for example all ending |
| 275 |
with /ubuntu/ for usual Ubuntu mirror URLs. If redirection is not |
| 276 |
used (i.e. the target URL list is empty) the original URL from |
| 277 |
client's request is used to get the data. |
| 278 |
|
| 279 |
Repository: |
| 280 |
|
| 281 |
A (cache) repository is the internal identifier which declares the |
| 282 |
scope in which `Merging/Redirection' specs are applied. It also |
| 283 |
represents the name of an internal cache subdirectory. |
| 284 |
|
| 285 |
4.3.1 Writing Remap-... configuration |
| 286 |
|
| 287 |
When use cases for merging/redirection are identified and a |
| 288 |
repository name is chosen, these components are written into |
| 289 |
configuration directives starting with Remap- which follow the |
| 290 |
simple syntax: |
| 291 |
|
| 292 |
Remap-`RepositoryName': `MergingURLs' ; `TargetURLs' ; |
| 293 |
`OptionalFlags' |
| 294 |
|
| 295 |
The repository name is a symbolic name which should be chosen |
| 296 |
carefully and should not be changed afterwards, otherwise the |
| 297 |
data might become inaccessible for clients until the files are |
| 298 |
extracted and reimported semi-manually. Internally, this string |
| 299 |
shares the namespace with host names and/or top directory names |
| 300 |
of other URLs. Name collisions can cause nasty side effects and |
| 301 |
should be avoided. Recommended names are made up from alphanumeric |
| 302 |
or URL-friendly characters. Also, a repository name should not be |
| 303 |
associated to a real hostname. Examples for good names: `archlinux- |
| 304 |
repo', `debianlocal'. Examples for bad names: `fedora.example.com', |
| 305 |
`_very&weird'. |
| 306 |
|
| 307 |
The `TargetURLs' part is optional (see `Redirection' description |
| 308 |
above). If multiple targets are specified, the order of servers here |
| 309 |
defines their order of preference (see also the `NetworkTimeout' |
| 310 |
option and additional notes below). |
| 311 |
|
| 312 |
Both URL lists simply contain URLs separated by spaces. The |
| 313 |
strings must be properly URL-encoded. Since all URLs are assumed |
| 314 |
to belong to http:// protocol and point to a remote directory, the |
| 315 |
`http://' protocol prefix and trailing slashes are optional. There |
| 316 |
is no hard limit to the number of URLs. However, for readability |
| 317 |
reasons it's recommended to put them into separate list files (see |
| 318 |
section 4.2) and specify the particular list files with tags like |
| 319 |
`file:urlsDebian.list' instead of writing them into a single line. |
| 320 |
Raw URLs and `file:...' lists can be mixed. |
| 321 |
|
| 322 |
Fully configured Remap lines can look like: |
| 323 |
|
| 324 |
Example I: |
| 325 |
|
| 326 |
Remap- |
| 327 |
debrep: ftp.de.debian.org/debian http://ftp.at.debian.org/debian |
| 328 |
|
| 329 |
for the use case: small home network, clients have de... or at... |
| 330 |
servers in their sources.list files and use acng as HTTP proxy. Now |
| 331 |
the files are still downloaded from at... or de... mirrors depending |
| 332 |
on the user request, but already cached data is served to both, |
| 333 |
at... and de... users. |
| 334 |
|
| 335 |
Example II: |
| 336 |
|
| 337 |
Remap- |
| 338 |
ubuntu: file:ubumir.lst ; 192.168.17.23/pu ca.archive.ubuntu.com/ubuntu |
| 339 |
|
| 340 |
for the use case: small home network, clients have various Ubuntu |
| 341 |
mirrors (which are listed in ubumir.lst) in their sources.list files |
| 342 |
and use acng as HTTP proxy. All requests are redirected to a mirror |
| 343 |
in the /pu directory of some local machine. When that machine is |
| 344 |
down, Canadian public server is used instead. |
| 345 |
|
| 346 |
4.3.2 Special tricks and additional notes |
| 347 |
|
| 348 |
There are some implementation details (partially explained above) |
| 349 |
and some configuration options related to repository settings which |
| 350 |
should be mentioned explicitly. |
| 351 |
|
| 352 |
The internal cache directory tree follows the URL requests from |
| 353 |
the clients unless modified by Remapping rules. For proxy- |
| 354 |
style configuration on the user side, it is always the hostname |
| 355 |
of the requested URL. But if clients access the apt-cacher- |
| 356 |
ng server like a regular mirror (not using APT's proxy config) |
| 357 |
then it's just passed as regular directory name. And at this |
| 358 |
point, it's possible to use Remapping constructs to access |
| 359 |
random remote locations while the client assumes to download |
| 360 |
from a subdirectory of apt-cacher-ng (as http server). This is |
| 361 |
configured by simply using /some/directory/string/ instead of |
| 362 |
URLs in the `Merging' list to let your clients download from |
| 363 |
http://acngserver/some/directory/string/... paths. |
| 364 |
|
| 365 |
If multiple Remap- lines for the same `Repository' are specified, |
| 366 |
the contents of both URL lists are merged. |
| 367 |
|
| 368 |
On some restricted networks, it may be needed to enforce the use |
| 369 |
of predefined mirrors. If the `ForceManaged' option is set, only |
| 370 |
requests to URL matched in some Remap-... config is allowed. |
| 371 |
|
| 372 |
Sometimes, it may be needed to execute a system command before |
| 373 |
connection to certain machines is established. This is possible by |
| 374 |
associating commands with a repository declaration, i.e. by storing |
| 375 |
a file named like `repositoryname'.hooks in the main configuration |
| 376 |
directory. It can contain PreUp, Down and DownTimeout settings. |
| 377 |
PreUp/Down are executed by the system shell and it's up to the |
| 378 |
administrator to make sure that no malicious code is contained there |
| 379 |
and that the execution of these commands does not cause significant |
| 380 |
delays for other apt-cacher-ng users. See package documentation for |
| 381 |
an exemplary hooks file. |
| 382 |
|
| 383 |
If the Redirection part contains multiple URLs, the server prefers |
| 384 |
to use them in the order of appearance. On success, the first |
| 385 |
target is used all the time, and so this should be the preferred |
| 386 |
mirror (note: "success" means getting a started download or a non- |
| 387 |
critical failure in this context. A "404 File not found" status |
| 388 |
is not considered critical since client's apt can expect and use |
| 389 |
it to check the existence of remote files and then change its own |
| 390 |
behaviour accordingly). |
| 391 |
|
| 392 |
And finally, there is an optional third field in the `Remap' |
| 393 |
directives which can contain extra flags to modify downloading |
| 394 |
behavior in the scope of that particular cache repository. |
| 395 |
|
| 396 |
- `keyfile=...' The meaning of this setting is: if any real |
| 397 |
download error (status code 400 and higher) happens on a file |
| 398 |
which path ends with the specified string then the target server |
| 399 |
is blacklisted (considered faulty) immediately and this download |
| 400 |
(and subsequent ones requested by this client connection) are |
| 401 |
retried from other servers (see `TargetURLs' description above). |
| 402 |
Can be used multiple times to define a list. See below for |
| 403 |
documented example. |
| 404 |
|
| 405 |
- `deltasrc=URL' Configures the base URL used to download |
| 406 |
.debdelta files. The path hierarchy below this URL should |
| 407 |
correspond to the source URLs and file paths in the cache. Only |
| 408 |
one URL can be specified at the moment. It is used for explicit |
| 409 |
mirroring operations, see section 8.14 for details. |
| 410 |
|
| 411 |
- `proxy=proxyspec' Configures an alternative proxy URL which |
| 412 |
overrides the global proxy setting in the context of this |
| 413 |
repository. Can be set empty to disable proxy usage. |
| 414 |
|
| 415 |
Config example: |
| 416 |
|
| 417 |
Remap-debrep: file:deb_mirror*.gz ; file:backends_debian ; |
| 418 |
keyfile=Release keyfile=.deb |
| 419 |
|
| 420 |
If the first mirror from backends_debian goes wild and returns 404 |
| 421 |
responses for everything then the next candidate will be used. |
| 422 |
However, while this feature can improve redundancy for certain |
| 423 |
installations it needs to be used with care! Some file types are |
| 424 |
allowed to be missing and apt interprets their absence to change its |
| 425 |
behavior as needed. keyfile= should only match files which have an |
| 426 |
essential role and which disappearance is undoubtful indication of a |
| 427 |
broken server. |
| 428 |
|
| 429 |
Chapter 5: Security |
| 430 |
------------------- |
| 431 |
|
| 432 |
Like many data storing daemons with predictable filenames, apt- |
| 433 |
cacher-ng is vulnerable to symlink attacks and similar malicious |
| 434 |
actions. Therefore, the user must make sure that the cache and log |
| 435 |
directories are writable only to the user account under which apt- |
| 436 |
cacher-ng is running. |
| 437 |
|
| 438 |
General network access control can be established with operating |
| 439 |
system's facilities in kernel space (see below) or in user space |
| 440 |
(e.g. with inetd, see section 5.3). There is also experimental |
| 441 |
built-in access filter feature which uses the TCP Wrapper library |
| 442 |
for filtering (i.e. uses the same mechanism as tcpd). See section |
| 443 |
5.2 for details. |
| 444 |
|
| 445 |
As to the program's internal security, apt-cacher-ng has been |
| 446 |
developed with concern about a certain grade of attacks from |
| 447 |
internal users as well as from malicious external hosts. However, |
| 448 |
no guarantees can be made about the security of the program. It is |
| 449 |
recommended to run apt-cacher-ng under a dedicated system account |
| 450 |
which has no write access to any files outside of the cache and log |
| 451 |
directories. Refer to the manuals of the administration utilities |
| 452 |
of the system distribution (like start-stop-daemon) to create the |
| 453 |
required configuration. |
| 454 |
|
| 455 |
If relaxed permissions are required, e.g. to make files group- |
| 456 |
writeable, this can be established through the appropriate use |
| 457 |
of umask command in the startup scripts of apt-cacher-ng (see |
| 458 |
`/etc/default/apt-cacher-ng', for example) and the sticky bit on the |
| 459 |
cache directories (see `chmod(1)' manpage for details). However, |
| 460 |
write permissions should be assigned very carefully because that |
| 461 |
may make the server vulnerable to symlink attacks and like-minded |
| 462 |
activities. |
| 463 |
|
| 464 |
The management web interface which allows execution of certain |
| 465 |
operations can be protected by HTTP credentials (username/password). |
| 466 |
The trust levels for most actions depend on their purpose. Those |
| 467 |
which should never be able to destroy important local files can be |
| 468 |
triggered by anyone if no password is set. And on the other hand, |
| 469 |
some operations are considered dangerous and can only be accessed |
| 470 |
when the admin password is configured and was entered correctly. |
| 471 |
|
| 472 |
5.1 Access control by IP interface |
| 473 |
|
| 474 |
A simple control method for incoming connections is listening only |
| 475 |
to network interfaces which are inside a secure perimeter, e.g. |
| 476 |
belong to the local network. See section 8.10 for details on this |
| 477 |
configuration parameter. The default setting is listening to all |
| 478 |
interfaces. |
| 479 |
|
| 480 |
5.2 Access control with libwrap |
| 481 |
|
| 482 |
If the access control with operating system's ip filters is not |
| 483 |
possible, the embedded filtering might be using instead. It is |
| 484 |
turned on if the feature is included at build time and the `UseWrap' |
| 485 |
option is set to non-zero. If `UseWrap' is not set at all, it might |
| 486 |
be enabled implicitly if the hosts.allow or hosts.deny files have |
| 487 |
rules for apt-cacher-ng. |
| 488 |
|
| 489 |
For more information about host access list creation, refer to |
| 490 |
hosts_access(5) manual page (or manual pages for hosts.allow and |
| 491 |
hosts.deny). |
| 492 |
|
| 493 |
The libwrap filtering method has a little drawback compared to |
| 494 |
alternatives. The host checks are called in single-threaded context |
| 495 |
and can add some lag for other incoming connections. |
| 496 |
|
| 497 |
5.3 Access control with inetd |
| 498 |
|
| 499 |
In some situations, access filtering by client IP or hostname might |
| 500 |
be not supported directly or there are other reasons to use inetd |
| 501 |
to wrap access to apt-cacher inetd. For this case, an inetd daemon |
| 502 |
is shipped with the package which makes the use of tcpd possible. |
| 503 |
Installation is done in following steps: |
| 504 |
|
| 505 |
1. compile the inetd bridge tool "in.acng", if not already done |
| 506 |
(check `/usr/lib/apt-cacher-ng'). |
| 507 |
|
| 508 |
2. Edit apt-cacher-ng's configuration (acng.conf, for example), and |
| 509 |
set a path for a new file in a writable directory, like this: |
| 510 |
|
| 511 |
`SocketPath:/var/run/apt-cacher-ng/socket' |
| 512 |
|
| 513 |
3. Edit /etc/inetd.conf and add following line with appropriate path |
| 514 |
names and TCP port: |
| 515 |
|
| 516 |
3143 stream tcp nowait user /usr/sbin/tcpd |
| 517 |
/usr/local/sbin/in.acng /var/run/apt-cacher-ng/socket |
| 518 |
|
| 519 |
4. Edit hosts.allow and other files to configure ACLs for port 3143. |
| 520 |
See tcpd(8) and related manpages for further details. |
| 521 |
|
| 522 |
5. Configure clients to use the alternative port (3143 in the |
| 523 |
example above). |
| 524 |
|
| 525 |
5.4 Access control with iptables |
| 526 |
|
| 527 |
Looking at performance, the most efficient way to estables access |
| 528 |
control by source IP is the use of system's native mechanisms. For |
| 529 |
the Linux kernel, it can be created with iptables rules like in the |
| 530 |
following example, usually stored in system's firewall setup script |
| 531 |
(or any init script like `/etc/rc.local'): |
| 532 |
|
| 533 |
iptables -A INPUT -p tcp --dport 3142 --source 127.0.0.0/8 -j ACCEPT |
| 534 |
iptables -A INPUT -p tcp --dport 3142 --source 192.168.0.0/16 -j ACCEPT |
| 535 |
iptables -A INPUT -p tcp --dport 3142 -j REJECT --reject-with tcp-reset |
| 536 |
|
| 537 |
5.5 Target port filter |
| 538 |
|
| 539 |
In the default configuration, apt-cacher-ng restricts the choice |
| 540 |
of the target port on the remote mirror to the default HTTP port |
| 541 |
(80). This is intended to prevent creation of HTTP tunnels and |
| 542 |
similar malicious user activities. |
| 543 |
|
| 544 |
Cache administrator can define an own list of allowed ports by |
| 545 |
setting them as (space separated) list with the AllowUserPorts |
| 546 |
configuration directive. |
| 547 |
|
| 548 |
Alternatively, access to custom TCP ports can be established through |
| 549 |
custom remapping specification. |
| 550 |
|
| 551 |
Chapter 6: Distribution specific instructions |
| 552 |
--------------------------------------------- |
| 553 |
|
| 554 |
6.1 Debian and Ubuntu |
| 555 |
|
| 556 |
Use as is. Report bugs using reportbug (Debian) or to Launchpad |
| 557 |
(Ubuntu). |
| 558 |
|
| 559 |
6.2 OpenSUSE |
| 560 |
|
| 561 |
Server can be used as is with limited expiration (see below, and see |
| 562 |
`INSTALL' file for compilation hints). The merging mode (multiple |
| 563 |
servers mapped into the same repository) is not preconfigured in the |
| 564 |
example configuration. This is object to research, competent support |
| 565 |
is required. |
| 566 |
|
| 567 |
Clients can configure apt-cacher-ng as central proxy in Yast |
| 568 |
("Network devices"/"Proxy" tab). If this is not desirable then each |
| 569 |
sofware source can be edited to be redirected through the server. |
| 570 |
This can be done in the Software Installer view of Yast2, see menu |
| 571 |
Configuration/Repositories. To edit them quickly, switch to URL |
| 572 |
editing mode and insert `host:port/' (of the apt-cacher-ng server) |
| 573 |
between `http://' and the source server name. |
| 574 |
|
| 575 |
6.3 Fedora Core |
| 576 |
|
| 577 |
Attempts to add apt-cacher-ng support ended up in pain and the |
| 578 |
author lost any motivation in further research on this subject. |
| 579 |
|
| 580 |
6.4 Arch Linux |
| 581 |
|
| 582 |
Mostly usable. The mirror structure design has been identified by |
| 583 |
trial-and-error and the regular expressions might need some tuning |
| 584 |
by experts. |
| 585 |
|
| 586 |
The installer seems to have no way to specify a dedicated proxy but |
| 587 |
it's possible to edit the source URL and insert the `host:port/' |
| 588 |
part into it. The pacman mirror list can be modified the same way. |
| 589 |
|
| 590 |
Expiration code should work in the basic mode (index data is |
| 591 |
extracted from *.db.tar.gz files). File checksum checking mode |
| 592 |
might also work (untested). The example configuration contains a |
| 593 |
preconfigured list of mirrors which can be rebuilt with the Makefile |
| 594 |
if needed. The preferred backend server can be specified like with |
| 595 |
other distributions (see above for details). |
| 596 |
|
| 597 |
6.5 Sourceforge mirror network |
| 598 |
|
| 599 |
Not a Linux distro but commonly used by those to download certain |
| 600 |
files. Therefore most mirrors can get unified access cache sharing |
| 601 |
the files in the same cache repository. Some .exe files there are |
| 602 |
never expired. |
| 603 |
|
| 604 |
6.6 Cygwin mirrors |
| 605 |
|
| 606 |
While not being a pure Linux distro it's mostly GNU and has a nice |
| 607 |
mirror setup. Proxy server can be used as is with limited expiration |
| 608 |
(see below). Apt-cacher-ng can also be compiled and used on Windows |
| 609 |
machines in Cygwin environment (see `INSTALL' file for details). |
| 610 |
|
| 611 |
Clients need to specify the server as HTTP proxy in the setup.exe |
| 612 |
wizard, only HTTP mirrors should be selected in the mirror list. |
| 613 |
|
| 614 |
6.7 Limited expiration |
| 615 |
|
| 616 |
The expiration code for Non-Debian/Ubuntu repositories is quite |
| 617 |
limited due to lack of man-power or know-how. For some distros, |
| 618 |
the support is limited to checks of the filename and no further |
| 619 |
validation is supported. Therefore, the extra validation of path |
| 620 |
location or file contents should NOT be turned on when running |
| 621 |
expiration with data from that distros in the cache, because good |
| 622 |
data may be deleted in this case. |
| 623 |
|
| 624 |
Chapter 7: Maintenance |
| 625 |
---------------------- |
| 626 |
|
| 627 |
There are few optional tasks that need to be executed by the |
| 628 |
administrator from time to time or during the initial configuration. |
| 629 |
|
| 630 |
7.1 Cache cleanup |
| 631 |
|
| 632 |
If a package is no longer downloadable by APT clients then its |
| 633 |
files are also not referenced in any index file and can be |
| 634 |
removed. This rule also applies to most volatile files from the |
| 635 |
distribution metadata. For example, Debian's Release file references |
| 636 |
some Packages and Sources files or Diff-Index file, and those do |
| 637 |
reference most other non-volatile files (binary packages, source |
| 638 |
packages, index diffs, ...). |
| 639 |
|
| 640 |
7.1.1 Manual expiration |
| 641 |
|
| 642 |
To run the cleanup action manually visit the report page in a |
| 643 |
browser and trigger the `Expiration' operation there. |
| 644 |
|
| 645 |
There are different flags configuring the parameters of this |
| 646 |
tracking described below. Usually just the filename is sufficient |
| 647 |
to consider a file in the cache as a valid (downloadable) file. |
| 648 |
This is ok in most cases but sometimes leads to false positives, |
| 649 |
i.e. when another repository in the cache refers to a file with the |
| 650 |
same name but the reference to the original location is gone. On |
| 651 |
the other hand there can be cases where the assignment to different |
| 652 |
repositories happened by mistake and administrator would like to |
| 653 |
merge repositories later on. |
| 654 |
|
| 655 |
For most files the checksum values are also provided in the index |
| 656 |
files and so the file contents can be validated as well. This |
| 657 |
requires reading of the whole cache archive to generate local |
| 658 |
checksums. It should also not be done when apt-cacher-ng is being |
| 659 |
used (file locking is not used here). |
| 660 |
|
| 661 |
Usually it's necessary to bring various index files |
| 662 |
(Release,Sources,Packages,Index) in sync with the repository. This |
| 663 |
is necessary because apt works around the whole file download by |
| 664 |
fetching small patches for the original file, and this mode of |
| 665 |
operation is not supported yet by apt-cacher-ng (and might still |
| 666 |
be unreliable). When this synchronization fails, the index files |
| 667 |
might be incomplete or obsolete or damaged, and they might no longer |
| 668 |
contain references to some files in the cache. Abortion of the |
| 669 |
cleanup process is advisable in this case. |
| 670 |
|
| 671 |
There is also a precaution mechanism designed to prevent the |
| 672 |
destruction of cache contents when some volatile index files have |
| 673 |
been lost temporarily. The results of cache examination are stored |
| 674 |
in a list with the date when the particular files became orphaned. |
| 675 |
The removals are only executed after few days (configurable, see |
| 676 |
configuration file) unless they are removed from this list in the |
| 677 |
meantime. |
| 678 |
|
| 679 |
Parameters of `Expiration': |
| 680 |
|
| 681 |
Skip header checks |
| 682 |
|
| 683 |
By default, header description file of every package is opened |
| 684 |
and checked for bad data and for obvious inconsistencies (like |
| 685 |
local file being larger than specified by server). Which means |
| 686 |
opening reading a few kilobytes from disk for almost every file |
| 687 |
in the cache, and slightly degrades performance of the process. |
| 688 |
This option skips that basic checks. |
| 689 |
|
| 690 |
Stop cleanup on errors during index update step |
| 691 |
|
| 692 |
Index files update is done first, on errors the expiration will |
| 693 |
be interrupted. |
| 694 |
|
| 695 |
Validate by file name AND file directory |
| 696 |
|
| 697 |
This option can be used to remove distribution stages. Example: |
| 698 |
to remove "oldstable" one just needs to delete the "Release" |
| 699 |
files in the cache and run Expiration with this option two |
| 700 |
times. There are some issues with this mode operation, see above |
| 701 |
for details. |
| 702 |
|
| 703 |
Validate by file name AND file contents (through checksum) |
| 704 |
|
| 705 |
Checking file size and contents where possible against the |
| 706 |
metadata in the index files. Note: future calls of Expiration |
| 707 |
process without this option will discard the results of this |
| 708 |
check and forget about corrupted files. Therefore, an action |
| 709 |
on this files needs to be done ASAP, like truncating them (see |
| 710 |
below) or removing via the removal operation (using the checkbox |
| 711 |
and the Delete button, see process output) or via the "Delete |
| 712 |
all unreferenced files" operation on the main control page. |
| 713 |
|
| 714 |
Force the download of index files |
| 715 |
|
| 716 |
Sometimes it may be needed to redownload all index files, |
| 717 |
explicitly replacing the cached versions. This flag enables this |
| 718 |
behaviour. |
| 719 |
|
| 720 |
Purge unreferenced files after scan |
| 721 |
|
| 722 |
Avoid the use of the orphan list and delete files instead. |
| 723 |
This option is dangerous and should not be used unless when |
| 724 |
absolutely no mistakes/problems can happen. Instead, it's |
| 725 |
possible to view the orphan list later and delete then (see |
| 726 |
control web interface). |
| 727 |
|
| 728 |
Truncate damaged files immediately |
| 729 |
|
| 730 |
If a file has been identified as damaged, it will be truncated |
| 731 |
(file size reset to 0). Setting this option is a good compromise |
| 732 |
for debugging purposes compared to the simple deletion since it |
| 733 |
will keep the header files on the disk, for further analysis of |
| 734 |
the problem's cause. |
| 735 |
|
| 736 |
More verbosity |
| 737 |
|
| 738 |
Shows more information, e.g. each scanned file when used with |
| 739 |
some of the other options. This might result in a very large |
| 740 |
HTML page, making the watching HTML browser very slow. |
| 741 |
|
| 742 |
In additional to the default scan run, there are some "Direct |
| 743 |
Action" buttons in the Web frontend. It's possible to see the |
| 744 |
temporary list of files that have been identified as orphaned |
| 745 |
(unreferenced), and it's possible to delete all files from that list |
| 746 |
immediately. To be used carefully! |
| 747 |
|
| 748 |
7.1.2 Automated cache cleanup |
| 749 |
|
| 750 |
A program called `acngtool' is shipped with the package. Among other |
| 751 |
functionality, this tool effectively implements a HTTP client which |
| 752 |
operates like a human who runs the expiration manually (see above). |
| 753 |
If needed, it would read the admin credentials and unix socket file |
| 754 |
path from the local configuration files. On Debian installations it |
| 755 |
is called by the file `/etc/cron.daily/apt-cacher-ng' so it should |
| 756 |
run automatically as a daily `cron' task. The results are usually |
| 757 |
not reported unless an error occurs, in which case some hints are |
| 758 |
written to the standard error output (i.e. sent in cron mails). |
| 759 |
|
| 760 |
The operator script can take some options from the environment, |
| 761 |
see below. The default operation mode is calling the expiration |
| 762 |
operation with default parameters and with credentials from local |
| 763 |
system's apt-cacher-ng installation. However, this can be changed |
| 764 |
with `ACNGREQ' variable. |
| 765 |
|
| 766 |
`ACNGIP=10.0.1.3' |
| 767 |
|
| 768 |
The network address for remote connection may be guessed |
| 769 |
incorrectly by the operator script. This variable can specify |
| 770 |
an explicit target to connect to, e.g. the same IP as the one |
| 771 |
used by the clients (unless this network connection is somehow |
| 772 |
restricted in the local setup). |
| 773 |
|
| 774 |
`HOSTNAME=localOrPublicName' |
| 775 |
|
| 776 |
When an error occurs, the operator script most likely adds an |
| 777 |
URL to be opened for further investigation. The host name of |
| 778 |
in this URL can be customized, i.e. can be set to a public |
| 779 |
domain name representing the server as accessible from the |
| 780 |
administrator's machine. |
| 781 |
|
| 782 |
`ACNGREQ=cgiparameters' |
| 783 |
|
| 784 |
Override the auto-detected command parameters with a custom |
| 785 |
query string. This is the part of a command URL from the |
| 786 |
management interface after the ? sign. |
| 787 |
|
| 788 |
7.1.3 Keeping latest versions of expired package files |
| 789 |
|
| 790 |
Sometimes it makes sense to keep a couple of versions of (Debian) |
| 791 |
packages even after they have been removed from remote source. It |
| 792 |
is possible to set an exceptional rule for package files which |
| 793 |
follow the naming and versioning scheme of .deb-packages. This extra |
| 794 |
handling is configured by the `KeepExtraVersions' options which |
| 795 |
tells how many of the top-latest versions shall be kept. The cache |
| 796 |
system needs the dpkg program and sufficient CPU power (depending on |
| 797 |
the option value). |
| 798 |
|
| 799 |
7.2 Removal of distribution releases |
| 800 |
|
| 801 |
Sometimes it's needed to remove all files from a distribution, i.e. |
| 802 |
when a new release became Stable and older package files are still |
| 803 |
lying around. In perfect conditions the reference tracking described |
| 804 |
above should take care of it and remove them soon. |
| 805 |
|
| 806 |
However, this solution will fail if the release files are still |
| 807 |
available on the server AND apt-cacher-ng learned their real |
| 808 |
location (i.e. the code name instead of not the release state name) |
| 809 |
and so they are refreshed during regular expiration. |
| 810 |
|
| 811 |
After all, if the old release is no longer used by local cache users |
| 812 |
then the extra disk usage can become problematic. This problem will |
| 813 |
go away after many months when the old release files are finally |
| 814 |
deleted on the servers, then the package expiration will start |
| 815 |
complaining for some days (the expiration delay) and only then the |
| 816 |
finally unreferenced files will be removed. |
| 817 |
|
| 818 |
To speed up this process, the local administrator can remove the |
| 819 |
index files of the old distribution release from the archive. This |
| 820 |
can be done by using the Expiration task in the administrator |
| 821 |
portal. On the end of the cache analysis, a helper form can be |
| 822 |
expanded where index file can be selected and the destruction of |
| 823 |
those cache contents can be ordered. |
| 824 |
|
| 825 |
Chapter 8: HOWTOs and FAQ |
| 826 |
------------------------- |
| 827 |
|
| 828 |
8.1 Package import |
| 829 |
|
| 830 |
Already existing packages can be imported into apt-cacher-ng's cache |
| 831 |
pool instead of downloading them. There are some restrictions: |
| 832 |
|
| 833 |
1. Don't try to import incomplete files. They will be refused since |
| 834 |
their contents cannot be checked against the archive metadata. |
| 835 |
|
| 836 |
2. If possible, don't import symbolic links. Even if doing so, |
| 837 |
they should not point to other files inside of the cache and |
| 838 |
especially not to other files under the `_import' directory. |
| 839 |
|
| 840 |
HOWTO: |
| 841 |
|
| 842 |
1. Make sure that apt-cacher-ng has valid index files in the cache. |
| 843 |
This is the tricky part. To get them right, a client needs to |
| 844 |
download them through apt-cacher-ng once. Therefore: |
| 845 |
|
| 846 |
1. Configure the server and one client before doing the import. |
| 847 |
See above for instructions. |
| 848 |
|
| 849 |
2. Run "apt-get update" on client(s) once to teach ACNG about |
| 850 |
remote locations of (volatile) index files. In some cases |
| 851 |
this is not sufficient. See the note on APT below for a |
| 852 |
workaround. |
| 853 |
|
| 854 |
2. Store copies of your .debs, .orig.tar.gz, ... somewhere in the |
| 855 |
`_import' subdirectory in the cache, ie. in `/var/cache/apt- |
| 856 |
cacher-ng/_import/'. The files may be links or symlinks, it does |
| 857 |
not matter. When done, apt-cacher will move those files to its |
| 858 |
own internal locations. Example: |
| 859 |
|
| 860 |
cd /var/cache |
| 861 |
mkdir apt-cacher-ng/_import |
| 862 |
cp -laf apt-proxy apt-cacher /var/cache/apt-cacher-ng/_import |
| 863 |
chown -R apt-cacher-ng:apt-cacher-ng apt-cacher-ng/_import |
| 864 |
|
| 865 |
3. Visit the report page and trigger the import action there. Check |
| 866 |
the results, look for (red) error messages. |
| 867 |
|
| 868 |
4. Check the `_import' directory again. All files that could be |
| 869 |
identified as referenced by archive metadata should no longer |
| 870 |
be there if they have been successfully moved. If some files |
| 871 |
have been left behind, check whether the client can use them, |
| 872 |
i.e. with "apt-cache policy ..." and/or checking checksums |
| 873 |
with md5sum/sha1sum tools. Probably they are no longer needed |
| 874 |
by anyone and therefore apt-cacher-ng just left them behind. |
| 875 |
If no, follow the instructions in 1 or do similar things for |
| 876 |
your distribution and retry the import operation. Setting the |
| 877 |
verbosity flag (see checkbox on the command-and-control page) |
| 878 |
can also help to discover the reason for the refusal to import |
| 879 |
the particular files. |
| 880 |
|
| 881 |
NOTE: APT is pretty efficient on avoiding unneccessary downloads |
| 882 |
which can make a proxy blind to some relevant files. ACNG makes some |
| 883 |
attempts to guess the remote locations of missed (not downloaded) |
| 884 |
files but these heuristics may fail, especially on non-Debian |
| 885 |
systems. When some files are permanently ignored, check the process |
| 886 |
output for messages about the update of Packages/Sources files. When |
| 887 |
some relevant package sources are missing there, there is a brute- |
| 888 |
force method for Debian/Ubuntu users to force their download to the |
| 889 |
client side. To do that, run: |
| 890 |
|
| 891 |
rm /var/cache/apt/*cache.bin |
| 892 |
rm /var/lib/apt/lists/*Packages |
| 893 |
rm /var/lib/apt/lists/*Sources |
| 894 |
|
| 895 |
on the client to purge APT's internal cache, and then rerun "apt-get |
| 896 |
update" there. |
| 897 |
|
| 898 |
8.2 Access to SSL/TLS remotes (HTTPS) |
| 899 |
|
| 900 |
It is possible to have encrypted access to remote sites via HTTPS |
| 901 |
protocol with recent versions of apt-cacher-ng if the OpenSSL |
| 902 |
support was enabled at compile time. However this leads certain side |
| 903 |
effects and complications; due to the nature of the HTTPS connection |
| 904 |
model, it is not possible to act as an intermediate server (e.g. |
| 905 |
caching proxy) by the same rules as with HTTP: |
| 906 |
|
| 907 |
- SSL ensures strict verification of remote host name. Therefore, |
| 908 |
you cannot add an URL to sources.list and make them point to |
| 909 |
"hostname of acng server": the download client would expect |
| 910 |
the remote site to offer the certificate of "hostname of acng |
| 911 |
server" but the certificate of the remote server would be used. |
| 912 |
|
| 913 |
- This restriction also applies if the LAN admin somehow manages |
| 914 |
to trick the client that it connects to the remote site but |
| 915 |
the real connection is silently rerouted to the local server: |
| 916 |
since the proxy in the middle cannot fake the real certificate |
| 917 |
it would need to create a custom one but that manipulation |
| 918 |
usually cannot go unnoticed by the client unless some local hack |
| 919 |
disables or manipulates the certificate verification. All parts |
| 920 |
of this should be considered a crude hack and won't be described |
| 921 |
here in detail. In fact, apt-cacher-ng does not implement local |
| 922 |
SSL server functionality (as of version 0.8.1), partly because |
| 923 |
of these considerations. |
| 924 |
|
| 925 |
- If the client assumes that it needs to access a HTTPS remote |
| 926 |
server through a HTTP proxy, it usually goes for the HTTP |
| 927 |
tunneling (see the Wikipedia article for details). This is |
| 928 |
a well-known and widely-used method but it comes with some |
| 929 |
disadvantages: the proxy has no control of the data flowing |
| 930 |
through it so the proxy cannot cache the data in any useful way; |
| 931 |
in environments with additional security restrictions, it's also |
| 932 |
hard to identify malicious users that abuse the service for |
| 933 |
illegitimate purposes. |
| 934 |
|
| 935 |
Considering these difficulties, there are three (and a half) methods |
| 936 |
to use SSL. |
| 937 |
|
| 938 |
- First, the "half method" - not using the proxy at all, |
| 939 |
configuring each client to not use the HTTP proxy for HTTPS |
| 940 |
urls. This will obviously disable central caching and requires |
| 941 |
the client has separated configuration options to set this. For |
| 942 |
Debian based distros, this can be done by adding a configuration |
| 943 |
like this: `Acquire::https::proxy "DIRECT";' to apt.conf or |
| 944 |
one of the apt.conf.d files. See section 8.4 for further |
| 945 |
information. |
| 946 |
|
| 947 |
- The "backend configuration method": if the clients access the |
| 948 |
remote sites through URLs remapped on the server, the cacher |
| 949 |
admin can add https URLs to backend lists instead of http urls. |
| 950 |
Data will be cached just like usual. |
| 951 |
|
| 952 |
- The "laissez-faire method": in acng.conf (or related) configure |
| 953 |
the `PassThroughPattern' option to contain a regex like `.*' |
| 954 |
and configure the clients to use apt-cacher-ng as HTTP proxy |
| 955 |
and let the clients connect to https URLs "as usual". Some |
| 956 |
limited access control can be achieved through adjustment of the |
| 957 |
regular expression (.* permits access to any host and any port, |
| 958 |
including 443 for https). Data is not cached on the server. |
| 959 |
|
| 960 |
- The "tell-me-what-you-need method": on the client side, modify |
| 961 |
the access URLs and change https to http and put the string |
| 962 |
"HTTPS///" between http:// and the host name. With this trick, |
| 963 |
the user client will access the proxy like going for a usual |
| 964 |
HTTP download and the proxy will access the remote URL with |
| 965 |
the https protocol. Caching (and file merging to repositories) |
| 966 |
will work and there is still enough flexibility for the users. |
| 967 |
The disadvantages of this method are basically the same as with |
| 968 |
the access URLs rewriting (see Section 3.2) but the method is |
| 969 |
still the preferred one by the apt-cacher-ng author. For apt's |
| 970 |
sources.list, the modification may look like the following |
| 971 |
example. |
| 972 |
|
| 973 |
deb http://HTTPS///get.docker.com/ubuntu docker main |
| 974 |
# If apt-cacher-ng is configured as proxy in APT, this makes it |
| 975 |
# switch internally to https://get.docker.com/ubuntu |
| 976 |
deb http://acnghost:3142/HTTPS///get.docker.com/ubuntu docker main |
| 977 |
# Basically the same just with access to apt-cacher-ng through |
| 978 |
# URL rewritting instead of setting http proxy. |
| 979 |
|
| 980 |
8.3 JIGDO usage |
| 981 |
|
| 982 |
It's possible to use apt-cacher-ng source with the jigdo-lite |
| 983 |
utility. There are some limitations, though: |
| 984 |
|
| 985 |
- since many mirrors do not distribute the jigdo files (or even |
| 986 |
nothing from cdimage.debian.org at all), there is a high chance |
| 987 |
to be redirected to a such mirror when using the backend-mapped |
| 988 |
configuration. I.e. when user follows the official documentation |
| 989 |
and edits wgetOpts in the jigdo configuration, it will fail in |
| 990 |
many cases. |
| 991 |
|
| 992 |
- apt-cacher-ng does not support .template files properly. They |
| 993 |
might be cached but will be expired (removed from cache), sooner |
| 994 |
or later. |
| 995 |
|
| 996 |
But it's possible to feed jigdo-lite with the package contents from |
| 997 |
your mirror. To do that, first start jigdo-lite as usual, something |
| 998 |
like: |
| 999 |
|
| 1000 |
`jigdo-lite http://cdimage.debian.org/.../...-DVD-1.jigdo' |
| 1001 |
|
| 1002 |
When asked about Debian mirror, enter something like: |
| 1003 |
|
| 1004 |
`http://proxy.host:3142/ftp.de.debian.org/debian/' |
| 1005 |
|
| 1006 |
i.e. construct the same URL as present in usual apt-cacher-ng's |
| 1007 |
user's sources.list. |
| 1008 |
|
| 1009 |
That's all, jigdo-lite will fetch the package files using apt- |
| 1010 |
cacher-ng proxy. |
| 1011 |
|
| 1012 |
8.4 Avoid use of apt-cacher-ng for certain hosts |
| 1013 |
|
| 1014 |
Sometimes clients might need to access some remote side directly |
| 1015 |
to do some non-file-transfer oriented work but still passing the |
| 1016 |
data through configured apt-cacher-ng proxy. Such remote hosts |
| 1017 |
can be marked for direct access in apt configuration, e.g. in |
| 1018 |
`/etc/apt/apt.conf': |
| 1019 |
|
| 1020 |
Acquire::HTTP::Proxy::archive.example.org "DIRECT"; |
| 1021 |
//or Acquire::HTTP::Proxy::archive.example.org "other.proxy:port" |
| 1022 |
|
| 1023 |
8.5 Avoid caching for certain domains or certain file types |
| 1024 |
|
| 1025 |
Sometimes clients to download through apt-cacher-ng but the data |
| 1026 |
shall not be stored on the harddisk of the server. To get it, use |
| 1027 |
the DontCache directive (see examples for details) to define such |
| 1028 |
files. |
| 1029 |
|
| 1030 |
8.6 How to make big download series faster |
| 1031 |
|
| 1032 |
Symptom: A common situation is a periodic download of hundreds of |
| 1033 |
files through apt-cacher-ng where just a half is present in the |
| 1034 |
cache. Although caching works fine, there are visible delays on some |
| 1035 |
files during the download. |
| 1036 |
|
| 1037 |
Possible cause and relief: the download from the real mirror gets |
| 1038 |
interrupted while apt-cacher-ng delivers a set of files from the |
| 1039 |
internal cache. While the connection is suspended, it times out and |
| 1040 |
needs to be recreated when a miss occurs, i.e. apt-cacher-ng has to |
| 1041 |
fetch more from the remote mirror. A workaround to this behaviour |
| 1042 |
is simple, provided that the remote mirror can handle long request |
| 1043 |
queues: set the pipelining depth to a very high value in apt.conf |
| 1044 |
file or one of its replacement files in /etc/apt/apt.conf.d/. With |
| 1045 |
something like: |
| 1046 |
|
| 1047 |
`Acquire::http { Pipeline-Depth "200"; } ' |
| 1048 |
|
| 1049 |
there is a higher chance to get the server connection "preheated" |
| 1050 |
before a stall occurs. |
| 1051 |
|
| 1052 |
8.7 How to import DVDs or ISO images |
| 1053 |
|
| 1054 |
First, it should be clear what is needed to be done. In order to |
| 1055 |
integrate the packages from a DVD or ISO image, read on in section |
| 1056 |
8.8. |
| 1057 |
|
| 1058 |
The situation with ISO files import is complicated. They are not |
| 1059 |
supported by the cache and there is also no expiration mode for |
| 1060 |
them. The feature might be considered for addition in some future |
| 1061 |
release of apt-cacher-ng. |
| 1062 |
|
| 1063 |
What is possible now is publishing a directory with ISO files using |
| 1064 |
its web server mode, see `LocalDirs' config option for details. |
| 1065 |
|
| 1066 |
8.8 How to integrate DVDs or ISO image data |
| 1067 |
|
| 1068 |
Integrating package files from DVD or ISO images is not much |
| 1069 |
different to the usual import operation, see above for instructions. |
| 1070 |
|
| 1071 |
One possible way to get files into the `_import' directory is simply |
| 1072 |
mounting it there: |
| 1073 |
|
| 1074 |
mount -o loop /dev/cdrom /var/cache/apt-cacher-ng/_import |
| 1075 |
|
| 1076 |
After running the import operation, the disk can be umounted and |
| 1077 |
removed. |
| 1078 |
|
| 1079 |
A possible variation is import via symlinks. This can make sense |
| 1080 |
where the space consumption must be reduced and the ISO image should |
| 1081 |
stay on the server for a long time. To achive this, the image should |
| 1082 |
be mounted at some mount point outside of the `_import' directory; |
| 1083 |
the mounted state should be made permanent, maybe via an /etc/fstab |
| 1084 |
entry with the loop option; then a symbolic link tree pointing to |
| 1085 |
the mountpoint location should be created in the `_import' directory |
| 1086 |
(something like `cp -as /mnt/image_jessie_01/pool /var/cache/apt- |
| 1087 |
cacher-ng/_import/'). The subsequent "import" operation should pick |
| 1088 |
up the symlinks and continue using them as links istead of file |
| 1089 |
copies. |
| 1090 |
|
| 1091 |
8.9 How to execute commands before and after going online? |
| 1092 |
|
| 1093 |
It is possible to configure custom commands which are executed |
| 1094 |
before the internet connection attempt and after a certain period |
| 1095 |
after closing the connection. The commands are bound to a remapping |
| 1096 |
configuration and the config file is named after the name of that |
| 1097 |
remapping config, like `debrep.hooks' for `Remap-debrep'. See |
| 1098 |
section 4.3.2, `conf/*.hooks' and `/usr/share/doc/apt-cacher- |
| 1099 |
ng/examples/*.hooks' files for details. |
| 1100 |
|
| 1101 |
8.10 Listen to only specific interfaces or IP protocols |
| 1102 |
|
| 1103 |
Unless configured explicitely, the server listens to any interface |
| 1104 |
with IPv4 or IPv6 protocol. To disable some of this, use the |
| 1105 |
`BindAddress' option. It should contain a list of IP adresseses |
| 1106 |
associated with particular network interfaces, separated by space. |
| 1107 |
When option is set then the server won't listen to addresses or |
| 1108 |
protocols not included there. |
| 1109 |
|
| 1110 |
To limit to specific IP protocol, the address should only be present |
| 1111 |
in the protocol specific syntax (like 192.0.43.10) will limit the |
| 1112 |
use to the specific protocol. |
| 1113 |
|
| 1114 |
The usual wildcard addresses can also be used to match all |
| 1115 |
interfaces configured for the specific protocol, like 0.0.0.0 for |
| 1116 |
IPv4. |
| 1117 |
|
| 1118 |
8.11 How to avoid use of IPv4 (or IPv6) where possible? |
| 1119 |
|
| 1120 |
Usually, outgoing hosts are accessed by the protocol and with the |
| 1121 |
target IP reported as the first candidate by operating system |
| 1122 |
facilities (getaddrinfo). It is possible to change this behavior, |
| 1123 |
i.e. to skip IPv6 or IPv4 versions or try IPv6 connection first |
| 1124 |
and then use IPv4 as alternative (or vice versa). See option |
| 1125 |
ConnectProto in configuration examples. |
| 1126 |
|
| 1127 |
8.12 Use the proxy without storing all data twice |
| 1128 |
|
| 1129 |
There is a general use case where the data storing behavior of APT |
| 1130 |
is not so fortunate. Imagine an old laptop with a slow and small |
| 1131 |
harddisk but a modern network connection (i.e. Cardbus-attached WLAN |
| 1132 |
card). But there is not enough space for APT to store the downloaded |
| 1133 |
packages on the local disk, or not enough to perform the upgrade |
| 1134 |
afterwards. |
| 1135 |
|
| 1136 |
A plausible workaround in this case are moving contents of |
| 1137 |
/var/cache/apt/archives directory to a mounted NFS share and |
| 1138 |
replacing the original directory with a symlink (or bind-mount to |
| 1139 |
the mentioned share). However, this solution would transfer all data |
| 1140 |
at least three times over network. Another plausible workaround |
| 1141 |
might be the use of curlftpfs which would embedd a remote FTP share |
| 1142 |
which then can be specified as file:// URL in sources.list. However, |
| 1143 |
this solution won't work with a local HTTP proxy like apt-cacher- |
| 1144 |
ng (and httpfs http://sourceforge.net/projects/httpfs/ is not an |
| 1145 |
alternative because it works only with a single file per mount). |
| 1146 |
|
| 1147 |
As real alternative, apt-cacher-ng comes with an own implementation |
| 1148 |
of a http file system called `acngfs'. It makes some assumptions of |
| 1149 |
proxy's behaviour in order to emulate a real directory structure. |
| 1150 |
Directories can be entered but not browsed (i.e. content listing |
| 1151 |
is disallowed because of HTTP protocol limitations). Anyhow, this |
| 1152 |
solution is good enough for APT. When it's checking the contents of |
| 1153 |
the data source located on acngfs share, it reads the file contents |
| 1154 |
of just the files required for the update which makes the apt- |
| 1155 |
cacher-ng server download them on-the-fly. |
| 1156 |
|
| 1157 |
And finally, angfs usage can be optimized for local access. This |
| 1158 |
works best if the proxy daemons runs on the same machine as acngfs |
| 1159 |
and there are hundreds of packages to update while filesystem access |
| 1160 |
costs are negligible. Here the cache directory can be specified in |
| 1161 |
acngfs parameters, and then it gets files directly from the cache if |
| 1162 |
they are completely downloaded and don't have volatile contents. |
| 1163 |
|
| 1164 |
8.13 Optional semi-automatic use of proxy |
| 1165 |
|
| 1166 |
Apt-Cacher NG daemon has some optional operations modes regading the |
| 1167 |
use of external proxy (configured with "Proxy" setting). The default |
| 1168 |
mode means use of that proxy for all remote connections. |
| 1169 |
|
| 1170 |
However, in some environments like when running on a portable |
| 1171 |
system, the "upstream" proxy server may not reachable when running |
| 1172 |
outside of home LAN area. For this situation, there are automated |
| 1173 |
ways to detect it and to switch to direct internet access without |
| 1174 |
additional configuration work. |
| 1175 |
|
| 1176 |
One way is to use a (short) timeout value (OptProxyTimeout setting) |
| 1177 |
which simply makes a failed connection attempt after the timeout |
| 1178 |
being interpret as broken proxy. Then the proxy is not used |
| 1179 |
in context of this connection until a new connection is to be |
| 1180 |
established. If OptProxyCheckInterval is set (see below) then this |
| 1181 |
change is effective for the configured time span. |
| 1182 |
|
| 1183 |
Another way is through a custom (shell) command - when it returns |
| 1184 |
successfully, the proxy is used, otherwise not and the command |
| 1185 |
will be rerun only after a specified period. The command may |
| 1186 |
check IP routes or a specific router address in the MAC pool |
| 1187 |
or similar traces of the current environment. This setting |
| 1188 |
(OptProxyCheckCommand) is also affected by another option |
| 1189 |
(OptProxyCheckInterval) that specifies how often this check command |
| 1190 |
shall be rerun. In the meantime, the configured proxy is considered |
| 1191 |
faulty. |
| 1192 |
|
| 1193 |
Hint: this special modes can be combined with non-caching behavior |
| 1194 |
(see above) and another ACNG proxy in the home LAN to get smart |
| 1195 |
caching for laptop users. |
| 1196 |
|
| 1197 |
8.14 Partial Mirroring |
| 1198 |
|
| 1199 |
It is possible to create a partial local mirror of a remote package |
| 1200 |
repository. The method to do this is usually known as pre-caching. A |
| 1201 |
such mirror would contain all files available to apt through `apt- |
| 1202 |
cacher-ng', making the cache server suitable for pure off-line use. |
| 1203 |
|
| 1204 |
The config uses index files in the local cache in order to declare |
| 1205 |
which remote files shall be mirrored. Choice of relevant files |
| 1206 |
decides which branch, which architecture or which source tree is to |
| 1207 |
be mirrored. For convenience, it's possible to use glob expressions |
| 1208 |
to create semi-dynamic list. The format is shell-like and relative |
| 1209 |
to cache directory, a shell running in the cache directory can be |
| 1210 |
helpful to verify the correctness. |
| 1211 |
|
| 1212 |
Example: |
| 1213 |
|
| 1214 |
PrecacheFor: debrep/dists/unstable/*/binary-amd64/Packages* |
| 1215 |
|
| 1216 |
PrecacheFor: emacs.naquadah.org/unstable/* |
| 1217 |
|
| 1218 |
Assuming that debrep repository is configured with proper remapping |
| 1219 |
setup (see above), this would download all Debian packages listed |
| 1220 |
for amd64 architecture in the unstable branch. |
| 1221 |
|
| 1222 |
There is also support for faster file update using deltas, see |
| 1223 |
Debdelta for details. The delta_uri URL mentioned there needs to be |
| 1224 |
added as deltasrc option, see section 4.3.2 for details. |
| 1225 |
|
| 1226 |
The operation is triggered using the web interface, various options |
| 1227 |
or estimation mode can also be configured there. The CGI URL |
| 1228 |
generated by the browser can be called with other clients to repeat |
| 1229 |
this job, for example in a daily executed script. For regular tools |
| 1230 |
like wget or curl, remember the need of quotation and secrecy of |
| 1231 |
user/password data - command calls might expose them to local users. |
| 1232 |
Alternatively, the tool `acngtool' (see above) can be used in order |
| 1233 |
to run such commands even over a secured socket which prevents |
| 1234 |
potential exposure of the credentials. |
| 1235 |
|
| 1236 |
Chapter 9: Troubleshooting |
| 1237 |
-------------------------- |
| 1238 |
|
| 1239 |
9.1 Debugging |
| 1240 |
|
| 1241 |
Preliminary meanings of Debug option settings are: |
| 1242 |
|
| 1243 |
- 0: No debug printing |
| 1244 |
|
| 1245 |
- 1: Log file buffers are flushed faster |
| 1246 |
|
| 1247 |
- 2: Some additional information appears within usual |
| 1248 |
transfer/error logs |
| 1249 |
|
| 1250 |
- 4: extra debug information is written to apt-cacher.err (also |
| 1251 |
enables lots of additional trace points when apt-cacher-ng |
| 1252 |
binary is built with debug configuration, see section 9.6 for |
| 1253 |
details) |
| 1254 |
|
| 1255 |
To combine that settings, add them (i.e. 7 enables all messages and |
| 1256 |
log flushing) |
| 1257 |
|
| 1258 |
Getting HTTP headers from apt-get works like this: |
| 1259 |
|
| 1260 |
apt-get update -o Debug::Acquire::Http=true |
| 1261 |
|
| 1262 |
9.2 Problem: keeps delivering damaged files |
| 1263 |
|
| 1264 |
Even in this millennium, sometimes damaged files are downloaded from |
| 1265 |
the server and are stored in the cache. Sometimes lazy maintainers |
| 1266 |
of 3rd party archives replace package files with the same name |
| 1267 |
but different contents. Sometimes the server's file system gets |
| 1268 |
corrupted without detection by the OS. |
| 1269 |
|
| 1270 |
Anyhow, there might be cases where cached data becomes invalid. |
| 1271 |
Volatile files might be replaced by fixed version on some future |
| 1272 |
download but static package files are never changed upon completion |
| 1273 |
and even incomplete downloads are resumed and keep bad data |
| 1274 |
downloaded before. |
| 1275 |
|
| 1276 |
Usually the damage is only discovered by the client later. The |
| 1277 |
particular file can be located in the cache and replaced manually. |
| 1278 |
And if there are many of them, a mass file check might be needed to |
| 1279 |
clean the mess. Fortunately, there are helpers in cache maintenance |
| 1280 |
interface to automate this process. |
| 1281 |
|
| 1282 |
To start, visit the web control interface and check the options of |
| 1283 |
Expiration task. Enable the check for explicit paths and the check |
| 1284 |
of data contents, then start the expiration. With this parameters, |
| 1285 |
complete files with incorrect checksum are detected. The default |
| 1286 |
action for such files is adding them to a list of damage files. |
| 1287 |
After that, the "Delete damaged files" button in the main web page |
| 1288 |
can be used to remove them (or the Show button to display them |
| 1289 |
first). Alternatively, the checkboxes appearing aside of each damage |
| 1290 |
detection can be used together with the control buttons which appear |
| 1291 |
at the end of the report. And another way of dealing with them is |
| 1292 |
truncating (setting to zero size). This can be done on-the-fly and |
| 1293 |
is enabled by the expiration parameters, or with the appropriate |
| 1294 |
command button in the web interface. |
| 1295 |
|
| 1296 |
NOTE: several index files and related support files can create false |
| 1297 |
positives, i.e. as incomplete or bad files. This usually happens |
| 1298 |
because their volatile contents has changed but the file was not |
| 1299 |
downloaded for a while and another version of it was used instead |
| 1300 |
(like bzip2-compressed instead of gzip-compressed or uncompressed). |
| 1301 |
The default code attempts to detect files with good reasons to stay |
| 1302 |
in the cache and does not mark them as damaged. |
| 1303 |
|
| 1304 |
9.3 Problem: regular expiration action reproducibly aborts |
| 1305 |
|
| 1306 |
A quick investigation of action logs should help identifying the |
| 1307 |
problem. A typical one is a mirror listed somewhere which is not |
| 1308 |
reachable when expiration runs. |
| 1309 |
|
| 1310 |
Unfortunately there is no simple and safe way to solve this. One |
| 1311 |
method is setting the ExAbortOnProblems configuration variable, but |
| 1312 |
this can destroy the whole cache if a bigger problem with index |
| 1313 |
file occurs and this state remains unnoticed for many days until |
| 1314 |
ExTreshold period (see configuration) is over. |
| 1315 |
|
| 1316 |
Another way is listing the index files of the faulty mirrors to |
| 1317 |
a special file. It needs to be stored as "ignore_list" in the |
| 1318 |
configuration directory and store one path name per line with paths |
| 1319 |
relative to the cache directory, as seen in the error messages. |
| 1320 |
|
| 1321 |
9.4 Problem: cacher suddenly terminates, log reports IO errors |
| 1322 |
|
| 1323 |
For simplicity and memory saving reasons, apt-cacher-ng assumes that |
| 1324 |
some files can be opened exclusively for reading and they don't |
| 1325 |
suddenly become unreadable. Unfortunatelly, in some conditions and |
| 1326 |
in case of IO errors, this doesn't work and the file systems sends a |
| 1327 |
fatal signal and eventually terminates the program. |
| 1328 |
|
| 1329 |
To track this down to the likely reasons, it's possible to execute a |
| 1330 |
custom script in the moment of the reception of the fatal signal. To |
| 1331 |
do that, add something like this to the configuration: |
| 1332 |
|
| 1333 |
BusAction = ls -l /proc/$PPID/ | mail -s SIGBUS! root |
| 1334 |
|
| 1335 |
This command would send a list of opened files with paths to the |
| 1336 |
"root" mail user, and the root shall check the state of those files |
| 1337 |
(maybe run them through md5sum or similar). |
| 1338 |
|
| 1339 |
NOTE: Beware of the security implications of this configuration |
| 1340 |
option. It runs regular shell code in context of the daemon user in |
| 1341 |
a blocking way, so it may be vulnerable to symlink attacks and it |
| 1342 |
will delay the automatic restart of the daemon through systemd. |
| 1343 |
|
| 1344 |
9.5 Problem: download fails with 503 ... status message |
| 1345 |
|
| 1346 |
Code 503 usually represents an internal failure which could not be |
| 1347 |
described correctly by other HTTP status codes. In the most cases |
| 1348 |
it's caused by file system errors or incorrect cache directory |
| 1349 |
setup, like files or directories with incorrect owner, missing |
| 1350 |
write/read permissions for the effective user account or other |
| 1351 |
system related exceptions like running out of disk space. |
| 1352 |
|
| 1353 |
The log file `apt-cacher.err' located in the `LogDir' directory |
| 1354 |
should document more details. In case it doesn't, setting the |
| 1355 |
`Debug' config option to a higher value might reveal more |
| 1356 |
information. |
| 1357 |
|
| 1358 |
Fixing permission problems shouldn't be a real challenge for system |
| 1359 |
administrators. Usually, a command set like this should do the trick |
| 1360 |
on Debian/Ubuntu systems, assuming that all group users should |
| 1361 |
receive write access to the cache files: |
| 1362 |
|
| 1363 |
chown -R apt-cacher-ng:apt-cacher-ng /var/cache/apt-cacher-ng |
| 1364 |
chmod -R a+rX,g+rw,u+rw /var/cache/apt-cacher-ng |
| 1365 |
|
| 1366 |
9.6 Problem: `apt-get' freezes when downloading files |
| 1367 |
|
| 1368 |
Solution: First, check: |
| 1369 |
|
| 1370 |
- Free disk space and inode usage (`df', `df -i') |
| 1371 |
|
| 1372 |
- Internet connection to the remote sites (browse them via HTTP, |
| 1373 |
e.g. visiting http://ftp.your.mirror) |
| 1374 |
|
| 1375 |
If nothing helps then you may have hit a spooky problem which |
| 1376 |
is hard to track down. If you like, help the author on problem |
| 1377 |
identification. To do that, do: |
| 1378 |
|
| 1379 |
su - |
| 1380 |
# enter root password |
| 1381 |
cd /tmp |
| 1382 |
apt-get source apt-cacher-ng |
| 1383 |
apt-get build-dep apt-cacher-ng |
| 1384 |
cd apt-cacher-ng-* |
| 1385 |
./build.sh DEBUG |
| 1386 |
/etc/init.d/apt-cacher-ng stop |
| 1387 |
builddir/apt-cacher-ng -c /etc/apt-cacher-ng logdir=/tmp foreground=1 debug=7 |
| 1388 |
# (let apt-get run now, on timeouts just wait >> 20 seconds) |
| 1389 |
# stop the daemon with Ctrl-C |
| 1390 |
/etc/init.d/apt-cacher-ng start |
| 1391 |
# compress /tmp/apt-cacher.err and send it to author |
| 1392 |
chown -R apt-cacher-ng:apt-cacher-ng /var/cache/apt-cacher-ng |
| 1393 |
|
| 1394 |
The value of debug can be varied to have different verbosity (see |
| 1395 |
section 9.1 for more information about Debug levels). |
| 1396 |
|
| 1397 |
9.7 `apt-get' reports corrupted bzip2 data |
| 1398 |
|
| 1399 |
Symptoms: apt-get fails to run through "update" no matter what you |
| 1400 |
do. And you may have get a message like this one. |
| 1401 |
|
| 1402 |
99% [6 Packages bzip2 0] [Waiting for headers] [Waiting for headers] |
| 1403 |
bzip2: Data integrity error when decompressing. |
| 1404 |
Input file = (stdin), output file = (stdout) |
| 1405 |
|
| 1406 |
It is possible that the compressed file(s) have become corrupted. |
| 1407 |
You can use the -tvv option to test integrity of such files. |
| 1408 |
|
| 1409 |
You can use the `bzip2recover' program to attempt to recover |
| 1410 |
data from undamaged sections of corrupted files. |
| 1411 |
|
| 1412 |
Err http://debian.netcologne.de unstable/main Packages |
| 1413 |
Sub-process bzip2 returned an error code (2) |
| 1414 |
|
| 1415 |
- This might be one of Apt's problem with insufficient handling |
| 1416 |
of errors, i.e. passing incomplete files to bzip2 on premature |
| 1417 |
connection termination. Retry the update and it might work. |
| 1418 |
|
| 1419 |
- Another issue is more severe: old versions of apt-cacher-ng had |
| 1420 |
a bug which could cause data corruption while resuming downloads |
| 1421 |
however this problem appears only in unusual conditions. To |
| 1422 |
make sure there are no broken files in your repository, run the |
| 1423 |
Expiration task with content verification enabled, and also |
| 1424 |
"immediate deletion" (or delete later after checking the list). |
| 1425 |
See section 7.1.1 for details. |
| 1426 |
|
| 1427 |
9.8 Problem: `apt-cacher-ng' refuses to start with "Address already in |
| 1428 |
use" |
| 1429 |
|
| 1430 |
Another service is already listening on the port which apt-cacher-ng |
| 1431 |
is configured to use. This might be the apt-cacher daemon which used |
| 1432 |
the same port number by default. To identify the daemon behind that |
| 1433 |
process, use the fuser utility, executing it as root for IPv4 and |
| 1434 |
IPv6 protocol versions. Example: |
| 1435 |
|
| 1436 |
fuser -4 -v -n tcp 3142 |
| 1437 |
fuser -6 -v -n tcp 3142 |
| 1438 |
USER PID ACCESS COMMAND |
| 1439 |
3142/tcp: xwwwfsd 17914 F.... xwwwfsd |
| 1440 |
|
| 1441 |
(where 3142 is the port number from the apt-cacher-ng configuration |
| 1442 |
file). To resolve the collision, reconfigure the other daemon or |
| 1443 |
apt-cacher-ng to use another free port (and reconfigure the clients |
| 1444 |
to use the new apt-cacher-ng port accordingly). |
| 1445 |
|
| 1446 |
Chapter 10: Known Bugs and Limitations |
| 1447 |
-------------------------------------- |
| 1448 |
|
| 1449 |
- Only the "Basic" type of HTTP proxy authentication is supported |
| 1450 |
at the moment |
| 1451 |
|
| 1452 |
- Only HTTP HEAD and GET commands are supported properly. |
| 1453 |
|
| 1454 |
- CONNECT (for https/SSL/TLS tunnels) is supported but needs to be |
| 1455 |
enabled by local administrators, see section 8.2 for details |
| 1456 |
|
| 1457 |
- POST support is restricted to apt-listbugs actions and is |
| 1458 |
actually implemented as simple data forwarding with minor header |
| 1459 |
manipulation. If additional proxy is configured, the client will |
| 1460 |
be redirected to the HTTPS version of bugs.debian.org service, |
| 1461 |
i.e. switching to the CONNECT method (see above) |
| 1462 |
|
| 1463 |
- Transparent proxy mode not implemented yet |
| 1464 |
|
| 1465 |
- The metadata update in maintenance jobs (like expiration task) |
| 1466 |
attempts to guess the best source of update based on original |
| 1467 |
source information and current backend configuration. In case |
| 1468 |
of repeating failures, please run a "forced" download of index |
| 1469 |
files through the web interface |
| 1470 |
|
| 1471 |
- See TODO file in apt-cacher-ng source for more related |
| 1472 |
information |
| 1473 |
|
| 1474 |
Chapter 11: Contact |
| 1475 |
------------------- |
| 1476 |
|
| 1477 |
The planned features are listed in the file TODO. Don't hesitate to |
| 1478 |
contact the author if you really need something not found there and |
| 1479 |
you can explain the severity of your request. |
| 1480 |
|
| 1481 |
There are also a public mailing list and request trackers available |
| 1482 |
on the Alioth project page. |
| 1483 |
|
| 1484 |
And last, but not least: feel free to express your gratitude by |
| 1485 |
donating a small amount of money via PayPal (to [email protected]). |
| 1486 |
|
| 1487 |
[Eduard Bloch, Sun, 19 Apr 2015 10:25:49 +0200] |