| 1 |
.\" -*- mode: troff; coding: utf-8 -*- |
| 2 |
.\" Automatically generated by Pod::Man 5.0102 (Pod::Simple 3.45) |
| 3 |
.\" |
| 4 |
.\" Standard preamble: |
| 5 |
.\" ======================================================================== |
| 6 |
.de Sp \" Vertical space (when we can't use .PP) |
| 7 |
.if t .sp .5v |
| 8 |
.if n .sp |
| 9 |
.. |
| 10 |
.de Vb \" Begin verbatim text |
| 11 |
.ft CW |
| 12 |
.nf |
| 13 |
.ne \\$1 |
| 14 |
.. |
| 15 |
.de Ve \" End verbatim text |
| 16 |
.ft R |
| 17 |
.fi |
| 18 |
.. |
| 19 |
.\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>. |
| 20 |
.ie n \{\ |
| 21 |
. ds C` "" |
| 22 |
. ds C' "" |
| 23 |
'br\} |
| 24 |
.el\{\ |
| 25 |
. ds C` |
| 26 |
. ds C' |
| 27 |
'br\} |
| 28 |
.\" |
| 29 |
.\" Escape single quotes in literal strings from groff's Unicode transform. |
| 30 |
.ie \n(.g .ds Aq \(aq |
| 31 |
.el .ds Aq ' |
| 32 |
.\" |
| 33 |
.\" If the F register is >0, we'll generate index entries on stderr for |
| 34 |
.\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index |
| 35 |
.\" entries marked with X<> in POD. Of course, you'll have to process the |
| 36 |
.\" output yourself in some meaningful fashion. |
| 37 |
.\" |
| 38 |
.\" Avoid warning from groff about undefined register 'F'. |
| 39 |
.de IX |
| 40 |
.. |
| 41 |
.nr rF 0 |
| 42 |
.if \n(.g .if rF .nr rF 1 |
| 43 |
.if (\n(rF:(\n(.g==0)) \{\ |
| 44 |
. if \nF \{\ |
| 45 |
. de IX |
| 46 |
. tm Index:\\$1\t\\n%\t"\\$2" |
| 47 |
.. |
| 48 |
. if !\nF==2 \{\ |
| 49 |
. nr % 0 |
| 50 |
. nr F 2 |
| 51 |
. \} |
| 52 |
. \} |
| 53 |
.\} |
| 54 |
.rr rF |
| 55 |
.\" ======================================================================== |
| 56 |
.\" |
| 57 |
.IX Title "WGET 1" |
| 58 |
.TH WGET 1 2025-03-08 "GNU Wget 1.25.0" "GNU Wget" |
| 59 |
.\" For nroff, turn off justification. Always turn off hyphenation; it makes |
| 60 |
.\" way too many mistakes in technical documents. |
| 61 |
.if n .ad l |
| 62 |
.nh |
| 63 |
.SH NAME |
| 64 |
Wget \- The non\-interactive network downloader. |
| 65 |
.SH SYNOPSIS |
| 66 |
.IX Header "SYNOPSIS" |
| 67 |
wget [\fIoption\fR]... [\fIURL\fR]... |
| 68 |
.SH DESCRIPTION |
| 69 |
.IX Header "DESCRIPTION" |
| 70 |
GNU Wget is a free utility for non-interactive download of files from |
| 71 |
the Web. It supports HTTP, HTTPS, and FTP protocols, as |
| 72 |
well as retrieval through HTTP proxies. |
| 73 |
.PP |
| 74 |
Wget is non-interactive, meaning that it can work in the background, |
| 75 |
while the user is not logged on. This allows you to start a retrieval |
| 76 |
and disconnect from the system, letting Wget finish the work. By |
| 77 |
contrast, most of the Web browsers require constant user's presence, |
| 78 |
which can be a great hindrance when transferring a lot of data. |
| 79 |
.PP |
| 80 |
Wget can follow links in HTML, XHTML, and CSS pages, to |
| 81 |
create local versions of remote web sites, fully recreating the |
| 82 |
directory structure of the original site. This is sometimes referred to |
| 83 |
as "recursive downloading." While doing that, Wget respects the Robot |
| 84 |
Exclusion Standard (\fI/robots.txt\fR). Wget can be instructed to |
| 85 |
convert the links in downloaded files to point at the local files, for |
| 86 |
offline viewing. |
| 87 |
.PP |
| 88 |
Wget has been designed for robustness over slow or unstable network |
| 89 |
connections; if a download fails due to a network problem, it will |
| 90 |
keep retrying until the whole file has been retrieved. If the server |
| 91 |
supports regetting, it will instruct the server to continue the |
| 92 |
download from where it left off. |
| 93 |
.SH OPTIONS |
| 94 |
.IX Header "OPTIONS" |
| 95 |
.SS "Option Syntax" |
| 96 |
.IX Subsection "Option Syntax" |
| 97 |
Since Wget uses GNU getopt to process command-line arguments, every |
| 98 |
option has a long form along with the short one. Long options are |
| 99 |
more convenient to remember, but take time to type. You may freely |
| 100 |
mix different option styles, or specify options after the command-line |
| 101 |
arguments. Thus you may write: |
| 102 |
.PP |
| 103 |
.Vb 1 |
| 104 |
\& wget \-r \-\-tries=10 http://fly.srk.fer.hr/ \-o log |
| 105 |
.Ve |
| 106 |
.PP |
| 107 |
The space between the option accepting an argument and the argument may |
| 108 |
be omitted. Instead of \fB\-o log\fR you can write \fB\-olog\fR. |
| 109 |
.PP |
| 110 |
You may put several options that do not require arguments together, |
| 111 |
like: |
| 112 |
.PP |
| 113 |
.Vb 1 |
| 114 |
\& wget \-drc <URL> |
| 115 |
.Ve |
| 116 |
.PP |
| 117 |
This is completely equivalent to: |
| 118 |
.PP |
| 119 |
.Vb 1 |
| 120 |
\& wget \-d \-r \-c <URL> |
| 121 |
.Ve |
| 122 |
.PP |
| 123 |
Since the options can be specified after the arguments, you may |
| 124 |
terminate them with \fB\-\-\fR. So the following will try to download |
| 125 |
URL \fB\-x\fR, reporting failure to \fIlog\fR: |
| 126 |
.PP |
| 127 |
.Vb 1 |
| 128 |
\& wget \-o log \-\- \-x |
| 129 |
.Ve |
| 130 |
.PP |
| 131 |
The options that accept comma-separated lists all respect the convention |
| 132 |
that specifying an empty list clears its value. This can be useful to |
| 133 |
clear the \fI.wgetrc\fR settings. For instance, if your \fI.wgetrc\fR |
| 134 |
sets \f(CW\*(C`exclude_directories\*(C'\fR to \fI/cgi\-bin\fR, the following |
| 135 |
example will first reset it, and then set it to exclude \fI/~nobody\fR |
| 136 |
and \fI/~somebody\fR. You can also clear the lists in \fI.wgetrc\fR. |
| 137 |
.PP |
| 138 |
.Vb 1 |
| 139 |
\& wget \-X "" \-X /~nobody,/~somebody |
| 140 |
.Ve |
| 141 |
.PP |
| 142 |
Most options that do not accept arguments are \fIboolean\fR options, |
| 143 |
so named because their state can be captured with a yes-or-no |
| 144 |
("boolean") variable. For example, \fB\-\-follow\-ftp\fR tells Wget |
| 145 |
to follow FTP links from HTML files and, on the other hand, |
| 146 |
\&\fB\-\-no\-glob\fR tells it not to perform file globbing on FTP URLs. A |
| 147 |
boolean option is either \fIaffirmative\fR or \fInegative\fR |
| 148 |
(beginning with \fB\-\-no\fR). All such options share several |
| 149 |
properties. |
| 150 |
.PP |
| 151 |
Unless stated otherwise, it is assumed that the default behavior is |
| 152 |
the opposite of what the option accomplishes. For example, the |
| 153 |
documented existence of \fB\-\-follow\-ftp\fR assumes that the default |
| 154 |
is to \fInot\fR follow FTP links from HTML pages. |
| 155 |
.PP |
| 156 |
Affirmative options can be negated by prepending the \fB\-\-no\-\fR to |
| 157 |
the option name; negative options can be negated by omitting the |
| 158 |
\&\fB\-\-no\-\fR prefix. This might seem superfluous\-\-\-if the default for |
| 159 |
an affirmative option is to not do something, then why provide a way |
| 160 |
to explicitly turn it off? But the startup file may in fact change |
| 161 |
the default. For instance, using \f(CW\*(C`follow_ftp = on\*(C'\fR in |
| 162 |
\&\fI.wgetrc\fR makes Wget \fIfollow\fR FTP links by default, and |
| 163 |
using \fB\-\-no\-follow\-ftp\fR is the only way to restore the factory |
| 164 |
default from the command line. |
| 165 |
.SS "Basic Startup Options" |
| 166 |
.IX Subsection "Basic Startup Options" |
| 167 |
.IP \fB\-V\fR 4 |
| 168 |
.IX Item "-V" |
| 169 |
.PD 0 |
| 170 |
.IP \fB\-\-version\fR 4 |
| 171 |
.IX Item "--version" |
| 172 |
.PD |
| 173 |
Display the version of Wget. |
| 174 |
.IP \fB\-h\fR 4 |
| 175 |
.IX Item "-h" |
| 176 |
.PD 0 |
| 177 |
.IP \fB\-\-help\fR 4 |
| 178 |
.IX Item "--help" |
| 179 |
.PD |
| 180 |
Print a help message describing all of Wget's command-line options. |
| 181 |
.IP \fB\-b\fR 4 |
| 182 |
.IX Item "-b" |
| 183 |
.PD 0 |
| 184 |
.IP \fB\-\-background\fR 4 |
| 185 |
.IX Item "--background" |
| 186 |
.PD |
| 187 |
Go to background immediately after startup. If no output file is |
| 188 |
specified via the \fB\-o\fR, output is redirected to \fIwget-log\fR. |
| 189 |
.IP "\fB\-e\fR \fIcommand\fR" 4 |
| 190 |
.IX Item "-e command" |
| 191 |
.PD 0 |
| 192 |
.IP "\fB\-\-execute\fR \fIcommand\fR" 4 |
| 193 |
.IX Item "--execute command" |
| 194 |
.PD |
| 195 |
Execute \fIcommand\fR as if it were a part of \fI.wgetrc\fR. A command thus invoked will be executed |
| 196 |
\&\fIafter\fR the commands in \fI.wgetrc\fR, thus taking precedence over |
| 197 |
them. If you need to specify more than one wgetrc command, use multiple |
| 198 |
instances of \fB\-e\fR. |
| 199 |
.SS "Logging and Input File Options" |
| 200 |
.IX Subsection "Logging and Input File Options" |
| 201 |
.IP "\fB\-o\fR \fIlogfile\fR" 4 |
| 202 |
.IX Item "-o logfile" |
| 203 |
.PD 0 |
| 204 |
.IP \fB\-\-output\-file=\fR\fIlogfile\fR 4 |
| 205 |
.IX Item "--output-file=logfile" |
| 206 |
.PD |
| 207 |
Log all messages to \fIlogfile\fR. The messages are normally reported |
| 208 |
to standard error. |
| 209 |
.IP "\fB\-a\fR \fIlogfile\fR" 4 |
| 210 |
.IX Item "-a logfile" |
| 211 |
.PD 0 |
| 212 |
.IP \fB\-\-append\-output=\fR\fIlogfile\fR 4 |
| 213 |
.IX Item "--append-output=logfile" |
| 214 |
.PD |
| 215 |
Append to \fIlogfile\fR. This is the same as \fB\-o\fR, only it appends |
| 216 |
to \fIlogfile\fR instead of overwriting the old log file. If |
| 217 |
\&\fIlogfile\fR does not exist, a new file is created. |
| 218 |
.IP \fB\-d\fR 4 |
| 219 |
.IX Item "-d" |
| 220 |
.PD 0 |
| 221 |
.IP \fB\-\-debug\fR 4 |
| 222 |
.IX Item "--debug" |
| 223 |
.PD |
| 224 |
Turn on debug output, meaning various information important to the |
| 225 |
developers of Wget if it does not work properly. Your system |
| 226 |
administrator may have chosen to compile Wget without debug support, in |
| 227 |
which case \fB\-d\fR will not work. Please note that compiling with |
| 228 |
debug support is always safe\-\-\-Wget compiled with the debug support will |
| 229 |
\&\fInot\fR print any debug info unless requested with \fB\-d\fR. |
| 230 |
.IP \fB\-q\fR 4 |
| 231 |
.IX Item "-q" |
| 232 |
.PD 0 |
| 233 |
.IP \fB\-\-quiet\fR 4 |
| 234 |
.IX Item "--quiet" |
| 235 |
.PD |
| 236 |
Turn off Wget's output. |
| 237 |
.IP \fB\-v\fR 4 |
| 238 |
.IX Item "-v" |
| 239 |
.PD 0 |
| 240 |
.IP \fB\-\-verbose\fR 4 |
| 241 |
.IX Item "--verbose" |
| 242 |
.PD |
| 243 |
Turn on verbose output, with all the available data. The default output |
| 244 |
is verbose. |
| 245 |
.IP \fB\-nv\fR 4 |
| 246 |
.IX Item "-nv" |
| 247 |
.PD 0 |
| 248 |
.IP \fB\-\-no\-verbose\fR 4 |
| 249 |
.IX Item "--no-verbose" |
| 250 |
.PD |
| 251 |
Turn off verbose without being completely quiet (use \fB\-q\fR for |
| 252 |
that), which means that error messages and basic information still get |
| 253 |
printed. |
| 254 |
.IP \fB\-\-report\-speed=\fR\fItype\fR 4 |
| 255 |
.IX Item "--report-speed=type" |
| 256 |
Output bandwidth as \fItype\fR. The only accepted value is \fBbits\fR. |
| 257 |
.IP "\fB\-i\fR \fIfile\fR" 4 |
| 258 |
.IX Item "-i file" |
| 259 |
.PD 0 |
| 260 |
.IP \fB\-\-input\-file=\fR\fIfile\fR 4 |
| 261 |
.IX Item "--input-file=file" |
| 262 |
.PD |
| 263 |
Read URLs from a local or external \fIfile\fR. If \fB\-\fR is |
| 264 |
specified as \fIfile\fR, URLs are read from the standard input. |
| 265 |
(Use \fB./\-\fR to read from a file literally named \fB\-\fR.) |
| 266 |
.Sp |
| 267 |
If this function is used, no URLs need be present on the command |
| 268 |
line. If there are URLs both on the command line and in an input |
| 269 |
file, those on the command lines will be the first ones to be |
| 270 |
retrieved. If \fB\-\-force\-html\fR is not specified, then \fIfile\fR |
| 271 |
should consist of a series of URLs, one per line. |
| 272 |
.Sp |
| 273 |
However, if you specify \fB\-\-force\-html\fR, the document will be |
| 274 |
regarded as \fBhtml\fR. In that case you may have problems with |
| 275 |
relative links, which you can solve either by adding \f(CW\*(C`<base |
| 276 |
href="\fR\f(CIurl\fR\f(CW">\*(C'\fR to the documents or by specifying |
| 277 |
\&\fB\-\-base=\fR\fIurl\fR on the command line. |
| 278 |
.Sp |
| 279 |
If the \fIfile\fR is an external one, the document will be automatically |
| 280 |
treated as \fBhtml\fR if the Content-Type matches \fBtext/html\fR. |
| 281 |
Furthermore, the \fIfile\fR's location will be implicitly used as base |
| 282 |
href if none was specified. |
| 283 |
.Sp |
| 284 |
If the \fIfile\fR is a local file, on systems that support it, it will be |
| 285 |
opened with the \f(CW\*(C`O_NONBLOCK\*(C'\fR flag to allow non-blocking reads from the |
| 286 |
file. \fBWget\fR will attempt to continue reading from the file until |
| 287 |
EOF is reached or it is closed. This allows one to stream a list of files |
| 288 |
for retrieval where the list is dynamically generated during the execution |
| 289 |
of the process. Currently, this feature is not available on Windows platforms. |
| 290 |
.IP \fB\-\-input\-metalink=\fR\fIfile\fR 4 |
| 291 |
.IX Item "--input-metalink=file" |
| 292 |
Downloads files covered in local Metalink \fIfile\fR. Metalink version 3 |
| 293 |
and 4 are supported. |
| 294 |
.IP \fB\-\-keep\-badhash\fR 4 |
| 295 |
.IX Item "--keep-badhash" |
| 296 |
Keeps downloaded Metalink's files with a bad hash. It appends .badhash |
| 297 |
to the name of Metalink's files which have a checksum mismatch, except |
| 298 |
without overwriting existing files. |
| 299 |
.IP \fB\-\-metalink\-over\-http\fR 4 |
| 300 |
.IX Item "--metalink-over-http" |
| 301 |
Issues HTTP HEAD request instead of GET and extracts Metalink metadata |
| 302 |
from response headers. Then it switches to Metalink download. |
| 303 |
If no valid Metalink metadata is found, it falls back to ordinary HTTP download. |
| 304 |
Enables \fBContent-Type: application/metalink4+xml\fR files download/processing. |
| 305 |
.IP \fB\-\-metalink\-index=\fR\fInumber\fR 4 |
| 306 |
.IX Item "--metalink-index=number" |
| 307 |
Set the Metalink \fBapplication/metalink4+xml\fR metaurl ordinal |
| 308 |
NUMBER. From 1 to the total number of "application/metalink4+xml" |
| 309 |
available. Specify 0 or \fBinf\fR to choose the first good one. |
| 310 |
Metaurls, such as those from a \fB\-\-metalink\-over\-http\fR, may have |
| 311 |
been sorted by priority key's value; keep this in mind to choose the |
| 312 |
right NUMBER. |
| 313 |
.IP \fB\-\-preferred\-location\fR 4 |
| 314 |
.IX Item "--preferred-location" |
| 315 |
Set preferred location for Metalink resources. This has effect if multiple |
| 316 |
resources with same priority are available. |
| 317 |
.IP \fB\-\-xattr\fR 4 |
| 318 |
.IX Item "--xattr" |
| 319 |
Enable use of file system's extended attributes to save the |
| 320 |
original URL and the Referer HTTP header value if used. |
| 321 |
.Sp |
| 322 |
Be aware that the URL might contain private information like |
| 323 |
access tokens or credentials. |
| 324 |
.IP \fB\-F\fR 4 |
| 325 |
.IX Item "-F" |
| 326 |
.PD 0 |
| 327 |
.IP \fB\-\-force\-html\fR 4 |
| 328 |
.IX Item "--force-html" |
| 329 |
.PD |
| 330 |
When input is read from a file, force it to be treated as an HTML |
| 331 |
file. This enables you to retrieve relative links from existing |
| 332 |
HTML files on your local disk, by adding \f(CW\*(C`<base |
| 333 |
href="\fR\f(CIurl\fR\f(CW">\*(C'\fR to HTML, or using the \fB\-\-base\fR command-line |
| 334 |
option. |
| 335 |
.IP "\fB\-B\fR \fIURL\fR" 4 |
| 336 |
.IX Item "-B URL" |
| 337 |
.PD 0 |
| 338 |
.IP \fB\-\-base=\fR\fIURL\fR 4 |
| 339 |
.IX Item "--base=URL" |
| 340 |
.PD |
| 341 |
Resolves relative links using \fIURL\fR as the point of reference, |
| 342 |
when reading links from an HTML file specified via the |
| 343 |
\&\fB\-i\fR/\fB\-\-input\-file\fR option (together with |
| 344 |
\&\fB\-\-force\-html\fR, or when the input file was fetched remotely from |
| 345 |
a server describing it as HTML). This is equivalent to the |
| 346 |
presence of a \f(CW\*(C`BASE\*(C'\fR tag in the HTML input file, with |
| 347 |
\&\fIURL\fR as the value for the \f(CW\*(C`href\*(C'\fR attribute. |
| 348 |
.Sp |
| 349 |
For instance, if you specify \fBhttp://foo/bar/a.html\fR for |
| 350 |
\&\fIURL\fR, and Wget reads \fB../baz/b.html\fR from the input file, it |
| 351 |
would be resolved to \fBhttp://foo/baz/b.html\fR. |
| 352 |
.IP \fB\-\-config=\fR\fIFILE\fR 4 |
| 353 |
.IX Item "--config=FILE" |
| 354 |
Specify the location of a startup file you wish to use instead of the |
| 355 |
default one(s). Use \-\-no\-config to disable reading of config files. |
| 356 |
If both \-\-config and \-\-no\-config are given, \-\-no\-config is ignored. |
| 357 |
.IP \fB\-\-rejected\-log=\fR\fIlogfile\fR 4 |
| 358 |
.IX Item "--rejected-log=logfile" |
| 359 |
Logs all URL rejections to \fIlogfile\fR as comma separated values. The values |
| 360 |
include the reason of rejection, the URL and the parent URL it was found in. |
| 361 |
.SS "Download Options" |
| 362 |
.IX Subsection "Download Options" |
| 363 |
.IP \fB\-\-bind\-address=\fR\fIADDRESS\fR 4 |
| 364 |
.IX Item "--bind-address=ADDRESS" |
| 365 |
When making client TCP/IP connections, bind to \fIADDRESS\fR on |
| 366 |
the local machine. \fIADDRESS\fR may be specified as a hostname or IP |
| 367 |
address. This option can be useful if your machine is bound to multiple |
| 368 |
IPs. |
| 369 |
.IP \fB\-\-bind\-dns\-address=\fR\fIADDRESS\fR 4 |
| 370 |
.IX Item "--bind-dns-address=ADDRESS" |
| 371 |
[libcares only] |
| 372 |
This address overrides the route for DNS requests. If you ever need to |
| 373 |
circumvent the standard settings from /etc/resolv.conf, this option together |
| 374 |
with \fB\-\-dns\-servers\fR is your friend. |
| 375 |
\&\fIADDRESS\fR must be specified either as IPv4 or IPv6 address. |
| 376 |
Wget needs to be built with libcares for this option to be available. |
| 377 |
.IP \fB\-\-dns\-servers=\fR\fIADDRESSES\fR 4 |
| 378 |
.IX Item "--dns-servers=ADDRESSES" |
| 379 |
[libcares only] |
| 380 |
The given address(es) override the standard nameserver |
| 381 |
addresses, e.g. as configured in /etc/resolv.conf. |
| 382 |
\&\fIADDRESSES\fR may be specified either as IPv4 or IPv6 addresses, |
| 383 |
comma-separated. |
| 384 |
Wget needs to be built with libcares for this option to be available. |
| 385 |
.IP "\fB\-t\fR \fInumber\fR" 4 |
| 386 |
.IX Item "-t number" |
| 387 |
.PD 0 |
| 388 |
.IP \fB\-\-tries=\fR\fInumber\fR 4 |
| 389 |
.IX Item "--tries=number" |
| 390 |
.PD |
| 391 |
Set number of tries to \fInumber\fR. Specify 0 or \fBinf\fR for |
| 392 |
infinite retrying. The default is to retry 20 times, with the exception |
| 393 |
of fatal errors like "connection refused" or "not found" (404), |
| 394 |
which are not retried. |
| 395 |
.IP "\fB\-O\fR \fIfile\fR" 4 |
| 396 |
.IX Item "-O file" |
| 397 |
.PD 0 |
| 398 |
.IP \fB\-\-output\-document=\fR\fIfile\fR 4 |
| 399 |
.IX Item "--output-document=file" |
| 400 |
.PD |
| 401 |
The documents will not be written to the appropriate files, but all |
| 402 |
will be concatenated together and written to \fIfile\fR. If \fB\-\fR |
| 403 |
is used as \fIfile\fR, documents will be printed to standard output, |
| 404 |
disabling link conversion. (Use \fB./\-\fR to print to a file |
| 405 |
literally named \fB\-\fR.) |
| 406 |
.Sp |
| 407 |
Use of \fB\-O\fR is \fInot\fR intended to mean simply "use the name |
| 408 |
\&\fIfile\fR instead of the one in the URL;" rather, it is |
| 409 |
analogous to shell redirection: |
| 410 |
\&\fBwget \-O file http://foo\fR is intended to work like |
| 411 |
\&\fBwget \-O \- http://foo > file\fR; \fIfile\fR will be truncated |
| 412 |
immediately, and \fIall\fR downloaded content will be written there. |
| 413 |
.Sp |
| 414 |
For this reason, \fB\-N\fR (for timestamp-checking) is not supported |
| 415 |
in combination with \fB\-O\fR: since \fIfile\fR is always newly |
| 416 |
created, it will always have a very new timestamp. A warning will be |
| 417 |
issued if this combination is used. |
| 418 |
.Sp |
| 419 |
Similarly, using \fB\-r\fR or \fB\-p\fR with \fB\-O\fR may not work as |
| 420 |
you expect: Wget won't just download the first file to \fIfile\fR and |
| 421 |
then download the rest to their normal names: \fIall\fR downloaded |
| 422 |
content will be placed in \fIfile\fR. This was disabled in version |
| 423 |
1.11, but has been reinstated (with a warning) in 1.11.2, as there are |
| 424 |
some cases where this behavior can actually have some use. |
| 425 |
.Sp |
| 426 |
A combination with \fB\-nc\fR is only accepted if the given output |
| 427 |
file does not exist. |
| 428 |
.Sp |
| 429 |
Note that a combination with \fB\-k\fR is only permitted when |
| 430 |
downloading a single document, as in that case it will just convert |
| 431 |
all relative URIs to external ones; \fB\-k\fR makes no sense for |
| 432 |
multiple URIs when they're all being downloaded to a single file; |
| 433 |
\&\fB\-k\fR can be used only when the output is a regular file. |
| 434 |
.IP \fB\-nc\fR 4 |
| 435 |
.IX Item "-nc" |
| 436 |
.PD 0 |
| 437 |
.IP \fB\-\-no\-clobber\fR 4 |
| 438 |
.IX Item "--no-clobber" |
| 439 |
.PD |
| 440 |
If a file is downloaded more than once in the same directory, Wget's |
| 441 |
behavior depends on a few options, including \fB\-nc\fR. In certain |
| 442 |
cases, the local file will be \fIclobbered\fR, or overwritten, upon |
| 443 |
repeated download. In other cases it will be preserved. |
| 444 |
.Sp |
| 445 |
When running Wget without \fB\-N\fR, \fB\-nc\fR, \fB\-r\fR, or |
| 446 |
\&\fB\-p\fR, downloading the same file in the same directory will result |
| 447 |
in the original copy of \fIfile\fR being preserved and the second copy |
| 448 |
being named \fIfile\fR\fB.1\fR. If that file is downloaded yet |
| 449 |
again, the third copy will be named \fIfile\fR\fB.2\fR, and so on. |
| 450 |
(This is also the behavior with \fB\-nd\fR, even if \fB\-r\fR or |
| 451 |
\&\fB\-p\fR are in effect.) When \fB\-nc\fR is specified, this behavior |
| 452 |
is suppressed, and Wget will refuse to download newer copies of |
| 453 |
\&\fIfile\fR. Therefore, "\f(CW\*(C`no\-clobber\*(C'\fR" is actually a |
| 454 |
misnomer in this mode\-\-\-it's not clobbering that's prevented (as the |
| 455 |
numeric suffixes were already preventing clobbering), but rather the |
| 456 |
multiple version saving that's prevented. |
| 457 |
.Sp |
| 458 |
When running Wget with \fB\-r\fR or \fB\-p\fR, but without \fB\-N\fR, |
| 459 |
\&\fB\-nd\fR, or \fB\-nc\fR, re-downloading a file will result in the |
| 460 |
new copy simply overwriting the old. Adding \fB\-nc\fR will prevent |
| 461 |
this behavior, instead causing the original version to be preserved |
| 462 |
and any newer copies on the server to be ignored. |
| 463 |
.Sp |
| 464 |
When running Wget with \fB\-N\fR, with or without \fB\-r\fR or |
| 465 |
\&\fB\-p\fR, the decision as to whether or not to download a newer copy |
| 466 |
of a file depends on the local and remote timestamp and size of the |
| 467 |
file. \fB\-nc\fR may not be specified at the |
| 468 |
same time as \fB\-N\fR. |
| 469 |
.Sp |
| 470 |
A combination with \fB\-O\fR/\fB\-\-output\-document\fR is only accepted |
| 471 |
if the given output file does not exist. |
| 472 |
.Sp |
| 473 |
Note that when \fB\-nc\fR is specified, files with the suffixes |
| 474 |
\&\fB.html\fR or \fB.htm\fR will be loaded from the local disk and |
| 475 |
parsed as if they had been retrieved from the Web. |
| 476 |
.IP \fB\-\-backups=\fR\fIbackups\fR 4 |
| 477 |
.IX Item "--backups=backups" |
| 478 |
Before (over)writing a file, back up an existing file by adding a |
| 479 |
\&\fB.1\fR suffix (\fB_1\fR on VMS) to the file name. Such backup |
| 480 |
files are rotated to \fB.2\fR, \fB.3\fR, and so on, up to |
| 481 |
\&\fIbackups\fR (and lost beyond that). |
| 482 |
.IP \fB\-\-no\-netrc\fR 4 |
| 483 |
.IX Item "--no-netrc" |
| 484 |
Do not try to obtain credentials from \fI.netrc\fR file. By default |
| 485 |
\&\fI.netrc\fR file is searched for credentials in case none have been |
| 486 |
passed on command line and authentication is required. |
| 487 |
.IP \fB\-c\fR 4 |
| 488 |
.IX Item "-c" |
| 489 |
.PD 0 |
| 490 |
.IP \fB\-\-continue\fR 4 |
| 491 |
.IX Item "--continue" |
| 492 |
.PD |
| 493 |
Continue getting a partially-downloaded file. This is useful when you |
| 494 |
want to finish up a download started by a previous instance of Wget, or |
| 495 |
by another program. For instance: |
| 496 |
.Sp |
| 497 |
.Vb 1 |
| 498 |
\& wget \-c ftp://sunsite.doc.ic.ac.uk/ls\-lR.Z |
| 499 |
.Ve |
| 500 |
.Sp |
| 501 |
If there is a file named \fIls\-lR.Z\fR in the current directory, Wget |
| 502 |
will assume that it is the first portion of the remote file, and will |
| 503 |
ask the server to continue the retrieval from an offset equal to the |
| 504 |
length of the local file. |
| 505 |
.Sp |
| 506 |
Note that you don't need to specify this option if you just want the |
| 507 |
current invocation of Wget to retry downloading a file should the |
| 508 |
connection be lost midway through. This is the default behavior. |
| 509 |
\&\fB\-c\fR only affects resumption of downloads started \fIprior\fR to |
| 510 |
this invocation of Wget, and whose local files are still sitting around. |
| 511 |
.Sp |
| 512 |
Without \fB\-c\fR, the previous example would just download the remote |
| 513 |
file to \fIls\-lR.Z.1\fR, leaving the truncated \fIls\-lR.Z\fR file |
| 514 |
alone. |
| 515 |
.Sp |
| 516 |
If you use \fB\-c\fR on a non-empty file, and the server does not support |
| 517 |
continued downloading, Wget will restart the download from scratch and overwrite |
| 518 |
the existing file entirely. |
| 519 |
.Sp |
| 520 |
Beginning with Wget 1.7, if you use \fB\-c\fR on a file which is of |
| 521 |
equal size as the one on the server, Wget will refuse to download the |
| 522 |
file and print an explanatory message. The same happens when the file |
| 523 |
is smaller on the server than locally (presumably because it was changed |
| 524 |
on the server since your last download attempt)\-\-\-because "continuing" |
| 525 |
is not meaningful, no download occurs. |
| 526 |
.Sp |
| 527 |
On the other side of the coin, while using \fB\-c\fR, any file that's |
| 528 |
bigger on the server than locally will be considered an incomplete |
| 529 |
download and only \f(CW\*(C`(length(remote) \- length(local))\*(C'\fR bytes will be |
| 530 |
downloaded and tacked onto the end of the local file. This behavior can |
| 531 |
be desirable in certain cases\-\-\-for instance, you can use \fBwget \-c\fR |
| 532 |
to download just the new portion that's been appended to a data |
| 533 |
collection or log file. |
| 534 |
.Sp |
| 535 |
However, if the file is bigger on the server because it's been |
| 536 |
\&\fIchanged\fR, as opposed to just \fIappended\fR to, you'll end up |
| 537 |
with a garbled file. Wget has no way of verifying that the local file |
| 538 |
is really a valid prefix of the remote file. You need to be especially |
| 539 |
careful of this when using \fB\-c\fR in conjunction with \fB\-r\fR, |
| 540 |
since every file will be considered as an "incomplete download" candidate. |
| 541 |
.Sp |
| 542 |
Another instance where you'll get a garbled file if you try to use |
| 543 |
\&\fB\-c\fR is if you have a lame HTTP proxy that inserts a |
| 544 |
"transfer interrupted" string into the local file. In the future a |
| 545 |
"rollback" option may be added to deal with this case. |
| 546 |
.Sp |
| 547 |
Note that \fB\-c\fR only works with FTP servers and with HTTP |
| 548 |
servers that support the \f(CW\*(C`Range\*(C'\fR header. |
| 549 |
.IP \fB\-\-start\-pos=\fR\fIOFFSET\fR 4 |
| 550 |
.IX Item "--start-pos=OFFSET" |
| 551 |
Start downloading at zero-based position \fIOFFSET\fR. Offset may be expressed |
| 552 |
in bytes, kilobytes with the `k' suffix, or megabytes with the `m' suffix, etc. |
| 553 |
.Sp |
| 554 |
\&\fB\-\-start\-pos\fR has higher precedence over \fB\-\-continue\fR. When |
| 555 |
\&\fB\-\-start\-pos\fR and \fB\-\-continue\fR are both specified, wget will emit a |
| 556 |
warning then proceed as if \fB\-\-continue\fR was absent. |
| 557 |
.Sp |
| 558 |
Server support for continued download is required, otherwise \fB\-\-start\-pos\fR |
| 559 |
cannot help. See \fB\-c\fR for details. |
| 560 |
.IP \fB\-\-progress=\fR\fItype\fR 4 |
| 561 |
.IX Item "--progress=type" |
| 562 |
Select the type of the progress indicator you wish to use. Legal |
| 563 |
indicators are "dot" and "bar". |
| 564 |
.Sp |
| 565 |
The "bar" indicator is used by default. It draws an ASCII progress |
| 566 |
bar graphics (a.k.a "thermometer" display) indicating the status of |
| 567 |
retrieval. If the output is not a TTY, the "dot" bar will be used by |
| 568 |
default. |
| 569 |
.Sp |
| 570 |
Use \fB\-\-progress=dot\fR to switch to the "dot" display. It traces |
| 571 |
the retrieval by printing dots on the screen, each dot representing a |
| 572 |
fixed amount of downloaded data. |
| 573 |
.Sp |
| 574 |
The progress \fItype\fR can also take one or more parameters. The parameters |
| 575 |
vary based on the \fItype\fR selected. Parameters to \fItype\fR are passed by |
| 576 |
appending them to the type sperated by a colon (:) like this: |
| 577 |
\&\fB\-\-progress=\fR\fItype\fR\fB:\fR\fIparameter1\fR\fB:\fR\fIparameter2\fR. |
| 578 |
.Sp |
| 579 |
When using the dotted retrieval, you may set the \fIstyle\fR by |
| 580 |
specifying the type as \fBdot:\fR\fIstyle\fR. Different styles assign |
| 581 |
different meaning to one dot. With the \f(CW\*(C`default\*(C'\fR style each dot |
| 582 |
represents 1K, there are ten dots in a cluster and 50 dots in a line. |
| 583 |
The \f(CW\*(C`binary\*(C'\fR style has a more "computer"\-like orientation\-\-\-8K |
| 584 |
dots, 16\-dots clusters and 48 dots per line (which makes for 384K |
| 585 |
lines). The \f(CW\*(C`mega\*(C'\fR style is suitable for downloading large |
| 586 |
files\-\-\-each dot represents 64K retrieved, there are eight dots in a |
| 587 |
cluster, and 48 dots on each line (so each line contains 3M). |
| 588 |
If \f(CW\*(C`mega\*(C'\fR is not enough then you can use the \f(CW\*(C`giga\*(C'\fR |
| 589 |
style\-\-\-each dot represents 1M retrieved, there are eight dots in a |
| 590 |
cluster, and 32 dots on each line (so each line contains 32M). |
| 591 |
.Sp |
| 592 |
With \fB\-\-progress=bar\fR, there are currently two possible parameters, |
| 593 |
\&\fIforce\fR and \fInoscroll\fR. |
| 594 |
.Sp |
| 595 |
When the output is not a TTY, the progress bar always falls back to "dot", |
| 596 |
even if \fB\-\-progress=bar\fR was passed to Wget during invocation. This |
| 597 |
behaviour can be overridden and the "bar" output forced by using the "force" |
| 598 |
parameter as \fB\-\-progress=bar:force\fR. |
| 599 |
.Sp |
| 600 |
By default, the \fBbar\fR style progress bar scroll the name of the file from |
| 601 |
left to right for the file being downloaded if the filename exceeds the maximum |
| 602 |
length allotted for its display. In certain cases, such as with |
| 603 |
\&\fB\-\-progress=bar:force\fR, one may not want the scrolling filename in the |
| 604 |
progress bar. By passing the "noscroll" parameter, Wget can be forced to |
| 605 |
display as much of the filename as possible without scrolling through it. |
| 606 |
.Sp |
| 607 |
Note that you can set the default style using the \f(CW\*(C`progress\*(C'\fR |
| 608 |
command in \fI.wgetrc\fR. That setting may be overridden from the |
| 609 |
command line. For example, to force the bar output without scrolling, |
| 610 |
use \fB\-\-progress=bar:force:noscroll\fR. |
| 611 |
.IP \fB\-\-show\-progress\fR 4 |
| 612 |
.IX Item "--show-progress" |
| 613 |
Force wget to display the progress bar in any verbosity. |
| 614 |
.Sp |
| 615 |
By default, wget only displays the progress bar in verbose mode. One may |
| 616 |
however, want wget to display the progress bar on screen in conjunction with |
| 617 |
any other verbosity modes like \fB\-\-no\-verbose\fR or \fB\-\-quiet\fR. This |
| 618 |
is often a desired a property when invoking wget to download several small/large |
| 619 |
files. In such a case, wget could simply be invoked with this parameter to get |
| 620 |
a much cleaner output on the screen. |
| 621 |
.Sp |
| 622 |
This option will also force the progress bar to be printed to \fIstderr\fR when |
| 623 |
used alongside the \fB\-\-output\-file\fR option. |
| 624 |
.IP \fB\-N\fR 4 |
| 625 |
.IX Item "-N" |
| 626 |
.PD 0 |
| 627 |
.IP \fB\-\-timestamping\fR 4 |
| 628 |
.IX Item "--timestamping" |
| 629 |
.PD |
| 630 |
Turn on time-stamping. |
| 631 |
.IP \fB\-\-no\-if\-modified\-since\fR 4 |
| 632 |
.IX Item "--no-if-modified-since" |
| 633 |
Do not send If-Modified-Since header in \fB\-N\fR mode. Send preliminary HEAD |
| 634 |
request instead. This has only effect in \fB\-N\fR mode. |
| 635 |
.IP \fB\-\-no\-use\-server\-timestamps\fR 4 |
| 636 |
.IX Item "--no-use-server-timestamps" |
| 637 |
Don't set the local file's timestamp by the one on the server. |
| 638 |
.Sp |
| 639 |
By default, when a file is downloaded, its timestamps are set to |
| 640 |
match those from the remote file. This allows the use of |
| 641 |
\&\fB\-\-timestamping\fR on subsequent invocations of wget. However, it |
| 642 |
is sometimes useful to base the local file's timestamp on when it was |
| 643 |
actually downloaded; for that purpose, the |
| 644 |
\&\fB\-\-no\-use\-server\-timestamps\fR option has been provided. |
| 645 |
.IP \fB\-S\fR 4 |
| 646 |
.IX Item "-S" |
| 647 |
.PD 0 |
| 648 |
.IP \fB\-\-server\-response\fR 4 |
| 649 |
.IX Item "--server-response" |
| 650 |
.PD |
| 651 |
Print the headers sent by HTTP servers and responses sent by |
| 652 |
FTP servers. |
| 653 |
.IP \fB\-\-spider\fR 4 |
| 654 |
.IX Item "--spider" |
| 655 |
When invoked with this option, Wget will behave as a Web \fIspider\fR, |
| 656 |
which means that it will not download the pages, just check that they |
| 657 |
are there. For example, you can use Wget to check your bookmarks: |
| 658 |
.Sp |
| 659 |
.Vb 1 |
| 660 |
\& wget \-\-spider \-\-force\-html \-i bookmarks.html |
| 661 |
.Ve |
| 662 |
.Sp |
| 663 |
This feature needs much more work for Wget to get close to the |
| 664 |
functionality of real web spiders. |
| 665 |
.IP "\fB\-T seconds\fR" 4 |
| 666 |
.IX Item "-T seconds" |
| 667 |
.PD 0 |
| 668 |
.IP \fB\-\-timeout=\fR\fIseconds\fR 4 |
| 669 |
.IX Item "--timeout=seconds" |
| 670 |
.PD |
| 671 |
Set the network timeout to \fIseconds\fR seconds. This is equivalent |
| 672 |
to specifying \fB\-\-dns\-timeout\fR, \fB\-\-connect\-timeout\fR, and |
| 673 |
\&\fB\-\-read\-timeout\fR, all at the same time. |
| 674 |
.Sp |
| 675 |
When interacting with the network, Wget can check for timeout and |
| 676 |
abort the operation if it takes too long. This prevents anomalies |
| 677 |
like hanging reads and infinite connects. The only timeout enabled by |
| 678 |
default is a 900\-second read timeout. Setting a timeout to 0 disables |
| 679 |
it altogether. Unless you know what you are doing, it is best not to |
| 680 |
change the default timeout settings. |
| 681 |
.Sp |
| 682 |
All timeout-related options accept decimal values, as well as |
| 683 |
subsecond values. For example, \fB0.1\fR seconds is a legal (though |
| 684 |
unwise) choice of timeout. Subsecond timeouts are useful for checking |
| 685 |
server response times or for testing network latency. |
| 686 |
.IP \fB\-\-dns\-timeout=\fR\fIseconds\fR 4 |
| 687 |
.IX Item "--dns-timeout=seconds" |
| 688 |
Set the DNS lookup timeout to \fIseconds\fR seconds. DNS lookups that |
| 689 |
don't complete within the specified time will fail. By default, there |
| 690 |
is no timeout on DNS lookups, other than that implemented by system |
| 691 |
libraries. |
| 692 |
.IP \fB\-\-connect\-timeout=\fR\fIseconds\fR 4 |
| 693 |
.IX Item "--connect-timeout=seconds" |
| 694 |
Set the connect timeout to \fIseconds\fR seconds. TCP connections that |
| 695 |
take longer to establish will be aborted. By default, there is no |
| 696 |
connect timeout, other than that implemented by system libraries. |
| 697 |
.IP \fB\-\-read\-timeout=\fR\fIseconds\fR 4 |
| 698 |
.IX Item "--read-timeout=seconds" |
| 699 |
Set the read (and write) timeout to \fIseconds\fR seconds. The |
| 700 |
"time" of this timeout refers to \fIidle time\fR: if, at any point in |
| 701 |
the download, no data is received for more than the specified number |
| 702 |
of seconds, reading fails and the download is restarted. This option |
| 703 |
does not directly affect the duration of the entire download. |
| 704 |
.Sp |
| 705 |
Of course, the remote server may choose to terminate the connection |
| 706 |
sooner than this option requires. The default read timeout is 900 |
| 707 |
seconds. |
| 708 |
.IP \fB\-\-limit\-rate=\fR\fIamount\fR 4 |
| 709 |
.IX Item "--limit-rate=amount" |
| 710 |
Limit the download speed to \fIamount\fR bytes per second. Amount may |
| 711 |
be expressed in bytes, kilobytes with the \fBk\fR suffix, or megabytes |
| 712 |
with the \fBm\fR suffix. For example, \fB\-\-limit\-rate=20k\fR will |
| 713 |
limit the retrieval rate to 20KB/s. This is useful when, for whatever |
| 714 |
reason, you don't want Wget to consume the entire available bandwidth. |
| 715 |
.Sp |
| 716 |
This option allows the use of decimal numbers, usually in conjunction |
| 717 |
with power suffixes; for example, \fB\-\-limit\-rate=2.5k\fR is a legal |
| 718 |
value. |
| 719 |
.Sp |
| 720 |
Note that Wget implements the limiting by sleeping the appropriate |
| 721 |
amount of time after a network read that took less time than specified |
| 722 |
by the rate. Eventually this strategy causes the TCP transfer to slow |
| 723 |
down to approximately the specified rate. However, it may take some |
| 724 |
time for this balance to be achieved, so don't be surprised if limiting |
| 725 |
the rate doesn't work well with very small files. |
| 726 |
.IP "\fB\-w\fR \fIseconds\fR" 4 |
| 727 |
.IX Item "-w seconds" |
| 728 |
.PD 0 |
| 729 |
.IP \fB\-\-wait=\fR\fIseconds\fR 4 |
| 730 |
.IX Item "--wait=seconds" |
| 731 |
.PD |
| 732 |
Wait the specified number of seconds between the retrievals. Use of |
| 733 |
this option is recommended, as it lightens the server load by making the |
| 734 |
requests less frequent. Instead of in seconds, the time can be |
| 735 |
specified in minutes using the \f(CW\*(C`m\*(C'\fR suffix, in hours using \f(CW\*(C`h\*(C'\fR |
| 736 |
suffix, or in days using \f(CW\*(C`d\*(C'\fR suffix. |
| 737 |
.Sp |
| 738 |
Specifying a large value for this option is useful if the network or the |
| 739 |
destination host is down, so that Wget can wait long enough to |
| 740 |
reasonably expect the network error to be fixed before the retry. The |
| 741 |
waiting interval specified by this function is influenced by |
| 742 |
\&\f(CW\*(C`\-\-random\-wait\*(C'\fR, which see. |
| 743 |
.IP \fB\-\-waitretry=\fR\fIseconds\fR 4 |
| 744 |
.IX Item "--waitretry=seconds" |
| 745 |
If you don't want Wget to wait between \fIevery\fR retrieval, but only |
| 746 |
between retries of failed downloads, you can use this option. Wget will |
| 747 |
use \fIlinear backoff\fR, waiting 1 second after the first failure on a |
| 748 |
given file, then waiting 2 seconds after the second failure on that |
| 749 |
file, up to the maximum number of \fIseconds\fR you specify. |
| 750 |
.Sp |
| 751 |
By default, Wget will assume a value of 10 seconds. |
| 752 |
.IP \fB\-\-random\-wait\fR 4 |
| 753 |
.IX Item "--random-wait" |
| 754 |
Some web sites may perform log analysis to identify retrieval programs |
| 755 |
such as Wget by looking for statistically significant similarities in |
| 756 |
the time between requests. This option causes the time between requests |
| 757 |
to vary between 0.5 and 1.5 * \fIwait\fR seconds, where \fIwait\fR was |
| 758 |
specified using the \fB\-\-wait\fR option, in order to mask Wget's |
| 759 |
presence from such analysis. |
| 760 |
.Sp |
| 761 |
A 2001 article in a publication devoted to development on a popular |
| 762 |
consumer platform provided code to perform this analysis on the fly. |
| 763 |
Its author suggested blocking at the class C address level to ensure |
| 764 |
automated retrieval programs were blocked despite changing DHCP-supplied |
| 765 |
addresses. |
| 766 |
.Sp |
| 767 |
The \fB\-\-random\-wait\fR option was inspired by this ill-advised |
| 768 |
recommendation to block many unrelated users from a web site due to the |
| 769 |
actions of one. |
| 770 |
.IP \fB\-\-no\-proxy\fR 4 |
| 771 |
.IX Item "--no-proxy" |
| 772 |
Don't use proxies, even if the appropriate \f(CW*_proxy\fR environment |
| 773 |
variable is defined. |
| 774 |
.IP "\fB\-Q\fR \fIquota\fR" 4 |
| 775 |
.IX Item "-Q quota" |
| 776 |
.PD 0 |
| 777 |
.IP \fB\-\-quota=\fR\fIquota\fR 4 |
| 778 |
.IX Item "--quota=quota" |
| 779 |
.PD |
| 780 |
Specify download quota for automatic retrievals. The value can be |
| 781 |
specified in bytes (default), kilobytes (with \fBk\fR suffix), or |
| 782 |
megabytes (with \fBm\fR suffix). |
| 783 |
.Sp |
| 784 |
Note that quota will never affect downloading a single file. So if you |
| 785 |
specify \fBwget \-Q10k https://example.com/ls\-lR.gz\fR, all of the |
| 786 |
\&\fIls\-lR.gz\fR will be downloaded. The same goes even when several |
| 787 |
URLs are specified on the command-line. The quota is checked only |
| 788 |
at the end of each downloaded file, so it will never result in a partially |
| 789 |
downloaded file. Thus you may safely type \fBwget \-Q2m \-i sites\fR\-\-\-download |
| 790 |
will be aborted after the file that exhausts the quota is completely |
| 791 |
downloaded. |
| 792 |
.Sp |
| 793 |
Setting quota to 0 or to \fBinf\fR unlimits the download quota. |
| 794 |
.IP \fB\-\-no\-dns\-cache\fR 4 |
| 795 |
.IX Item "--no-dns-cache" |
| 796 |
Turn off caching of DNS lookups. Normally, Wget remembers the IP |
| 797 |
addresses it looked up from DNS so it doesn't have to repeatedly |
| 798 |
contact the DNS server for the same (typically small) set of hosts it |
| 799 |
retrieves from. This cache exists in memory only; a new Wget run will |
| 800 |
contact DNS again. |
| 801 |
.Sp |
| 802 |
However, it has been reported that in some situations it is not |
| 803 |
desirable to cache host names, even for the duration of a |
| 804 |
short-running application like Wget. With this option Wget issues a |
| 805 |
new DNS lookup (more precisely, a new call to \f(CW\*(C`gethostbyname\*(C'\fR or |
| 806 |
\&\f(CW\*(C`getaddrinfo\*(C'\fR) each time it makes a new connection. Please note |
| 807 |
that this option will \fInot\fR affect caching that might be |
| 808 |
performed by the resolving library or by an external caching layer, |
| 809 |
such as NSCD. |
| 810 |
.Sp |
| 811 |
If you don't understand exactly what this option does, you probably |
| 812 |
won't need it. |
| 813 |
.IP \fB\-\-restrict\-file\-names=\fR\fImodes\fR 4 |
| 814 |
.IX Item "--restrict-file-names=modes" |
| 815 |
Change which characters found in remote URLs must be escaped during |
| 816 |
generation of local filenames. Characters that are \fIrestricted\fR |
| 817 |
by this option are escaped, i.e. replaced with \fR\f(CB%HH\fR\fB\fR, where |
| 818 |
\&\fBHH\fR is the hexadecimal number that corresponds to the restricted |
| 819 |
character. This option may also be used to force all alphabetical |
| 820 |
cases to be either lower\- or uppercase. |
| 821 |
.Sp |
| 822 |
By default, Wget escapes the characters that are not valid or safe as |
| 823 |
part of file names on your operating system, as well as control |
| 824 |
characters that are typically unprintable. This option is useful for |
| 825 |
changing these defaults, perhaps because you are downloading to a |
| 826 |
non-native partition, or because you want to disable escaping of the |
| 827 |
control characters, or you want to further restrict characters to only |
| 828 |
those in the ASCII range of values. |
| 829 |
.Sp |
| 830 |
The \fImodes\fR are a comma-separated set of text values. The |
| 831 |
acceptable values are \fBunix\fR, \fBwindows\fR, \fBnocontrol\fR, |
| 832 |
\&\fBascii\fR, \fBlowercase\fR, and \fBuppercase\fR. The values |
| 833 |
\&\fBunix\fR and \fBwindows\fR are mutually exclusive (one will |
| 834 |
override the other), as are \fBlowercase\fR and |
| 835 |
\&\fBuppercase\fR. Those last are special cases, as they do not change |
| 836 |
the set of characters that would be escaped, but rather force local |
| 837 |
file paths to be converted either to lower\- or uppercase. |
| 838 |
.Sp |
| 839 |
When "unix" is specified, Wget escapes the character \fB/\fR and |
| 840 |
the control characters in the ranges 0\-\-31 and 128\-\-159. This is the |
| 841 |
default on Unix-like operating systems. |
| 842 |
.Sp |
| 843 |
When "windows" is given, Wget escapes the characters \fB\e\fR, |
| 844 |
\&\fB|\fR, \fB/\fR, \fB:\fR, \fB?\fR, \fB"\fR, \fB*\fR, \fB<\fR, |
| 845 |
\&\fB>\fR, and the control characters in the ranges 0\-\-31 and 128\-\-159. |
| 846 |
In addition to this, Wget in Windows mode uses \fB+\fR instead of |
| 847 |
\&\fB:\fR to separate host and port in local file names, and uses |
| 848 |
\&\fB@\fR instead of \fB?\fR to separate the query portion of the file |
| 849 |
name from the rest. Therefore, a URL that would be saved as |
| 850 |
\&\fBwww.xemacs.org:4300/search.pl?input=blah\fR in Unix mode would be |
| 851 |
saved as \fBwww.xemacs.org+4300/search.pl@input=blah\fR in Windows |
| 852 |
mode. This mode is the default on Windows. |
| 853 |
.Sp |
| 854 |
If you specify \fBnocontrol\fR, then the escaping of the control |
| 855 |
characters is also switched off. This option may make sense |
| 856 |
when you are downloading URLs whose names contain UTF\-8 characters, on |
| 857 |
a system which can save and display filenames in UTF\-8 (some possible |
| 858 |
byte values used in UTF\-8 byte sequences fall in the range of values |
| 859 |
designated by Wget as "controls"). |
| 860 |
.Sp |
| 861 |
The \fBascii\fR mode is used to specify that any bytes whose values |
| 862 |
are outside the range of ASCII characters (that is, greater than |
| 863 |
127) shall be escaped. This can be useful when saving filenames |
| 864 |
whose encoding does not match the one used locally. |
| 865 |
.IP \fB\-4\fR 4 |
| 866 |
.IX Item "-4" |
| 867 |
.PD 0 |
| 868 |
.IP \fB\-\-inet4\-only\fR 4 |
| 869 |
.IX Item "--inet4-only" |
| 870 |
.IP \fB\-6\fR 4 |
| 871 |
.IX Item "-6" |
| 872 |
.IP \fB\-\-inet6\-only\fR 4 |
| 873 |
.IX Item "--inet6-only" |
| 874 |
.PD |
| 875 |
Force connecting to IPv4 or IPv6 addresses. With \fB\-\-inet4\-only\fR |
| 876 |
or \fB\-4\fR, Wget will only connect to IPv4 hosts, ignoring AAAA |
| 877 |
records in DNS, and refusing to connect to IPv6 addresses specified in |
| 878 |
URLs. Conversely, with \fB\-\-inet6\-only\fR or \fB\-6\fR, Wget will |
| 879 |
only connect to IPv6 hosts and ignore A records and IPv4 addresses. |
| 880 |
.Sp |
| 881 |
Neither options should be needed normally. By default, an IPv6\-aware |
| 882 |
Wget will use the address family specified by the host's DNS record. |
| 883 |
If the DNS responds with both IPv4 and IPv6 addresses, Wget will try |
| 884 |
them in sequence until it finds one it can connect to. (Also see |
| 885 |
\&\f(CW\*(C`\-\-prefer\-family\*(C'\fR option described below.) |
| 886 |
.Sp |
| 887 |
These options can be used to deliberately force the use of IPv4 or |
| 888 |
IPv6 address families on dual family systems, usually to aid debugging |
| 889 |
or to deal with broken network configuration. Only one of |
| 890 |
\&\fB\-\-inet6\-only\fR and \fB\-\-inet4\-only\fR may be specified at the |
| 891 |
same time. Neither option is available in Wget compiled without IPv6 |
| 892 |
support. |
| 893 |
.IP \fB\-\-prefer\-family=none/IPv4/IPv6\fR 4 |
| 894 |
.IX Item "--prefer-family=none/IPv4/IPv6" |
| 895 |
When given a choice of several addresses, connect to the addresses |
| 896 |
with specified address family first. The address order returned by |
| 897 |
DNS is used without change by default. |
| 898 |
.Sp |
| 899 |
This avoids spurious errors and connect attempts when accessing hosts |
| 900 |
that resolve to both IPv6 and IPv4 addresses from IPv4 networks. For |
| 901 |
example, \fBwww.kame.net\fR resolves to |
| 902 |
\&\fB2001:200:0:8002:203:47ff:fea5:3085\fR and to |
| 903 |
\&\fB203.178.141.194\fR. When the preferred family is \f(CW\*(C`IPv4\*(C'\fR, the |
| 904 |
IPv4 address is used first; when the preferred family is \f(CW\*(C`IPv6\*(C'\fR, |
| 905 |
the IPv6 address is used first; if the specified value is \f(CW\*(C`none\*(C'\fR, |
| 906 |
the address order returned by DNS is used without change. |
| 907 |
.Sp |
| 908 |
Unlike \fB\-4\fR and \fB\-6\fR, this option doesn't inhibit access to |
| 909 |
any address family, it only changes the \fIorder\fR in which the |
| 910 |
addresses are accessed. Also note that the reordering performed by |
| 911 |
this option is \fIstable\fR\-\-\-it doesn't affect order of addresses of |
| 912 |
the same family. That is, the relative order of all IPv4 addresses |
| 913 |
and of all IPv6 addresses remains intact in all cases. |
| 914 |
.IP \fB\-\-retry\-connrefused\fR 4 |
| 915 |
.IX Item "--retry-connrefused" |
| 916 |
Consider "connection refused" a transient error and try again. |
| 917 |
Normally Wget gives up on a URL when it is unable to connect to the |
| 918 |
site because failure to connect is taken as a sign that the server is |
| 919 |
not running at all and that retries would not help. This option is |
| 920 |
for mirroring unreliable sites whose servers tend to disappear for |
| 921 |
short periods of time. |
| 922 |
.IP \fB\-\-user=\fR\fIuser\fR 4 |
| 923 |
.IX Item "--user=user" |
| 924 |
.PD 0 |
| 925 |
.IP \fB\-\-password=\fR\fIpassword\fR 4 |
| 926 |
.IX Item "--password=password" |
| 927 |
.PD |
| 928 |
Specify the username \fIuser\fR and password \fIpassword\fR for both |
| 929 |
FTP and HTTP file retrieval. These parameters can be overridden |
| 930 |
using the \fB\-\-ftp\-user\fR and \fB\-\-ftp\-password\fR options for |
| 931 |
FTP connections and the \fB\-\-http\-user\fR and \fB\-\-http\-password\fR |
| 932 |
options for HTTP connections. |
| 933 |
.IP \fB\-\-ask\-password\fR 4 |
| 934 |
.IX Item "--ask-password" |
| 935 |
Prompt for a password for each connection established. Cannot be specified |
| 936 |
when \fB\-\-password\fR is being used, because they are mutually exclusive. |
| 937 |
.IP \fB\-\-use\-askpass=\fR\fIcommand\fR 4 |
| 938 |
.IX Item "--use-askpass=command" |
| 939 |
Prompt for a user and password using the specified command. If no command is |
| 940 |
specified then the command in the environment variable WGET_ASKPASS is used. |
| 941 |
If WGET_ASKPASS is not set then the command in the environment variable |
| 942 |
SSH_ASKPASS is used. |
| 943 |
.Sp |
| 944 |
You can set the default command for use-askpass in the \fI.wgetrc\fR. That |
| 945 |
setting may be overridden from the command line. |
| 946 |
.IP \fB\-\-no\-iri\fR 4 |
| 947 |
.IX Item "--no-iri" |
| 948 |
Turn off internationalized URI (IRI) support. Use \fB\-\-iri\fR to |
| 949 |
turn it on. IRI support is activated by default. |
| 950 |
.Sp |
| 951 |
You can set the default state of IRI support using the \f(CW\*(C`iri\*(C'\fR |
| 952 |
command in \fI.wgetrc\fR. That setting may be overridden from the |
| 953 |
command line. |
| 954 |
.IP \fB\-\-local\-encoding=\fR\fIencoding\fR 4 |
| 955 |
.IX Item "--local-encoding=encoding" |
| 956 |
Force Wget to use \fIencoding\fR as the default system encoding. That affects |
| 957 |
how Wget converts URLs specified as arguments from locale to UTF\-8 for |
| 958 |
IRI support. |
| 959 |
.Sp |
| 960 |
Wget use the function \f(CWnl_langinfo()\fR and then the \f(CW\*(C`CHARSET\*(C'\fR |
| 961 |
environment variable to get the locale. If it fails, ASCII is used. |
| 962 |
.Sp |
| 963 |
You can set the default local encoding using the \f(CW\*(C`local_encoding\*(C'\fR |
| 964 |
command in \fI.wgetrc\fR. That setting may be overridden from the |
| 965 |
command line. |
| 966 |
.IP \fB\-\-remote\-encoding=\fR\fIencoding\fR 4 |
| 967 |
.IX Item "--remote-encoding=encoding" |
| 968 |
Force Wget to use \fIencoding\fR as the default remote server encoding. |
| 969 |
That affects how Wget converts URIs found in files from remote encoding |
| 970 |
to UTF\-8 during a recursive fetch. This options is only useful for |
| 971 |
IRI support, for the interpretation of non-ASCII characters. |
| 972 |
.Sp |
| 973 |
For HTTP, remote encoding can be found in HTTP \f(CW\*(C`Content\-Type\*(C'\fR |
| 974 |
header and in HTML \f(CW\*(C`Content\-Type http\-equiv\*(C'\fR meta tag. |
| 975 |
.Sp |
| 976 |
You can set the default encoding using the \f(CW\*(C`remoteencoding\*(C'\fR |
| 977 |
command in \fI.wgetrc\fR. That setting may be overridden from the |
| 978 |
command line. |
| 979 |
.IP \fB\-\-unlink\fR 4 |
| 980 |
.IX Item "--unlink" |
| 981 |
Force Wget to unlink file instead of clobbering existing file. This |
| 982 |
option is useful for downloading to the directory with hardlinks. |
| 983 |
.SS "Directory Options" |
| 984 |
.IX Subsection "Directory Options" |
| 985 |
.IP \fB\-nd\fR 4 |
| 986 |
.IX Item "-nd" |
| 987 |
.PD 0 |
| 988 |
.IP \fB\-\-no\-directories\fR 4 |
| 989 |
.IX Item "--no-directories" |
| 990 |
.PD |
| 991 |
Do not create a hierarchy of directories when retrieving recursively. |
| 992 |
With this option turned on, all files will get saved to the current |
| 993 |
directory, without clobbering (if a name shows up more than once, the |
| 994 |
filenames will get extensions \fB.n\fR). |
| 995 |
.IP \fB\-x\fR 4 |
| 996 |
.IX Item "-x" |
| 997 |
.PD 0 |
| 998 |
.IP \fB\-\-force\-directories\fR 4 |
| 999 |
.IX Item "--force-directories" |
| 1000 |
.PD |
| 1001 |
The opposite of \fB\-nd\fR\-\-\-create a hierarchy of directories, even if |
| 1002 |
one would not have been created otherwise. E.g. \fBwget \-x |
| 1003 |
http://fly.srk.fer.hr/robots.txt\fR will save the downloaded file to |
| 1004 |
\&\fIfly.srk.fer.hr/robots.txt\fR. |
| 1005 |
.IP \fB\-nH\fR 4 |
| 1006 |
.IX Item "-nH" |
| 1007 |
.PD 0 |
| 1008 |
.IP \fB\-\-no\-host\-directories\fR 4 |
| 1009 |
.IX Item "--no-host-directories" |
| 1010 |
.PD |
| 1011 |
Disable generation of host-prefixed directories. By default, invoking |
| 1012 |
Wget with \fB\-r http://fly.srk.fer.hr/\fR will create a structure of |
| 1013 |
directories beginning with \fIfly.srk.fer.hr/\fR. This option disables |
| 1014 |
such behavior. |
| 1015 |
.IP \fB\-\-protocol\-directories\fR 4 |
| 1016 |
.IX Item "--protocol-directories" |
| 1017 |
Use the protocol name as a directory component of local file names. For |
| 1018 |
example, with this option, \fBwget \-r http://\fR\fIhost\fR will save to |
| 1019 |
\&\fBhttp/\fR\fIhost\fR\fB/...\fR rather than just to \fIhost\fR\fB/...\fR. |
| 1020 |
.IP \fB\-\-cut\-dirs=\fR\fInumber\fR 4 |
| 1021 |
.IX Item "--cut-dirs=number" |
| 1022 |
Ignore \fInumber\fR directory components. This is useful for getting a |
| 1023 |
fine-grained control over the directory where recursive retrieval will |
| 1024 |
be saved. |
| 1025 |
.Sp |
| 1026 |
Take, for example, the directory at |
| 1027 |
\&\fBftp://ftp.xemacs.org/pub/xemacs/\fR. If you retrieve it with |
| 1028 |
\&\fB\-r\fR, it will be saved locally under |
| 1029 |
\&\fIftp.xemacs.org/pub/xemacs/\fR. While the \fB\-nH\fR option can |
| 1030 |
remove the \fIftp.xemacs.org/\fR part, you are still stuck with |
| 1031 |
\&\fIpub/xemacs\fR. This is where \fB\-\-cut\-dirs\fR comes in handy; it |
| 1032 |
makes Wget not "see" \fInumber\fR remote directory components. Here |
| 1033 |
are several examples of how \fB\-\-cut\-dirs\fR option works. |
| 1034 |
.Sp |
| 1035 |
.Vb 4 |
| 1036 |
\& No options \-> ftp.xemacs.org/pub/xemacs/ |
| 1037 |
\& \-nH \-> pub/xemacs/ |
| 1038 |
\& \-nH \-\-cut\-dirs=1 \-> xemacs/ |
| 1039 |
\& \-nH \-\-cut\-dirs=2 \-> . |
| 1040 |
\& |
| 1041 |
\& \-\-cut\-dirs=1 \-> ftp.xemacs.org/xemacs/ |
| 1042 |
\& ... |
| 1043 |
.Ve |
| 1044 |
.Sp |
| 1045 |
If you just want to get rid of the directory structure, this option is |
| 1046 |
similar to a combination of \fB\-nd\fR and \fB\-P\fR. However, unlike |
| 1047 |
\&\fB\-nd\fR, \fB\-\-cut\-dirs\fR does not lose with subdirectories\-\-\-for |
| 1048 |
instance, with \fB\-nH \-\-cut\-dirs=1\fR, a \fIbeta/\fR subdirectory will |
| 1049 |
be placed to \fIxemacs/beta\fR, as one would expect. |
| 1050 |
.IP "\fB\-P\fR \fIprefix\fR" 4 |
| 1051 |
.IX Item "-P prefix" |
| 1052 |
.PD 0 |
| 1053 |
.IP \fB\-\-directory\-prefix=\fR\fIprefix\fR 4 |
| 1054 |
.IX Item "--directory-prefix=prefix" |
| 1055 |
.PD |
| 1056 |
Set directory prefix to \fIprefix\fR. The \fIdirectory prefix\fR is the |
| 1057 |
directory where all other files and subdirectories will be saved to, |
| 1058 |
i.e. the top of the retrieval tree. The default is \fB.\fR (the |
| 1059 |
current directory). |
| 1060 |
.SS "HTTP Options" |
| 1061 |
.IX Subsection "HTTP Options" |
| 1062 |
.IP \fB\-\-default\-page=\fR\fIname\fR 4 |
| 1063 |
.IX Item "--default-page=name" |
| 1064 |
Use \fIname\fR as the default file name when it isn't known (i.e., for |
| 1065 |
URLs that end in a slash), instead of \fIindex.html\fR. |
| 1066 |
.IP \fB\-E\fR 4 |
| 1067 |
.IX Item "-E" |
| 1068 |
.PD 0 |
| 1069 |
.IP \fB\-\-adjust\-extension\fR 4 |
| 1070 |
.IX Item "--adjust-extension" |
| 1071 |
.PD |
| 1072 |
If a file of type \fBapplication/xhtml+xml\fR or \fBtext/html\fR is |
| 1073 |
downloaded and the URL does not end with the regexp |
| 1074 |
\&\fB\e.[Hh][Tt][Mm][Ll]?\fR, this option will cause the suffix \fB.html\fR |
| 1075 |
to be appended to the local filename. This is useful, for instance, when |
| 1076 |
you're mirroring a remote site that uses \fB.asp\fR pages, but you want |
| 1077 |
the mirrored pages to be viewable on your stock Apache server. Another |
| 1078 |
good use for this is when you're downloading CGI-generated materials. A URL |
| 1079 |
like \fBhttp://site.com/article.cgi?25\fR will be saved as |
| 1080 |
\&\fIarticle.cgi?25.html\fR. |
| 1081 |
.Sp |
| 1082 |
Note that filenames changed in this way will be re-downloaded every time |
| 1083 |
you re-mirror a site, because Wget can't tell that the local |
| 1084 |
\&\fIX.html\fR file corresponds to remote URL \fIX\fR (since |
| 1085 |
it doesn't yet know that the URL produces output of type |
| 1086 |
\&\fBtext/html\fR or \fBapplication/xhtml+xml\fR. |
| 1087 |
.Sp |
| 1088 |
As of version 1.12, Wget will also ensure that any downloaded files of |
| 1089 |
type \fBtext/css\fR end in the suffix \fB.css\fR, and the option was |
| 1090 |
renamed from \fB\-\-html\-extension\fR, to better reflect its new |
| 1091 |
behavior. The old option name is still acceptable, but should now be |
| 1092 |
considered deprecated. |
| 1093 |
.Sp |
| 1094 |
As of version 1.19.2, Wget will also ensure that any downloaded files with |
| 1095 |
a \f(CW\*(C`Content\-Encoding\*(C'\fR of \fBbr\fR, \fBcompress\fR, \fBdeflate\fR |
| 1096 |
or \fBgzip\fR end in the suffix \fB.br\fR, \fB.Z\fR, \fB.zlib\fR |
| 1097 |
and \fB.gz\fR respectively. |
| 1098 |
.Sp |
| 1099 |
At some point in the future, this option may well be expanded to |
| 1100 |
include suffixes for other types of content, including content types |
| 1101 |
that are not parsed by Wget. |
| 1102 |
.IP \fB\-\-http\-user=\fR\fIuser\fR 4 |
| 1103 |
.IX Item "--http-user=user" |
| 1104 |
.PD 0 |
| 1105 |
.IP \fB\-\-http\-password=\fR\fIpassword\fR 4 |
| 1106 |
.IX Item "--http-password=password" |
| 1107 |
.PD |
| 1108 |
Specify the username \fIuser\fR and password \fIpassword\fR on an |
| 1109 |
HTTP server. According to the type of the challenge, Wget will |
| 1110 |
encode them using either the \f(CW\*(C`basic\*(C'\fR (insecure), |
| 1111 |
the \f(CW\*(C`digest\*(C'\fR, or the Windows \f(CW\*(C`NTLM\*(C'\fR authentication scheme. |
| 1112 |
.Sp |
| 1113 |
Another way to specify username and password is in the URL itself. Either method reveals your password to anyone who |
| 1114 |
bothers to run \f(CW\*(C`ps\*(C'\fR. To prevent the passwords from being seen, |
| 1115 |
use the \fB\-\-use\-askpass\fR or store them in \fI.wgetrc\fR or \fI.netrc\fR, |
| 1116 |
and make sure to protect those files from other users with \f(CW\*(C`chmod\*(C'\fR. If |
| 1117 |
the passwords are really important, do not leave them lying in those files |
| 1118 |
either\-\-\-edit the files and delete them after Wget has started the download. |
| 1119 |
.IP \fB\-\-no\-http\-keep\-alive\fR 4 |
| 1120 |
.IX Item "--no-http-keep-alive" |
| 1121 |
Turn off the "keep-alive" feature for HTTP downloads. Normally, Wget |
| 1122 |
asks the server to keep the connection open so that, when you download |
| 1123 |
more than one document from the same server, they get transferred over |
| 1124 |
the same TCP connection. This saves time and at the same time reduces |
| 1125 |
the load on the server. |
| 1126 |
.Sp |
| 1127 |
This option is useful when, for some reason, persistent (keep-alive) |
| 1128 |
connections don't work for you, for example due to a server bug or due |
| 1129 |
to the inability of server-side scripts to cope with the connections. |
| 1130 |
.IP \fB\-\-no\-cache\fR 4 |
| 1131 |
.IX Item "--no-cache" |
| 1132 |
Disable server-side cache. In this case, Wget will send the remote |
| 1133 |
server appropriate directives (\fBCache-Control: no-cache\fR and |
| 1134 |
\&\fBPragma: no-cache\fR) to get the file from the remote service, |
| 1135 |
rather than returning the cached version. This is especially useful |
| 1136 |
for retrieving and flushing out-of-date documents on proxy servers. |
| 1137 |
.Sp |
| 1138 |
Caching is allowed by default. |
| 1139 |
.IP \fB\-\-no\-cookies\fR 4 |
| 1140 |
.IX Item "--no-cookies" |
| 1141 |
Disable the use of cookies. Cookies are a mechanism for maintaining |
| 1142 |
server-side state. The server sends the client a cookie using the |
| 1143 |
\&\f(CW\*(C`Set\-Cookie\*(C'\fR header, and the client responds with the same cookie |
| 1144 |
upon further requests. Since cookies allow the server owners to keep |
| 1145 |
track of visitors and for sites to exchange this information, some |
| 1146 |
consider them a breach of privacy. The default is to use cookies; |
| 1147 |
however, \fIstoring\fR cookies is not on by default. |
| 1148 |
.IP "\fB\-\-load\-cookies\fR \fIfile\fR" 4 |
| 1149 |
.IX Item "--load-cookies file" |
| 1150 |
Load cookies from \fIfile\fR before the first HTTP retrieval. |
| 1151 |
\&\fIfile\fR is a textual file in the format originally used by Netscape's |
| 1152 |
\&\fIcookies.txt\fR file. |
| 1153 |
.Sp |
| 1154 |
You will typically use this option when mirroring sites that require |
| 1155 |
that you be logged in to access some or all of their content. The login |
| 1156 |
process typically works by the web server issuing an HTTP cookie |
| 1157 |
upon receiving and verifying your credentials. The cookie is then |
| 1158 |
resent by the browser when accessing that part of the site, and so |
| 1159 |
proves your identity. |
| 1160 |
.Sp |
| 1161 |
Mirroring such a site requires Wget to send the same cookies your |
| 1162 |
browser sends when communicating with the site. This is achieved by |
| 1163 |
\&\fB\-\-load\-cookies\fR\-\-\-simply point Wget to the location of the |
| 1164 |
\&\fIcookies.txt\fR file, and it will send the same cookies your browser |
| 1165 |
would send in the same situation. Different browsers keep textual |
| 1166 |
cookie files in different locations: |
| 1167 |
.RS 4 |
| 1168 |
.ie n .IP """Netscape 4.x.""" 4 |
| 1169 |
.el .IP "\f(CWNetscape 4.x.\fR" 4 |
| 1170 |
.IX Item "Netscape 4.x." |
| 1171 |
The cookies are in \fI~/.netscape/cookies.txt\fR. |
| 1172 |
.ie n .IP """Mozilla and Netscape 6.x.""" 4 |
| 1173 |
.el .IP "\f(CWMozilla and Netscape 6.x.\fR" 4 |
| 1174 |
.IX Item "Mozilla and Netscape 6.x." |
| 1175 |
Mozilla's cookie file is also named \fIcookies.txt\fR, located |
| 1176 |
somewhere under \fI~/.mozilla\fR, in the directory of your profile. |
| 1177 |
The full path usually ends up looking somewhat like |
| 1178 |
\&\fI~/.mozilla/default/some-weird-string/cookies.txt\fR. |
| 1179 |
.ie n .IP """Internet Explorer.""" 4 |
| 1180 |
.el .IP "\f(CWInternet Explorer.\fR" 4 |
| 1181 |
.IX Item "Internet Explorer." |
| 1182 |
You can produce a cookie file Wget can use by using the File menu, |
| 1183 |
Import and Export, Export Cookies. This has been tested with Internet |
| 1184 |
Explorer 5; it is not guaranteed to work with earlier versions. |
| 1185 |
.ie n .IP """Other browsers.""" 4 |
| 1186 |
.el .IP "\f(CWOther browsers.\fR" 4 |
| 1187 |
.IX Item "Other browsers." |
| 1188 |
If you are using a different browser to create your cookies, |
| 1189 |
\&\fB\-\-load\-cookies\fR will only work if you can locate or produce a |
| 1190 |
cookie file in the Netscape format that Wget expects. |
| 1191 |
.RE |
| 1192 |
.RS 4 |
| 1193 |
.Sp |
| 1194 |
If you cannot use \fB\-\-load\-cookies\fR, there might still be an |
| 1195 |
alternative. If your browser supports a "cookie manager", you can use |
| 1196 |
it to view the cookies used when accessing the site you're mirroring. |
| 1197 |
Write down the name and value of the cookie, and manually instruct Wget |
| 1198 |
to send those cookies, bypassing the "official" cookie support: |
| 1199 |
.Sp |
| 1200 |
.Vb 1 |
| 1201 |
\& wget \-\-no\-cookies \-\-header "Cookie: <name>=<value>" |
| 1202 |
.Ve |
| 1203 |
.RE |
| 1204 |
.IP "\fB\-\-save\-cookies\fR \fIfile\fR" 4 |
| 1205 |
.IX Item "--save-cookies file" |
| 1206 |
Save cookies to \fIfile\fR before exiting. This will not save cookies |
| 1207 |
that have expired or that have no expiry time (so-called "session |
| 1208 |
cookies"), but also see \fB\-\-keep\-session\-cookies\fR. |
| 1209 |
.IP \fB\-\-keep\-session\-cookies\fR 4 |
| 1210 |
.IX Item "--keep-session-cookies" |
| 1211 |
When specified, causes \fB\-\-save\-cookies\fR to also save session |
| 1212 |
cookies. Session cookies are normally not saved because they are |
| 1213 |
meant to be kept in memory and forgotten when you exit the browser. |
| 1214 |
Saving them is useful on sites that require you to log in or to visit |
| 1215 |
the home page before you can access some pages. With this option, |
| 1216 |
multiple Wget runs are considered a single browser session as far as |
| 1217 |
the site is concerned. |
| 1218 |
.Sp |
| 1219 |
Since the cookie file format does not normally carry session cookies, |
| 1220 |
Wget marks them with an expiry timestamp of 0. Wget's |
| 1221 |
\&\fB\-\-load\-cookies\fR recognizes those as session cookies, but it might |
| 1222 |
confuse other browsers. Also note that cookies so loaded will be |
| 1223 |
treated as other session cookies, which means that if you want |
| 1224 |
\&\fB\-\-save\-cookies\fR to preserve them again, you must use |
| 1225 |
\&\fB\-\-keep\-session\-cookies\fR again. |
| 1226 |
.IP \fB\-\-ignore\-length\fR 4 |
| 1227 |
.IX Item "--ignore-length" |
| 1228 |
Unfortunately, some HTTP servers (CGI programs, to be more |
| 1229 |
precise) send out bogus \f(CW\*(C`Content\-Length\*(C'\fR headers, which makes Wget |
| 1230 |
go wild, as it thinks not all the document was retrieved. You can spot |
| 1231 |
this syndrome if Wget retries getting the same document again and again, |
| 1232 |
each time claiming that the (otherwise normal) connection has closed on |
| 1233 |
the very same byte. |
| 1234 |
.Sp |
| 1235 |
With this option, Wget will ignore the \f(CW\*(C`Content\-Length\*(C'\fR header\-\-\-as |
| 1236 |
if it never existed. |
| 1237 |
.IP \fB\-\-header=\fR\fIheader-line\fR 4 |
| 1238 |
.IX Item "--header=header-line" |
| 1239 |
Send \fIheader-line\fR along with the rest of the headers in each |
| 1240 |
HTTP request. The supplied header is sent as-is, which means it |
| 1241 |
must contain name and value separated by colon, and must not contain |
| 1242 |
newlines. |
| 1243 |
.Sp |
| 1244 |
You may define more than one additional header by specifying |
| 1245 |
\&\fB\-\-header\fR more than once. |
| 1246 |
.Sp |
| 1247 |
.Vb 3 |
| 1248 |
\& wget \-\-header=\*(AqAccept\-Charset: iso\-8859\-2\*(Aq \e |
| 1249 |
\& \-\-header=\*(AqAccept\-Language: hr\*(Aq \e |
| 1250 |
\& http://fly.srk.fer.hr/ |
| 1251 |
.Ve |
| 1252 |
.Sp |
| 1253 |
Specification of an empty string as the header value will clear all |
| 1254 |
previous user-defined headers. |
| 1255 |
.Sp |
| 1256 |
As of Wget 1.10, this option can be used to override headers otherwise |
| 1257 |
generated automatically. This example instructs Wget to connect to |
| 1258 |
localhost, but to specify \fBfoo.bar\fR in the \f(CW\*(C`Host\*(C'\fR header: |
| 1259 |
.Sp |
| 1260 |
.Vb 1 |
| 1261 |
\& wget \-\-header="Host: foo.bar" http://localhost/ |
| 1262 |
.Ve |
| 1263 |
.Sp |
| 1264 |
In versions of Wget prior to 1.10 such use of \fB\-\-header\fR caused |
| 1265 |
sending of duplicate headers. |
| 1266 |
.IP \fB\-\-compression=\fR\fItype\fR 4 |
| 1267 |
.IX Item "--compression=type" |
| 1268 |
Choose the type of compression to be used. Legal values are |
| 1269 |
\&\fBauto\fR, \fBgzip\fR and \fBnone\fR. |
| 1270 |
.Sp |
| 1271 |
If \fBauto\fR or \fBgzip\fR are specified, Wget asks the server to |
| 1272 |
compress the file using the gzip compression format. If the server |
| 1273 |
compresses the file and responds with the \f(CW\*(C`Content\-Encoding\*(C'\fR |
| 1274 |
header field set appropriately, the file will be decompressed |
| 1275 |
automatically. |
| 1276 |
.Sp |
| 1277 |
If \fBnone\fR is specified, wget will not ask the server to compress |
| 1278 |
the file and will not decompress any server responses. This is the default. |
| 1279 |
.Sp |
| 1280 |
Compression support is currently experimental. In case it is turned on, |
| 1281 |
please report any bugs to \f(CW\*(C`bug\[email protected]\*(C'\fR. |
| 1282 |
.IP \fB\-\-max\-redirect=\fR\fInumber\fR 4 |
| 1283 |
.IX Item "--max-redirect=number" |
| 1284 |
Specifies the maximum number of redirections to follow for a resource. |
| 1285 |
The default is 20, which is usually far more than necessary. However, on |
| 1286 |
those occasions where you want to allow more (or fewer), this is the |
| 1287 |
option to use. |
| 1288 |
.IP \fB\-\-proxy\-user=\fR\fIuser\fR 4 |
| 1289 |
.IX Item "--proxy-user=user" |
| 1290 |
.PD 0 |
| 1291 |
.IP \fB\-\-proxy\-password=\fR\fIpassword\fR 4 |
| 1292 |
.IX Item "--proxy-password=password" |
| 1293 |
.PD |
| 1294 |
Specify the username \fIuser\fR and password \fIpassword\fR for |
| 1295 |
authentication on a proxy server. Wget will encode them using the |
| 1296 |
\&\f(CW\*(C`basic\*(C'\fR authentication scheme. |
| 1297 |
.Sp |
| 1298 |
Security considerations similar to those with \fB\-\-http\-password\fR |
| 1299 |
pertain here as well. |
| 1300 |
.IP \fB\-\-referer=\fR\fIurl\fR 4 |
| 1301 |
.IX Item "--referer=url" |
| 1302 |
Include `Referer: \fIurl\fR' header in HTTP request. Useful for |
| 1303 |
retrieving documents with server-side processing that assume they are |
| 1304 |
always being retrieved by interactive web browsers and only come out |
| 1305 |
properly when Referer is set to one of the pages that point to them. |
| 1306 |
.IP \fB\-\-save\-headers\fR 4 |
| 1307 |
.IX Item "--save-headers" |
| 1308 |
Save the headers sent by the HTTP server to the file, preceding the |
| 1309 |
actual contents, with an empty line as the separator. |
| 1310 |
.IP "\fB\-U\fR \fIagent-string\fR" 4 |
| 1311 |
.IX Item "-U agent-string" |
| 1312 |
.PD 0 |
| 1313 |
.IP \fB\-\-user\-agent=\fR\fIagent-string\fR 4 |
| 1314 |
.IX Item "--user-agent=agent-string" |
| 1315 |
.PD |
| 1316 |
Identify as \fIagent-string\fR to the HTTP server. |
| 1317 |
.Sp |
| 1318 |
The HTTP protocol allows the clients to identify themselves using a |
| 1319 |
\&\f(CW\*(C`User\-Agent\*(C'\fR header field. This enables distinguishing the |
| 1320 |
WWW software, usually for statistical purposes or for tracing of |
| 1321 |
protocol violations. Wget normally identifies as |
| 1322 |
\&\fBWget/\fR\fIversion\fR, \fIversion\fR being the current version |
| 1323 |
number of Wget. |
| 1324 |
.Sp |
| 1325 |
However, some sites have been known to impose the policy of tailoring |
| 1326 |
the output according to the \f(CW\*(C`User\-Agent\*(C'\fR\-supplied information. |
| 1327 |
While this is not such a bad idea in theory, it has been abused by |
| 1328 |
servers denying information to clients other than (historically) |
| 1329 |
Netscape or, more frequently, Microsoft Internet Explorer. This |
| 1330 |
option allows you to change the \f(CW\*(C`User\-Agent\*(C'\fR line issued by Wget. |
| 1331 |
Use of this option is discouraged, unless you really know what you are |
| 1332 |
doing. |
| 1333 |
.Sp |
| 1334 |
Specifying empty user agent with \fB\-\-user\-agent=""\fR instructs Wget |
| 1335 |
not to send the \f(CW\*(C`User\-Agent\*(C'\fR header in HTTP requests. |
| 1336 |
.IP \fB\-\-post\-data=\fR\fIstring\fR 4 |
| 1337 |
.IX Item "--post-data=string" |
| 1338 |
.PD 0 |
| 1339 |
.IP \fB\-\-post\-file=\fR\fIfile\fR 4 |
| 1340 |
.IX Item "--post-file=file" |
| 1341 |
.PD |
| 1342 |
Use POST as the method for all HTTP requests and send the specified |
| 1343 |
data in the request body. \fB\-\-post\-data\fR sends \fIstring\fR as |
| 1344 |
data, whereas \fB\-\-post\-file\fR sends the contents of \fIfile\fR. |
| 1345 |
Other than that, they work in exactly the same way. In particular, |
| 1346 |
they \fIboth\fR expect content of the form \f(CW\*(C`key1=value1&key2=value2\*(C'\fR, |
| 1347 |
with percent-encoding for special characters; the only difference is |
| 1348 |
that one expects its content as a command-line parameter and the other |
| 1349 |
accepts its content from a file. In particular, \fB\-\-post\-file\fR is |
| 1350 |
\&\fInot\fR for transmitting files as form attachments: those must |
| 1351 |
appear as \f(CW\*(C`key=value\*(C'\fR data (with appropriate percent-coding) just |
| 1352 |
like everything else. Wget does not currently support |
| 1353 |
\&\f(CW\*(C`multipart/form\-data\*(C'\fR for transmitting POST data; only |
| 1354 |
\&\f(CW\*(C`application/x\-www\-form\-urlencoded\*(C'\fR. Only one of |
| 1355 |
\&\fB\-\-post\-data\fR and \fB\-\-post\-file\fR should be specified. |
| 1356 |
.Sp |
| 1357 |
Please note that wget does not require the content to be of the form |
| 1358 |
\&\f(CW\*(C`key1=value1&key2=value2\*(C'\fR, and neither does it test for it. Wget will |
| 1359 |
simply transmit whatever data is provided to it. Most servers however expect |
| 1360 |
the POST data to be in the above format when processing HTML Forms. |
| 1361 |
.Sp |
| 1362 |
When sending a POST request using the \fB\-\-post\-file\fR option, Wget treats |
| 1363 |
the file as a binary file and will send every character in the POST request |
| 1364 |
without stripping trailing newline or formfeed characters. Any other control |
| 1365 |
characters in the text will also be sent as-is in the POST request. |
| 1366 |
.Sp |
| 1367 |
Please be aware that Wget needs to know the size of the POST data in |
| 1368 |
advance. Therefore the argument to \f(CW\*(C`\-\-post\-file\*(C'\fR must be a regular |
| 1369 |
file; specifying a FIFO or something like \fI/dev/stdin\fR won't work. |
| 1370 |
It's not quite clear how to work around this limitation inherent in |
| 1371 |
HTTP/1.0. Although HTTP/1.1 introduces \fIchunked\fR transfer that |
| 1372 |
doesn't require knowing the request length in advance, a client can't |
| 1373 |
use chunked unless it knows it's talking to an HTTP/1.1 server. And it |
| 1374 |
can't know that until it receives a response, which in turn requires the |
| 1375 |
request to have been completed \-\- a chicken-and-egg problem. |
| 1376 |
.Sp |
| 1377 |
Note: As of version 1.15 if Wget is redirected after the POST request is |
| 1378 |
completed, its behaviour will depend on the response code returned by the |
| 1379 |
server. In case of a 301 Moved Permanently, 302 Moved Temporarily or |
| 1380 |
307 Temporary Redirect, Wget will, in accordance with RFC2616, continue |
| 1381 |
to send a POST request. |
| 1382 |
In case a server wants the client to change the Request method upon |
| 1383 |
redirection, it should send a 303 See Other response code. |
| 1384 |
.Sp |
| 1385 |
This example shows how to log in to a server using POST and then proceed to |
| 1386 |
download the desired pages, presumably only accessible to authorized |
| 1387 |
users: |
| 1388 |
.Sp |
| 1389 |
.Vb 4 |
| 1390 |
\& # Log in to the server. This can be done only once. |
| 1391 |
\& wget \-\-save\-cookies cookies.txt \e |
| 1392 |
\& \-\-post\-data \*(Aquser=foo&password=bar\*(Aq \e |
| 1393 |
\& http://example.com/auth.php |
| 1394 |
\& |
| 1395 |
\& # Now grab the page or pages we care about. |
| 1396 |
\& wget \-\-load\-cookies cookies.txt \e |
| 1397 |
\& \-p http://example.com/interesting/article.php |
| 1398 |
.Ve |
| 1399 |
.Sp |
| 1400 |
If the server is using session cookies to track user authentication, |
| 1401 |
the above will not work because \fB\-\-save\-cookies\fR will not save |
| 1402 |
them (and neither will browsers) and the \fIcookies.txt\fR file will |
| 1403 |
be empty. In that case use \fB\-\-keep\-session\-cookies\fR along with |
| 1404 |
\&\fB\-\-save\-cookies\fR to force saving of session cookies. |
| 1405 |
.IP \fB\-\-method=\fR\fIHTTP-Method\fR 4 |
| 1406 |
.IX Item "--method=HTTP-Method" |
| 1407 |
For the purpose of RESTful scripting, Wget allows sending of other HTTP Methods |
| 1408 |
without the need to explicitly set them using \fB\-\-header=Header\-Line\fR. |
| 1409 |
Wget will use whatever string is passed to it after \fB\-\-method\fR as the HTTP |
| 1410 |
Method to the server. |
| 1411 |
.IP \fB\-\-body\-data=\fR\fIData-String\fR 4 |
| 1412 |
.IX Item "--body-data=Data-String" |
| 1413 |
.PD 0 |
| 1414 |
.IP \fB\-\-body\-file=\fR\fIData-File\fR 4 |
| 1415 |
.IX Item "--body-file=Data-File" |
| 1416 |
.PD |
| 1417 |
Must be set when additional data needs to be sent to the server along with the |
| 1418 |
Method specified using \fB\-\-method\fR. \fB\-\-body\-data\fR sends \fIstring\fR as |
| 1419 |
data, whereas \fB\-\-body\-file\fR sends the contents of \fIfile\fR. Other than that, |
| 1420 |
they work in exactly the same way. |
| 1421 |
.Sp |
| 1422 |
Currently, \fB\-\-body\-file\fR is \fInot\fR for transmitting files as a whole. |
| 1423 |
Wget does not currently support \f(CW\*(C`multipart/form\-data\*(C'\fR for transmitting data; |
| 1424 |
only \f(CW\*(C`application/x\-www\-form\-urlencoded\*(C'\fR. In the future, this may be changed |
| 1425 |
so that wget sends the \fB\-\-body\-file\fR as a complete file instead of sending its |
| 1426 |
contents to the server. Please be aware that Wget needs to know the contents of |
| 1427 |
BODY Data in advance, and hence the argument to \fB\-\-body\-file\fR should be a |
| 1428 |
regular file. See \fB\-\-post\-file\fR for a more detailed explanation. |
| 1429 |
Only one of \fB\-\-body\-data\fR and \fB\-\-body\-file\fR should be specified. |
| 1430 |
.Sp |
| 1431 |
If Wget is redirected after the request is completed, Wget will |
| 1432 |
suspend the current method and send a GET request till the redirection |
| 1433 |
is completed. This is true for all redirection response codes except |
| 1434 |
307 Temporary Redirect which is used to explicitly specify that the |
| 1435 |
request method should \fInot\fR change. Another exception is when |
| 1436 |
the method is set to \f(CW\*(C`POST\*(C'\fR, in which case the redirection rules |
| 1437 |
specified under \fB\-\-post\-data\fR are followed. |
| 1438 |
.IP \fB\-\-content\-disposition\fR 4 |
| 1439 |
.IX Item "--content-disposition" |
| 1440 |
If this is set to on, experimental (not fully-functional) support for |
| 1441 |
\&\f(CW\*(C`Content\-Disposition\*(C'\fR headers is enabled. This can currently result in |
| 1442 |
extra round-trips to the server for a \f(CW\*(C`HEAD\*(C'\fR request, and is known |
| 1443 |
to suffer from a few bugs, which is why it is not currently enabled by default. |
| 1444 |
.Sp |
| 1445 |
This option is useful for some file-downloading CGI programs that use |
| 1446 |
\&\f(CW\*(C`Content\-Disposition\*(C'\fR headers to describe what the name of a |
| 1447 |
downloaded file should be. |
| 1448 |
.Sp |
| 1449 |
When combined with \fB\-\-metalink\-over\-http\fR and \fB\-\-trust\-server\-names\fR, |
| 1450 |
a \fBContent-Type: application/metalink4+xml\fR file is named using the |
| 1451 |
\&\f(CW\*(C`Content\-Disposition\*(C'\fR filename field, if available. |
| 1452 |
.IP \fB\-\-content\-on\-error\fR 4 |
| 1453 |
.IX Item "--content-on-error" |
| 1454 |
If this is set to on, wget will not skip the content when the server responds |
| 1455 |
with a http status code that indicates error. |
| 1456 |
.IP \fB\-\-trust\-server\-names\fR 4 |
| 1457 |
.IX Item "--trust-server-names" |
| 1458 |
If this is set, on a redirect, the local file name will be based |
| 1459 |
on the redirection URL. By default the local file name is based on |
| 1460 |
the original URL. When doing recursive retrieving this can be helpful |
| 1461 |
because in many web sites redirected URLs correspond to an underlying |
| 1462 |
file structure, while link URLs do not. |
| 1463 |
.IP \fB\-\-auth\-no\-challenge\fR 4 |
| 1464 |
.IX Item "--auth-no-challenge" |
| 1465 |
If this option is given, Wget will send Basic HTTP authentication |
| 1466 |
information (plaintext username and password) for all requests, just |
| 1467 |
like Wget 1.10.2 and prior did by default. |
| 1468 |
.Sp |
| 1469 |
Use of this option is not recommended, and is intended only to support |
| 1470 |
some few obscure servers, which never send HTTP authentication |
| 1471 |
challenges, but accept unsolicited auth info, say, in addition to |
| 1472 |
form-based authentication. |
| 1473 |
.IP \fB\-\-retry\-on\-host\-error\fR 4 |
| 1474 |
.IX Item "--retry-on-host-error" |
| 1475 |
Consider host errors, such as "Temporary failure in name resolution", |
| 1476 |
as non-fatal, transient errors. |
| 1477 |
.IP \fB\-\-retry\-on\-http\-error=\fR\fIcode[,code,...]\fR 4 |
| 1478 |
.IX Item "--retry-on-http-error=code[,code,...]" |
| 1479 |
Consider given HTTP response codes as non-fatal, transient errors. |
| 1480 |
Supply a comma-separated list of 3\-digit HTTP response codes as |
| 1481 |
argument. Useful to work around special circumstances where retries |
| 1482 |
are required, but the server responds with an error code normally not |
| 1483 |
retried by Wget. Such errors might be 503 (Service Unavailable) and |
| 1484 |
429 (Too Many Requests). Retries enabled by this option are performed |
| 1485 |
subject to the normal retry timing and retry count limitations of |
| 1486 |
Wget. |
| 1487 |
.Sp |
| 1488 |
Using this option is intended to support special use cases only and is |
| 1489 |
generally not recommended, as it can force retries even in cases where |
| 1490 |
the server is actually trying to decrease its load. Please use wisely |
| 1491 |
and only if you know what you are doing. |
| 1492 |
.SS "HTTPS (SSL/TLS) Options" |
| 1493 |
.IX Subsection "HTTPS (SSL/TLS) Options" |
| 1494 |
To support encrypted HTTP (HTTPS) downloads, Wget must be compiled |
| 1495 |
with an external SSL library. The current default is GnuTLS. |
| 1496 |
In addition, Wget also supports HSTS (HTTP Strict Transport Security). |
| 1497 |
If Wget is compiled without SSL support, none of these options are available. |
| 1498 |
.IP \fB\-\-secure\-protocol=\fR\fIprotocol\fR 4 |
| 1499 |
.IX Item "--secure-protocol=protocol" |
| 1500 |
Choose the secure protocol to be used. Legal values are \fBauto\fR, |
| 1501 |
\&\fBSSLv2\fR, \fBSSLv3\fR, \fBTLSv1\fR, \fBTLSv1_1\fR, \fBTLSv1_2\fR, |
| 1502 |
\&\fBTLSv1_3\fR and \fBPFS\fR. If \fBauto\fR is used, the SSL library is |
| 1503 |
given the liberty of choosing the appropriate protocol automatically, which is |
| 1504 |
achieved by sending a TLSv1 greeting. This is the default. |
| 1505 |
.Sp |
| 1506 |
Specifying \fBSSLv2\fR, \fBSSLv3\fR, \fBTLSv1\fR, \fBTLSv1_1\fR, |
| 1507 |
\&\fBTLSv1_2\fR or \fBTLSv1_3\fR forces the use of the corresponding |
| 1508 |
protocol. This is useful when talking to old and buggy SSL server |
| 1509 |
implementations that make it hard for the underlying SSL library to choose |
| 1510 |
the correct protocol version. Fortunately, such servers are quite rare. |
| 1511 |
.Sp |
| 1512 |
Specifying \fBPFS\fR enforces the use of the so-called Perfect Forward |
| 1513 |
Security cipher suites. In short, PFS adds security by creating a one-time |
| 1514 |
key for each SSL connection. It has a bit more CPU impact on client and server. |
| 1515 |
We use known to be secure ciphers (e.g. no MD4) and the TLS protocol. This mode |
| 1516 |
also explicitly excludes non-PFS key exchange methods, such as RSA. |
| 1517 |
.IP \fB\-\-https\-only\fR 4 |
| 1518 |
.IX Item "--https-only" |
| 1519 |
When in recursive mode, only HTTPS links are followed. |
| 1520 |
.IP \fB\-\-ciphers\fR 4 |
| 1521 |
.IX Item "--ciphers" |
| 1522 |
Set the cipher list string. Typically this string sets the |
| 1523 |
cipher suites and other SSL/TLS options that the user wish should be used, in a |
| 1524 |
set order of preference (GnuTLS calls it 'priority string'). This string |
| 1525 |
will be fed verbatim to the SSL/TLS engine (OpenSSL or GnuTLS) and hence |
| 1526 |
its format and syntax is dependent on that. Wget will not process or manipulate it |
| 1527 |
in any way. Refer to the OpenSSL or GnuTLS documentation for more information. |
| 1528 |
.IP \fB\-\-no\-check\-certificate\fR 4 |
| 1529 |
.IX Item "--no-check-certificate" |
| 1530 |
Don't check the server certificate against the available certificate |
| 1531 |
authorities. Also don't require the URL host name to match the common |
| 1532 |
name presented by the certificate. |
| 1533 |
.Sp |
| 1534 |
As of Wget 1.10, the default is to verify the server's certificate |
| 1535 |
against the recognized certificate authorities, breaking the SSL |
| 1536 |
handshake and aborting the download if the verification fails. |
| 1537 |
Although this provides more secure downloads, it does break |
| 1538 |
interoperability with some sites that worked with previous Wget |
| 1539 |
versions, particularly those using self-signed, expired, or otherwise |
| 1540 |
invalid certificates. This option forces an "insecure" mode of |
| 1541 |
operation that turns the certificate verification errors into warnings |
| 1542 |
and allows you to proceed. |
| 1543 |
.Sp |
| 1544 |
If you encounter "certificate verification" errors or ones saying |
| 1545 |
that "common name doesn't match requested host name", you can use |
| 1546 |
this option to bypass the verification and proceed with the download. |
| 1547 |
\&\fIOnly use this option if you are otherwise convinced of the |
| 1548 |
site's authenticity, or if you really don't care about the validity of |
| 1549 |
its certificate.\fR It is almost always a bad idea not to check the |
| 1550 |
certificates when transmitting confidential or important data. |
| 1551 |
For self\-signed/internal certificates, you should download the certificate |
| 1552 |
and verify against that instead of forcing this insecure mode. |
| 1553 |
If you are really sure of not desiring any certificate verification, you |
| 1554 |
can specify \-\-check\-certificate=quiet to tell wget to not print any |
| 1555 |
warning about invalid certificates, albeit in most cases this is the |
| 1556 |
wrong thing to do. |
| 1557 |
.IP \fB\-\-certificate=\fR\fIfile\fR 4 |
| 1558 |
.IX Item "--certificate=file" |
| 1559 |
Use the client certificate stored in \fIfile\fR. This is needed for |
| 1560 |
servers that are configured to require certificates from the clients |
| 1561 |
that connect to them. Normally a certificate is not required and this |
| 1562 |
switch is optional. |
| 1563 |
.IP \fB\-\-certificate\-type=\fR\fItype\fR 4 |
| 1564 |
.IX Item "--certificate-type=type" |
| 1565 |
Specify the type of the client certificate. Legal values are |
| 1566 |
\&\fBPEM\fR (assumed by default) and \fBDER\fR, also known as |
| 1567 |
\&\fBASN1\fR. |
| 1568 |
.IP \fB\-\-private\-key=\fR\fIfile\fR 4 |
| 1569 |
.IX Item "--private-key=file" |
| 1570 |
Read the private key from \fIfile\fR. This allows you to provide the |
| 1571 |
private key in a file separate from the certificate. |
| 1572 |
.IP \fB\-\-private\-key\-type=\fR\fItype\fR 4 |
| 1573 |
.IX Item "--private-key-type=type" |
| 1574 |
Specify the type of the private key. Accepted values are \fBPEM\fR |
| 1575 |
(the default) and \fBDER\fR. |
| 1576 |
.IP \fB\-\-ca\-certificate=\fR\fIfile\fR 4 |
| 1577 |
.IX Item "--ca-certificate=file" |
| 1578 |
Use \fIfile\fR as the file with the bundle of certificate authorities |
| 1579 |
("CA") to verify the peers. The certificates must be in PEM format. |
| 1580 |
.Sp |
| 1581 |
Without this option Wget looks for CA certificates at the |
| 1582 |
system-specified locations, chosen at OpenSSL installation time. |
| 1583 |
.IP \fB\-\-ca\-directory=\fR\fIdirectory\fR 4 |
| 1584 |
.IX Item "--ca-directory=directory" |
| 1585 |
Specifies directory containing CA certificates in PEM format. Each |
| 1586 |
file contains one CA certificate, and the file name is based on a hash |
| 1587 |
value derived from the certificate. This is achieved by processing a |
| 1588 |
certificate directory with the \f(CW\*(C`c_rehash\*(C'\fR utility supplied with |
| 1589 |
OpenSSL. Using \fB\-\-ca\-directory\fR is more efficient than |
| 1590 |
\&\fB\-\-ca\-certificate\fR when many certificates are installed because |
| 1591 |
it allows Wget to fetch certificates on demand. |
| 1592 |
.Sp |
| 1593 |
Without this option Wget looks for CA certificates at the |
| 1594 |
system-specified locations, chosen at OpenSSL installation time. |
| 1595 |
.IP \fB\-\-crl\-file=\fR\fIfile\fR 4 |
| 1596 |
.IX Item "--crl-file=file" |
| 1597 |
Specifies a CRL file in \fIfile\fR. This is needed for certificates |
| 1598 |
that have been revocated by the CAs. |
| 1599 |
.IP \fB\-\-pinnedpubkey=file/hashes\fR 4 |
| 1600 |
.IX Item "--pinnedpubkey=file/hashes" |
| 1601 |
Tells wget to use the specified public key file (or hashes) to verify the peer. |
| 1602 |
This can be a path to a file which contains a single public key in PEM or DER |
| 1603 |
format, or any number of base64 encoded sha256 hashes preceded by "sha256//" |
| 1604 |
and separated by ";" |
| 1605 |
.Sp |
| 1606 |
When negotiating a TLS or SSL connection, the server sends a certificate |
| 1607 |
indicating its identity. A public key is extracted from this certificate and if |
| 1608 |
it does not exactly match the public key(s) provided to this option, wget will |
| 1609 |
abort the connection before sending or receiving any data. |
| 1610 |
.IP \fB\-\-random\-file=\fR\fIfile\fR 4 |
| 1611 |
.IX Item "--random-file=file" |
| 1612 |
[OpenSSL and LibreSSL only] |
| 1613 |
Use \fIfile\fR as the source of random data for seeding the |
| 1614 |
pseudo-random number generator on systems without \fI/dev/urandom\fR. |
| 1615 |
.Sp |
| 1616 |
On such systems the SSL library needs an external source of randomness |
| 1617 |
to initialize. Randomness may be provided by EGD (see |
| 1618 |
\&\fB\-\-egd\-file\fR below) or read from an external source specified by |
| 1619 |
the user. If this option is not specified, Wget looks for random data |
| 1620 |
in \f(CW$RANDFILE\fR or, if that is unset, in \fR\f(CI$HOME\fR\fI/.rnd\fR. |
| 1621 |
.Sp |
| 1622 |
If you're getting the "Could not seed OpenSSL PRNG; disabling SSL." |
| 1623 |
error, you should provide random data using some of the methods |
| 1624 |
described above. |
| 1625 |
.IP \fB\-\-egd\-file=\fR\fIfile\fR 4 |
| 1626 |
.IX Item "--egd-file=file" |
| 1627 |
[OpenSSL only] |
| 1628 |
Use \fIfile\fR as the EGD socket. EGD stands for \fIEntropy |
| 1629 |
Gathering Daemon\fR, a user-space program that collects data from |
| 1630 |
various unpredictable system sources and makes it available to other |
| 1631 |
programs that might need it. Encryption software, such as the SSL |
| 1632 |
library, needs sources of non-repeating randomness to seed the random |
| 1633 |
number generator used to produce cryptographically strong keys. |
| 1634 |
.Sp |
| 1635 |
OpenSSL allows the user to specify his own source of entropy using the |
| 1636 |
\&\f(CW\*(C`RAND_FILE\*(C'\fR environment variable. If this variable is unset, or |
| 1637 |
if the specified file does not produce enough randomness, OpenSSL will |
| 1638 |
read random data from EGD socket specified using this option. |
| 1639 |
.Sp |
| 1640 |
If this option is not specified (and the equivalent startup command is |
| 1641 |
not used), EGD is never contacted. EGD is not needed on modern Unix |
| 1642 |
systems that support \fI/dev/urandom\fR. |
| 1643 |
.IP \fB\-\-no\-hsts\fR 4 |
| 1644 |
.IX Item "--no-hsts" |
| 1645 |
Wget supports HSTS (HTTP Strict Transport Security, RFC 6797) by default. |
| 1646 |
Use \fB\-\-no\-hsts\fR to make Wget act as a non-HSTS-compliant UA. As a |
| 1647 |
consequence, Wget would ignore all the \f(CW\*(C`Strict\-Transport\-Security\*(C'\fR |
| 1648 |
headers, and would not enforce any existing HSTS policy. |
| 1649 |
.IP \fB\-\-hsts\-file=\fR\fIfile\fR 4 |
| 1650 |
.IX Item "--hsts-file=file" |
| 1651 |
By default, Wget stores its HSTS database in \fI~/.wget\-hsts\fR. |
| 1652 |
You can use \fB\-\-hsts\-file\fR to override this. Wget will use |
| 1653 |
the supplied file as the HSTS database. Such file must conform to the |
| 1654 |
correct HSTS database format used by Wget. If Wget cannot parse the provided |
| 1655 |
file, the behaviour is unspecified. |
| 1656 |
.Sp |
| 1657 |
The Wget's HSTS database is a plain text file. Each line contains an HSTS entry |
| 1658 |
(ie. a site that has issued a \f(CW\*(C`Strict\-Transport\-Security\*(C'\fR header and that |
| 1659 |
therefore has specified a concrete HSTS policy to be applied). Lines starting with |
| 1660 |
a dash (\f(CW\*(C`#\*(C'\fR) are ignored by Wget. Please note that in spite of this convenient |
| 1661 |
human-readability hand-hacking the HSTS database is generally not a good idea. |
| 1662 |
.Sp |
| 1663 |
An HSTS entry line consists of several fields separated by one or more whitespace: |
| 1664 |
.Sp |
| 1665 |
\&\f(CW\*(C`<hostname> SP [<port>] SP <include subdomains> SP <created> SP <max\-age>\*(C'\fR |
| 1666 |
.Sp |
| 1667 |
The \fIhostname\fR and \fIport\fR fields indicate the hostname and port to which |
| 1668 |
the given HSTS policy applies. The \fIport\fR field may be zero, and it will, in |
| 1669 |
most of the cases. That means that the port number will not be taken into account |
| 1670 |
when deciding whether such HSTS policy should be applied on a given request (only |
| 1671 |
the hostname will be evaluated). When \fIport\fR is different to zero, both the |
| 1672 |
target hostname and the port will be evaluated and the HSTS policy will only be applied |
| 1673 |
if both of them match. This feature has been included for testing/development purposes only. |
| 1674 |
The Wget testsuite (in \fItestenv/\fR) creates HSTS databases with explicit ports |
| 1675 |
with the purpose of ensuring Wget's correct behaviour. Applying HSTS policies to ports |
| 1676 |
other than the default ones is discouraged by RFC 6797 (see Appendix B "Differences |
| 1677 |
between HSTS Policy and Same-Origin Policy"). Thus, this functionality should not be used |
| 1678 |
in production environments and \fIport\fR will typically be zero. The last three fields |
| 1679 |
do what they are expected to. The field \fIinclude_subdomains\fR can either be \f(CW1\fR |
| 1680 |
or \f(CW0\fR and it signals whether the subdomains of the target domain should be |
| 1681 |
part of the given HSTS policy as well. The \fIcreated\fR and \fImax-age\fR fields |
| 1682 |
hold the timestamp values of when such entry was created (first seen by Wget) and the |
| 1683 |
HSTS-defined value 'max\-age', which states how long should that HSTS policy remain active, |
| 1684 |
measured in seconds elapsed since the timestamp stored in \fIcreated\fR. Once that time |
| 1685 |
has passed, that HSTS policy will no longer be valid and will eventually be removed |
| 1686 |
from the database. |
| 1687 |
.Sp |
| 1688 |
If you supply your own HSTS database via \fB\-\-hsts\-file\fR, be aware that Wget |
| 1689 |
may modify the provided file if any change occurs between the HSTS policies |
| 1690 |
requested by the remote servers and those in the file. When Wget exits, |
| 1691 |
it effectively updates the HSTS database by rewriting the database file with the new entries. |
| 1692 |
.Sp |
| 1693 |
If the supplied file does not exist, Wget will create one. This file will contain the new HSTS |
| 1694 |
entries. If no HSTS entries were generated (no \f(CW\*(C`Strict\-Transport\-Security\*(C'\fR headers |
| 1695 |
were sent by any of the servers) then no file will be created, not even an empty one. This |
| 1696 |
behaviour applies to the default database file (\fI~/.wget\-hsts\fR) as well: it will not be |
| 1697 |
created until some server enforces an HSTS policy. |
| 1698 |
.Sp |
| 1699 |
Care is taken not to override possible changes made by other Wget processes at |
| 1700 |
the same time over the HSTS database. Before dumping the updated HSTS entries |
| 1701 |
on the file, Wget will re-read it and merge the changes. |
| 1702 |
.Sp |
| 1703 |
Using a custom HSTS database and/or modifying an existing one is discouraged. |
| 1704 |
For more information about the potential security threats arose from such practice, |
| 1705 |
see section 14 "Security Considerations" of RFC 6797, specially section 14.9 |
| 1706 |
"Creative Manipulation of HSTS Policy Store". |
| 1707 |
.IP \fB\-\-warc\-file=\fR\fIfile\fR 4 |
| 1708 |
.IX Item "--warc-file=file" |
| 1709 |
Use \fIfile\fR as the destination WARC file. |
| 1710 |
.IP \fB\-\-warc\-header=\fR\fIstring\fR 4 |
| 1711 |
.IX Item "--warc-header=string" |
| 1712 |
Use \fIstring\fR into as the warcinfo record. |
| 1713 |
.IP \fB\-\-warc\-max\-size=\fR\fIsize\fR 4 |
| 1714 |
.IX Item "--warc-max-size=size" |
| 1715 |
Set the maximum size of the WARC files to \fIsize\fR. |
| 1716 |
.IP \fB\-\-warc\-cdx\fR 4 |
| 1717 |
.IX Item "--warc-cdx" |
| 1718 |
Write CDX index files. |
| 1719 |
.IP \fB\-\-warc\-dedup=\fR\fIfile\fR 4 |
| 1720 |
.IX Item "--warc-dedup=file" |
| 1721 |
Do not store records listed in this CDX file. |
| 1722 |
.IP \fB\-\-no\-warc\-compression\fR 4 |
| 1723 |
.IX Item "--no-warc-compression" |
| 1724 |
Do not compress WARC files with GZIP. |
| 1725 |
.IP \fB\-\-no\-warc\-digests\fR 4 |
| 1726 |
.IX Item "--no-warc-digests" |
| 1727 |
Do not calculate SHA1 digests. |
| 1728 |
.IP \fB\-\-no\-warc\-keep\-log\fR 4 |
| 1729 |
.IX Item "--no-warc-keep-log" |
| 1730 |
Do not store the log file in a WARC record. |
| 1731 |
.IP \fB\-\-warc\-tempdir=\fR\fIdir\fR 4 |
| 1732 |
.IX Item "--warc-tempdir=dir" |
| 1733 |
Specify the location for temporary files created by the WARC writer. |
| 1734 |
.SS "FTP Options" |
| 1735 |
.IX Subsection "FTP Options" |
| 1736 |
.IP \fB\-\-ftp\-user=\fR\fIuser\fR 4 |
| 1737 |
.IX Item "--ftp-user=user" |
| 1738 |
.PD 0 |
| 1739 |
.IP \fB\-\-ftp\-password=\fR\fIpassword\fR 4 |
| 1740 |
.IX Item "--ftp-password=password" |
| 1741 |
.PD |
| 1742 |
Specify the username \fIuser\fR and password \fIpassword\fR on an |
| 1743 |
FTP server. Without this, or the corresponding startup option, |
| 1744 |
the password defaults to \fB\-wget@\fR, normally used for anonymous |
| 1745 |
FTP. |
| 1746 |
.Sp |
| 1747 |
Another way to specify username and password is in the URL itself. Either method reveals your password to anyone who |
| 1748 |
bothers to run \f(CW\*(C`ps\*(C'\fR. To prevent the passwords from being seen, |
| 1749 |
store them in \fI.wgetrc\fR or \fI.netrc\fR, and make sure to protect |
| 1750 |
those files from other users with \f(CW\*(C`chmod\*(C'\fR. If the passwords are |
| 1751 |
really important, do not leave them lying in those files either\-\-\-edit |
| 1752 |
the files and delete them after Wget has started the download. |
| 1753 |
.IP \fB\-\-no\-remove\-listing\fR 4 |
| 1754 |
.IX Item "--no-remove-listing" |
| 1755 |
Don't remove the temporary \fI.listing\fR files generated by FTP |
| 1756 |
retrievals. Normally, these files contain the raw directory listings |
| 1757 |
received from FTP servers. Not removing them can be useful for |
| 1758 |
debugging purposes, or when you want to be able to easily check on the |
| 1759 |
contents of remote server directories (e.g. to verify that a mirror |
| 1760 |
you're running is complete). |
| 1761 |
.Sp |
| 1762 |
Note that even though Wget writes to a known filename for this file, |
| 1763 |
this is not a security hole in the scenario of a user making |
| 1764 |
\&\fI.listing\fR a symbolic link to \fI/etc/passwd\fR or something and |
| 1765 |
asking \f(CW\*(C`root\*(C'\fR to run Wget in his or her directory. Depending on |
| 1766 |
the options used, either Wget will refuse to write to \fI.listing\fR, |
| 1767 |
making the globbing/recursion/time\-stamping operation fail, or the |
| 1768 |
symbolic link will be deleted and replaced with the actual |
| 1769 |
\&\fI.listing\fR file, or the listing will be written to a |
| 1770 |
\&\fI.listing.number\fR file. |
| 1771 |
.Sp |
| 1772 |
Even though this situation isn't a problem, though, \f(CW\*(C`root\*(C'\fR should |
| 1773 |
never run Wget in a non-trusted user's directory. A user could do |
| 1774 |
something as simple as linking \fIindex.html\fR to \fI/etc/passwd\fR |
| 1775 |
and asking \f(CW\*(C`root\*(C'\fR to run Wget with \fB\-N\fR or \fB\-r\fR so the file |
| 1776 |
will be overwritten. |
| 1777 |
.IP \fB\-\-no\-glob\fR 4 |
| 1778 |
.IX Item "--no-glob" |
| 1779 |
Turn off FTP globbing. Globbing refers to the use of shell-like |
| 1780 |
special characters (\fIwildcards\fR), like \fB*\fR, \fB?\fR, \fB[\fR |
| 1781 |
and \fB]\fR to retrieve more than one file from the same directory at |
| 1782 |
once, like: |
| 1783 |
.Sp |
| 1784 |
.Vb 1 |
| 1785 |
\& wget ftp://gnjilux.srk.fer.hr/*.msg |
| 1786 |
.Ve |
| 1787 |
.Sp |
| 1788 |
By default, globbing will be turned on if the URL contains a |
| 1789 |
globbing character. This option may be used to turn globbing on or off |
| 1790 |
permanently. |
| 1791 |
.Sp |
| 1792 |
You may have to quote the URL to protect it from being expanded by |
| 1793 |
your shell. Globbing makes Wget look for a directory listing, which is |
| 1794 |
system-specific. This is why it currently works only with Unix FTP |
| 1795 |
servers (and the ones emulating Unix \f(CW\*(C`ls\*(C'\fR output). |
| 1796 |
.IP \fB\-\-no\-passive\-ftp\fR 4 |
| 1797 |
.IX Item "--no-passive-ftp" |
| 1798 |
Disable the use of the \fIpassive\fR FTP transfer mode. Passive FTP |
| 1799 |
mandates that the client connect to the server to establish the data |
| 1800 |
connection rather than the other way around. |
| 1801 |
.Sp |
| 1802 |
If the machine is connected to the Internet directly, both passive and |
| 1803 |
active FTP should work equally well. Behind most firewall and NAT |
| 1804 |
configurations passive FTP has a better chance of working. However, |
| 1805 |
in some rare firewall configurations, active FTP actually works when |
| 1806 |
passive FTP doesn't. If you suspect this to be the case, use this |
| 1807 |
option, or set \f(CW\*(C`passive_ftp=off\*(C'\fR in your init file. |
| 1808 |
.IP \fB\-\-preserve\-permissions\fR 4 |
| 1809 |
.IX Item "--preserve-permissions" |
| 1810 |
Preserve remote file permissions instead of permissions set by umask. |
| 1811 |
.IP \fB\-\-retr\-symlinks\fR 4 |
| 1812 |
.IX Item "--retr-symlinks" |
| 1813 |
By default, when retrieving FTP directories recursively and a symbolic link |
| 1814 |
is encountered, the symbolic link is traversed and the pointed-to files are |
| 1815 |
retrieved. Currently, Wget does not traverse symbolic links to directories to |
| 1816 |
download them recursively, though this feature may be added in the future. |
| 1817 |
.Sp |
| 1818 |
When \fB\-\-retr\-symlinks=no\fR is specified, the linked-to file is not |
| 1819 |
downloaded. Instead, a matching symbolic link is created on the local |
| 1820 |
file system. The pointed-to file will not be retrieved unless this recursive |
| 1821 |
retrieval would have encountered it separately and downloaded it anyway. This |
| 1822 |
option poses a security risk where a malicious FTP Server may cause Wget to |
| 1823 |
write to files outside of the intended directories through a specially crafted |
| 1824 |
\&.LISTING file. |
| 1825 |
.Sp |
| 1826 |
Note that when retrieving a file (not a directory) because it was |
| 1827 |
specified on the command-line, rather than because it was recursed to, |
| 1828 |
this option has no effect. Symbolic links are always traversed in this |
| 1829 |
case. |
| 1830 |
.SS "FTPS Options" |
| 1831 |
.IX Subsection "FTPS Options" |
| 1832 |
.IP \fB\-\-ftps\-implicit\fR 4 |
| 1833 |
.IX Item "--ftps-implicit" |
| 1834 |
This option tells Wget to use FTPS implicitly. Implicit FTPS consists of initializing |
| 1835 |
SSL/TLS from the very beginning of the control connection. This option does not send |
| 1836 |
an \f(CW\*(C`AUTH TLS\*(C'\fR command: it assumes the server speaks FTPS and directly starts an |
| 1837 |
SSL/TLS connection. If the attempt is successful, the session continues just like |
| 1838 |
regular FTPS (\f(CW\*(C`PBSZ\*(C'\fR and \f(CW\*(C`PROT\*(C'\fR are sent, etc.). |
| 1839 |
Implicit FTPS is no longer a requirement for FTPS implementations, and thus |
| 1840 |
many servers may not support it. If \fB\-\-ftps\-implicit\fR is passed and no explicit |
| 1841 |
port number specified, the default port for implicit FTPS, 990, will be used, instead |
| 1842 |
of the default port for the "normal" (explicit) FTPS which is the same as that of FTP, |
| 1843 |
21. |
| 1844 |
.IP \fB\-\-no\-ftps\-resume\-ssl\fR 4 |
| 1845 |
.IX Item "--no-ftps-resume-ssl" |
| 1846 |
Do not resume the SSL/TLS session in the data channel. When starting a data connection, |
| 1847 |
Wget tries to resume the SSL/TLS session previously started in the control connection. |
| 1848 |
SSL/TLS session resumption avoids performing an entirely new handshake by reusing |
| 1849 |
the SSL/TLS parameters of a previous session. Typically, the FTPS servers want it that way, |
| 1850 |
so Wget does this by default. Under rare circumstances however, one might want to |
| 1851 |
start an entirely new SSL/TLS session in every data connection. |
| 1852 |
This is what \fB\-\-no\-ftps\-resume\-ssl\fR is for. |
| 1853 |
.IP \fB\-\-ftps\-clear\-data\-connection\fR 4 |
| 1854 |
.IX Item "--ftps-clear-data-connection" |
| 1855 |
All the data connections will be in plain text. Only the control connection will be |
| 1856 |
under SSL/TLS. Wget will send a \f(CW\*(C`PROT C\*(C'\fR command to achieve this, which must be |
| 1857 |
approved by the server. |
| 1858 |
.IP \fB\-\-ftps\-fallback\-to\-ftp\fR 4 |
| 1859 |
.IX Item "--ftps-fallback-to-ftp" |
| 1860 |
Fall back to FTP if FTPS is not supported by the target server. For security reasons, |
| 1861 |
this option is not asserted by default. The default behaviour is to exit with an error. |
| 1862 |
If a server does not successfully reply to the initial \f(CW\*(C`AUTH TLS\*(C'\fR command, or in the |
| 1863 |
case of implicit FTPS, if the initial SSL/TLS connection attempt is rejected, it is |
| 1864 |
considered that such server does not support FTPS. |
| 1865 |
.SS "Recursive Retrieval Options" |
| 1866 |
.IX Subsection "Recursive Retrieval Options" |
| 1867 |
.IP \fB\-r\fR 4 |
| 1868 |
.IX Item "-r" |
| 1869 |
.PD 0 |
| 1870 |
.IP \fB\-\-recursive\fR 4 |
| 1871 |
.IX Item "--recursive" |
| 1872 |
.PD |
| 1873 |
Turn on recursive retrieving. The default maximum depth is 5. |
| 1874 |
.IP "\fB\-l\fR \fIdepth\fR" 4 |
| 1875 |
.IX Item "-l depth" |
| 1876 |
.PD 0 |
| 1877 |
.IP \fB\-\-level=\fR\fIdepth\fR 4 |
| 1878 |
.IX Item "--level=depth" |
| 1879 |
.PD |
| 1880 |
Set the maximum number of subdirectories that Wget will recurse into to \fIdepth\fR. |
| 1881 |
In order to prevent one from accidentally downloading very large websites when using recursion |
| 1882 |
this is limited to a depth of 5 by default, i.e., it will traverse at most 5 directories deep |
| 1883 |
starting from the provided URL. |
| 1884 |
Set \fB\-l 0\fR or \fB\-l inf\fR for infinite recursion depth. |
| 1885 |
.Sp |
| 1886 |
.Vb 1 |
| 1887 |
\& wget \-r \-l 0 http://<site>/1.html |
| 1888 |
.Ve |
| 1889 |
.Sp |
| 1890 |
Ideally, one would expect this to download just \fI1.html\fR. |
| 1891 |
but unfortunately this is not the case, because \fB\-l 0\fR is equivalent to |
| 1892 |
\&\fB\-l inf\fR\-\-\-that is, infinite recursion. To download a single HTML |
| 1893 |
page (or a handful of them), specify them all on the command line and leave away \fB\-r\fR |
| 1894 |
and \fB\-l\fR. To download the essential items to view a single HTML page, see \fBpage requisites\fR. |
| 1895 |
.IP \fB\-\-delete\-after\fR 4 |
| 1896 |
.IX Item "--delete-after" |
| 1897 |
This option tells Wget to delete every single file it downloads, |
| 1898 |
\&\fIafter\fR having done so. It is useful for pre-fetching popular |
| 1899 |
pages through a proxy, e.g.: |
| 1900 |
.Sp |
| 1901 |
.Vb 1 |
| 1902 |
\& wget \-r \-nd \-\-delete\-after http://whatever.com/~popular/page/ |
| 1903 |
.Ve |
| 1904 |
.Sp |
| 1905 |
The \fB\-r\fR option is to retrieve recursively, and \fB\-nd\fR to not |
| 1906 |
create directories. |
| 1907 |
.Sp |
| 1908 |
Note that \fB\-\-delete\-after\fR deletes files on the local machine. It |
| 1909 |
does not issue the \fBDELE\fR command to remote FTP sites, for |
| 1910 |
instance. Also note that when \fB\-\-delete\-after\fR is specified, |
| 1911 |
\&\fB\-\-convert\-links\fR is ignored, so \fB.orig\fR files are simply not |
| 1912 |
created in the first place. |
| 1913 |
.IP \fB\-k\fR 4 |
| 1914 |
.IX Item "-k" |
| 1915 |
.PD 0 |
| 1916 |
.IP \fB\-\-convert\-links\fR 4 |
| 1917 |
.IX Item "--convert-links" |
| 1918 |
.PD |
| 1919 |
After the download is complete, convert the links in the document to |
| 1920 |
make them suitable for local viewing. This affects not only the visible |
| 1921 |
hyperlinks, but any part of the document that links to external content, |
| 1922 |
such as embedded images, links to style sheets, hyperlinks to non-HTML |
| 1923 |
content, etc. |
| 1924 |
.Sp |
| 1925 |
Each link will be changed in one of the two ways: |
| 1926 |
.RS 4 |
| 1927 |
.IP \(bu 4 |
| 1928 |
The links to files that have been downloaded by Wget will be changed to |
| 1929 |
refer to the file they point to as a relative link. |
| 1930 |
.Sp |
| 1931 |
Example: if the downloaded file \fI/foo/doc.html\fR links to |
| 1932 |
\&\fI/bar/img.gif\fR, also downloaded, then the link in \fIdoc.html\fR |
| 1933 |
will be modified to point to \fB../bar/img.gif\fR. This kind of |
| 1934 |
transformation works reliably for arbitrary combinations of directories. |
| 1935 |
.IP \(bu 4 |
| 1936 |
The links to files that have not been downloaded by Wget will be changed |
| 1937 |
to include host name and absolute path of the location they point to. |
| 1938 |
.Sp |
| 1939 |
Example: if the downloaded file \fI/foo/doc.html\fR links to |
| 1940 |
\&\fI/bar/img.gif\fR (or to \fI../bar/img.gif\fR), then the link in |
| 1941 |
\&\fIdoc.html\fR will be modified to point to |
| 1942 |
\&\fIhttp://hostname/bar/img.gif\fR. |
| 1943 |
.RE |
| 1944 |
.RS 4 |
| 1945 |
.Sp |
| 1946 |
Because of this, local browsing works reliably: if a linked file was |
| 1947 |
downloaded, the link will refer to its local name; if it was not |
| 1948 |
downloaded, the link will refer to its full Internet address rather than |
| 1949 |
presenting a broken link. The fact that the former links are converted |
| 1950 |
to relative links ensures that you can move the downloaded hierarchy to |
| 1951 |
another directory. |
| 1952 |
.Sp |
| 1953 |
Note that only at the end of the download can Wget know which links have |
| 1954 |
been downloaded. Because of that, the work done by \fB\-k\fR will be |
| 1955 |
performed at the end of all the downloads. |
| 1956 |
.RE |
| 1957 |
.IP \fB\-\-convert\-file\-only\fR 4 |
| 1958 |
.IX Item "--convert-file-only" |
| 1959 |
This option converts only the filename part of the URLs, leaving the rest |
| 1960 |
of the URLs untouched. This filename part is sometimes referred to as the |
| 1961 |
"basename", although we avoid that term here in order not to cause confusion. |
| 1962 |
.Sp |
| 1963 |
It works particularly well in conjunction with \fB\-\-adjust\-extension\fR, although |
| 1964 |
this coupling is not enforced. It proves useful to populate Internet caches |
| 1965 |
with files downloaded from different hosts. |
| 1966 |
.Sp |
| 1967 |
Example: if some link points to \fI//foo.com/bar.cgi?xyz\fR with |
| 1968 |
\&\fB\-\-adjust\-extension\fR asserted and its local destination is intended to be |
| 1969 |
\&\fI./foo.com/bar.cgi?xyz.css\fR, then the link would be converted to |
| 1970 |
\&\fI//foo.com/bar.cgi?xyz.css\fR. Note that only the filename part has been |
| 1971 |
modified. The rest of the URL has been left untouched, including the net path |
| 1972 |
(\f(CW\*(C`//\*(C'\fR) which would otherwise be processed by Wget and converted to the |
| 1973 |
effective scheme (ie. \f(CW\*(C`http://\*(C'\fR). |
| 1974 |
.IP \fB\-K\fR 4 |
| 1975 |
.IX Item "-K" |
| 1976 |
.PD 0 |
| 1977 |
.IP \fB\-\-backup\-converted\fR 4 |
| 1978 |
.IX Item "--backup-converted" |
| 1979 |
.PD |
| 1980 |
When converting a file, back up the original version with a \fB.orig\fR |
| 1981 |
suffix. Affects the behavior of \fB\-N\fR. |
| 1982 |
.IP \fB\-m\fR 4 |
| 1983 |
.IX Item "-m" |
| 1984 |
.PD 0 |
| 1985 |
.IP \fB\-\-mirror\fR 4 |
| 1986 |
.IX Item "--mirror" |
| 1987 |
.PD |
| 1988 |
Turn on options suitable for mirroring. This option turns on recursion |
| 1989 |
and time-stamping, sets infinite recursion depth and keeps FTP |
| 1990 |
directory listings. It is currently equivalent to |
| 1991 |
\&\fB\-r \-N \-l inf \-\-no\-remove\-listing\fR. |
| 1992 |
.IP \fB\-p\fR 4 |
| 1993 |
.IX Item "-p" |
| 1994 |
.PD 0 |
| 1995 |
.IP \fB\-\-page\-requisites\fR 4 |
| 1996 |
.IX Item "--page-requisites" |
| 1997 |
.PD |
| 1998 |
This option causes Wget to download all the files that are necessary to |
| 1999 |
properly display a given HTML page. This includes such things as |
| 2000 |
inlined images, sounds, and referenced stylesheets. |
| 2001 |
.Sp |
| 2002 |
Ordinarily, when downloading a single HTML page, any requisite documents |
| 2003 |
that may be needed to display it properly are not downloaded. Using |
| 2004 |
\&\fB\-r\fR together with \fB\-l\fR can help, but since Wget does not |
| 2005 |
ordinarily distinguish between external and inlined documents, one is |
| 2006 |
generally left with "leaf documents" that are missing their |
| 2007 |
requisites. |
| 2008 |
.Sp |
| 2009 |
For instance, say document \fI1.html\fR contains an \f(CW\*(C`<IMG>\*(C'\fR tag |
| 2010 |
referencing \fI1.gif\fR and an \f(CW\*(C`<A>\*(C'\fR tag pointing to external |
| 2011 |
document \fI2.html\fR. Say that \fI2.html\fR is similar but that its |
| 2012 |
image is \fI2.gif\fR and it links to \fI3.html\fR. Say this |
| 2013 |
continues up to some arbitrarily high number. |
| 2014 |
.Sp |
| 2015 |
If one executes the command: |
| 2016 |
.Sp |
| 2017 |
.Vb 1 |
| 2018 |
\& wget \-r \-l 2 http://<site>/1.html |
| 2019 |
.Ve |
| 2020 |
.Sp |
| 2021 |
then \fI1.html\fR, \fI1.gif\fR, \fI2.html\fR, \fI2.gif\fR, and |
| 2022 |
\&\fI3.html\fR will be downloaded. As you can see, \fI3.html\fR is |
| 2023 |
without its requisite \fI3.gif\fR because Wget is simply counting the |
| 2024 |
number of hops (up to 2) away from \fI1.html\fR in order to determine |
| 2025 |
where to stop the recursion. However, with this command: |
| 2026 |
.Sp |
| 2027 |
.Vb 1 |
| 2028 |
\& wget \-r \-l 2 \-p http://<site>/1.html |
| 2029 |
.Ve |
| 2030 |
.Sp |
| 2031 |
all the above files \fIand\fR \fI3.html\fR's requisite \fI3.gif\fR |
| 2032 |
will be downloaded. Similarly, |
| 2033 |
.Sp |
| 2034 |
.Vb 1 |
| 2035 |
\& wget \-r \-l 1 \-p http://<site>/1.html |
| 2036 |
.Ve |
| 2037 |
.Sp |
| 2038 |
will cause \fI1.html\fR, \fI1.gif\fR, \fI2.html\fR, and \fI2.gif\fR |
| 2039 |
to be downloaded. One might think that: |
| 2040 |
.Sp |
| 2041 |
.Vb 1 |
| 2042 |
\& wget \-r \-l 0 \-p http://<site>/1.html |
| 2043 |
.Ve |
| 2044 |
.Sp |
| 2045 |
would download just \fI1.html\fR and \fI1.gif\fR, but unfortunately |
| 2046 |
this is not the case, because \fB\-l 0\fR is equivalent to |
| 2047 |
\&\fB\-l inf\fR\-\-\-that is, infinite recursion. To download a single HTML |
| 2048 |
page (or a handful of them, all specified on the command-line or in a |
| 2049 |
\&\fB\-i\fR URL input file) and its (or their) requisites, simply leave off |
| 2050 |
\&\fB\-r\fR and \fB\-l\fR: |
| 2051 |
.Sp |
| 2052 |
.Vb 1 |
| 2053 |
\& wget \-p http://<site>/1.html |
| 2054 |
.Ve |
| 2055 |
.Sp |
| 2056 |
Note that Wget will behave as if \fB\-r\fR had been specified, but only |
| 2057 |
that single page and its requisites will be downloaded. Links from that |
| 2058 |
page to external documents will not be followed. Actually, to download |
| 2059 |
a single page and all its requisites (even if they exist on separate |
| 2060 |
websites), and make sure the lot displays properly locally, this author |
| 2061 |
likes to use a few options in addition to \fB\-p\fR: |
| 2062 |
.Sp |
| 2063 |
.Vb 1 |
| 2064 |
\& wget \-E \-H \-k \-K \-p http://<site>/<document> |
| 2065 |
.Ve |
| 2066 |
.Sp |
| 2067 |
To finish off this topic, it's worth knowing that Wget's idea of an |
| 2068 |
external document link is any URL specified in an \f(CW\*(C`<A>\*(C'\fR tag, an |
| 2069 |
\&\f(CW\*(C`<AREA>\*(C'\fR tag, or a \f(CW\*(C`<LINK>\*(C'\fR tag other than \f(CW\*(C`<LINK |
| 2070 |
REL="stylesheet">\*(C'\fR. |
| 2071 |
.IP \fB\-\-strict\-comments\fR 4 |
| 2072 |
.IX Item "--strict-comments" |
| 2073 |
Turn on strict parsing of HTML comments. The default is to terminate |
| 2074 |
comments at the first occurrence of \fB\-\->\fR. |
| 2075 |
.Sp |
| 2076 |
According to specifications, HTML comments are expressed as SGML |
| 2077 |
\&\fIdeclarations\fR. Declaration is special markup that begins with |
| 2078 |
\&\fB<!\fR and ends with \fB>\fR, such as \fB<!DOCTYPE ...>\fR, that |
| 2079 |
may contain comments between a pair of \fB\-\-\fR delimiters. HTML |
| 2080 |
comments are "empty declarations", SGML declarations without any |
| 2081 |
non-comment text. Therefore, \fB<!\-\-foo\-\->\fR is a valid comment, and |
| 2082 |
so is \fB<!\-\-one\-\- \-\-two\-\->\fR, but \fB<!\-\-1\-\-2\-\->\fR is not. |
| 2083 |
.Sp |
| 2084 |
On the other hand, most HTML writers don't perceive comments as anything |
| 2085 |
other than text delimited with \fB<!\-\-\fR and \fB\-\->\fR, which is not |
| 2086 |
quite the same. For example, something like \fB<!\-\-\-\-\-\-\-\-\-\-\-\->\fR |
| 2087 |
works as a valid comment as long as the number of dashes is a multiple |
| 2088 |
of four (!). If not, the comment technically lasts until the next |
| 2089 |
\&\fB\-\-\fR, which may be at the other end of the document. Because of |
| 2090 |
this, many popular browsers completely ignore the specification and |
| 2091 |
implement what users have come to expect: comments delimited with |
| 2092 |
\&\fB<!\-\-\fR and \fB\-\->\fR. |
| 2093 |
.Sp |
| 2094 |
Until version 1.9, Wget interpreted comments strictly, which resulted in |
| 2095 |
missing links in many web pages that displayed fine in browsers, but had |
| 2096 |
the misfortune of containing non-compliant comments. Beginning with |
| 2097 |
version 1.9, Wget has joined the ranks of clients that implements |
| 2098 |
"naive" comments, terminating each comment at the first occurrence of |
| 2099 |
\&\fB\-\->\fR. |
| 2100 |
.Sp |
| 2101 |
If, for whatever reason, you want strict comment parsing, use this |
| 2102 |
option to turn it on. |
| 2103 |
.SS "Recursive Accept/Reject Options" |
| 2104 |
.IX Subsection "Recursive Accept/Reject Options" |
| 2105 |
.IP "\fB\-A\fR \fIacclist\fR \fB\-\-accept\fR \fIacclist\fR" 4 |
| 2106 |
.IX Item "-A acclist --accept acclist" |
| 2107 |
.PD 0 |
| 2108 |
.IP "\fB\-R\fR \fIrejlist\fR \fB\-\-reject\fR \fIrejlist\fR" 4 |
| 2109 |
.IX Item "-R rejlist --reject rejlist" |
| 2110 |
.PD |
| 2111 |
Specify comma-separated lists of file name suffixes or patterns to |
| 2112 |
accept or reject. Note that if |
| 2113 |
any of the wildcard characters, \fB*\fR, \fB?\fR, \fB[\fR or |
| 2114 |
\&\fB]\fR, appear in an element of \fIacclist\fR or \fIrejlist\fR, |
| 2115 |
it will be treated as a pattern, rather than a suffix. |
| 2116 |
In this case, you have to enclose the pattern into quotes to prevent |
| 2117 |
your shell from expanding it, like in \fB\-A "*.mp3"\fR or \fB\-A '*.mp3'\fR. |
| 2118 |
.IP "\fB\-\-accept\-regex\fR \fIurlregex\fR" 4 |
| 2119 |
.IX Item "--accept-regex urlregex" |
| 2120 |
.PD 0 |
| 2121 |
.IP "\fB\-\-reject\-regex\fR \fIurlregex\fR" 4 |
| 2122 |
.IX Item "--reject-regex urlregex" |
| 2123 |
.PD |
| 2124 |
Specify a regular expression to accept or reject the complete URL. |
| 2125 |
.IP "\fB\-\-regex\-type\fR \fIregextype\fR" 4 |
| 2126 |
.IX Item "--regex-type regextype" |
| 2127 |
Specify the regular expression type. Possible types are \fBposix\fR or |
| 2128 |
\&\fBpcre\fR. Note that to be able to use \fBpcre\fR type, wget has to be |
| 2129 |
compiled with libpcre support. |
| 2130 |
.IP "\fB\-D\fR \fIdomain-list\fR" 4 |
| 2131 |
.IX Item "-D domain-list" |
| 2132 |
.PD 0 |
| 2133 |
.IP \fB\-\-domains=\fR\fIdomain-list\fR 4 |
| 2134 |
.IX Item "--domains=domain-list" |
| 2135 |
.PD |
| 2136 |
Set domains to be followed. \fIdomain-list\fR is a comma-separated list |
| 2137 |
of domains. Note that it does \fInot\fR turn on \fB\-H\fR. |
| 2138 |
.IP "\fB\-\-exclude\-domains\fR \fIdomain-list\fR" 4 |
| 2139 |
.IX Item "--exclude-domains domain-list" |
| 2140 |
Specify the domains that are \fInot\fR to be followed. |
| 2141 |
.IP \fB\-\-follow\-ftp\fR 4 |
| 2142 |
.IX Item "--follow-ftp" |
| 2143 |
Follow FTP links from HTML documents. Without this option, |
| 2144 |
Wget will ignore all the FTP links. |
| 2145 |
.IP \fB\-\-follow\-tags=\fR\fIlist\fR 4 |
| 2146 |
.IX Item "--follow-tags=list" |
| 2147 |
Wget has an internal table of HTML tag / attribute pairs that it |
| 2148 |
considers when looking for linked documents during a recursive |
| 2149 |
retrieval. If a user wants only a subset of those tags to be |
| 2150 |
considered, however, he or she should be specify such tags in a |
| 2151 |
comma-separated \fIlist\fR with this option. |
| 2152 |
.IP \fB\-\-ignore\-tags=\fR\fIlist\fR 4 |
| 2153 |
.IX Item "--ignore-tags=list" |
| 2154 |
This is the opposite of the \fB\-\-follow\-tags\fR option. To skip |
| 2155 |
certain HTML tags when recursively looking for documents to download, |
| 2156 |
specify them in a comma-separated \fIlist\fR. |
| 2157 |
.Sp |
| 2158 |
In the past, this option was the best bet for downloading a single page |
| 2159 |
and its requisites, using a command-line like: |
| 2160 |
.Sp |
| 2161 |
.Vb 1 |
| 2162 |
\& wget \-\-ignore\-tags=a,area \-H \-k \-K \-r http://<site>/<document> |
| 2163 |
.Ve |
| 2164 |
.Sp |
| 2165 |
However, the author of this option came across a page with tags like |
| 2166 |
\&\f(CW\*(C`<LINK REL="home" HREF="/">\*(C'\fR and came to the realization that |
| 2167 |
specifying tags to ignore was not enough. One can't just tell Wget to |
| 2168 |
ignore \f(CW\*(C`<LINK>\*(C'\fR, because then stylesheets will not be downloaded. |
| 2169 |
Now the best bet for downloading a single page and its requisites is the |
| 2170 |
dedicated \fB\-\-page\-requisites\fR option. |
| 2171 |
.IP \fB\-\-ignore\-case\fR 4 |
| 2172 |
.IX Item "--ignore-case" |
| 2173 |
Ignore case when matching files and directories. This influences the |
| 2174 |
behavior of \-R, \-A, \-I, and \-X options, as well as globbing |
| 2175 |
implemented when downloading from FTP sites. For example, with this |
| 2176 |
option, \fB\-A "*.txt"\fR will match \fBfile1.txt\fR, but also |
| 2177 |
\&\fBfile2.TXT\fR, \fBfile3.TxT\fR, and so on. |
| 2178 |
The quotes in the example are to prevent the shell from expanding the |
| 2179 |
pattern. |
| 2180 |
.IP \fB\-H\fR 4 |
| 2181 |
.IX Item "-H" |
| 2182 |
.PD 0 |
| 2183 |
.IP \fB\-\-span\-hosts\fR 4 |
| 2184 |
.IX Item "--span-hosts" |
| 2185 |
.PD |
| 2186 |
Enable spanning across hosts when doing recursive retrieving. |
| 2187 |
.IP \fB\-L\fR 4 |
| 2188 |
.IX Item "-L" |
| 2189 |
.PD 0 |
| 2190 |
.IP \fB\-\-relative\fR 4 |
| 2191 |
.IX Item "--relative" |
| 2192 |
.PD |
| 2193 |
Follow relative links only. Useful for retrieving a specific home page |
| 2194 |
without any distractions, not even those from the same hosts. |
| 2195 |
.IP "\fB\-I\fR \fIlist\fR" 4 |
| 2196 |
.IX Item "-I list" |
| 2197 |
.PD 0 |
| 2198 |
.IP \fB\-\-include\-directories=\fR\fIlist\fR 4 |
| 2199 |
.IX Item "--include-directories=list" |
| 2200 |
.PD |
| 2201 |
Specify a comma-separated list of directories you wish to follow when |
| 2202 |
downloading. Elements |
| 2203 |
of \fIlist\fR may contain wildcards. |
| 2204 |
.IP "\fB\-X\fR \fIlist\fR" 4 |
| 2205 |
.IX Item "-X list" |
| 2206 |
.PD 0 |
| 2207 |
.IP \fB\-\-exclude\-directories=\fR\fIlist\fR 4 |
| 2208 |
.IX Item "--exclude-directories=list" |
| 2209 |
.PD |
| 2210 |
Specify a comma-separated list of directories you wish to exclude from |
| 2211 |
download. Elements of |
| 2212 |
\&\fIlist\fR may contain wildcards. |
| 2213 |
.IP \fB\-np\fR 4 |
| 2214 |
.IX Item "-np" |
| 2215 |
.PD 0 |
| 2216 |
.IP \fB\-\-no\-parent\fR 4 |
| 2217 |
.IX Item "--no-parent" |
| 2218 |
.PD |
| 2219 |
Do not ever ascend to the parent directory when retrieving recursively. |
| 2220 |
This is a useful option, since it guarantees that only the files |
| 2221 |
\&\fIbelow\fR a certain hierarchy will be downloaded. |
| 2222 |
.SH ENVIRONMENT |
| 2223 |
.IX Header "ENVIRONMENT" |
| 2224 |
Wget supports proxies for both HTTP and FTP retrievals. The |
| 2225 |
standard way to specify proxy location, which Wget recognizes, is using |
| 2226 |
the following environment variables: |
| 2227 |
.IP \fBhttp_proxy\fR 4 |
| 2228 |
.IX Item "http_proxy" |
| 2229 |
.PD 0 |
| 2230 |
.IP \fBhttps_proxy\fR 4 |
| 2231 |
.IX Item "https_proxy" |
| 2232 |
.PD |
| 2233 |
If set, the \fBhttp_proxy\fR and \fBhttps_proxy\fR variables should |
| 2234 |
contain the URLs of the proxies for HTTP and HTTPS |
| 2235 |
connections respectively. |
| 2236 |
.IP \fBftp_proxy\fR 4 |
| 2237 |
.IX Item "ftp_proxy" |
| 2238 |
This variable should contain the URL of the proxy for FTP |
| 2239 |
connections. It is quite common that \fBhttp_proxy\fR and |
| 2240 |
\&\fBftp_proxy\fR are set to the same URL. |
| 2241 |
.IP \fBno_proxy\fR 4 |
| 2242 |
.IX Item "no_proxy" |
| 2243 |
This variable should contain a comma-separated list of domain extensions |
| 2244 |
proxy should \fInot\fR be used for. For instance, if the value of |
| 2245 |
\&\fBno_proxy\fR is \fB.mit.edu\fR, proxy will not be used to retrieve |
| 2246 |
documents from MIT. |
| 2247 |
.SH "EXIT STATUS" |
| 2248 |
.IX Header "EXIT STATUS" |
| 2249 |
Wget may return one of several error codes if it encounters problems. |
| 2250 |
.ie n .IP 0 4 |
| 2251 |
.el .IP \f(CW0\fR 4 |
| 2252 |
.IX Item "0" |
| 2253 |
No problems occurred. |
| 2254 |
.ie n .IP 1 4 |
| 2255 |
.el .IP \f(CW1\fR 4 |
| 2256 |
.IX Item "1" |
| 2257 |
Generic error code. |
| 2258 |
.ie n .IP 2 4 |
| 2259 |
.el .IP \f(CW2\fR 4 |
| 2260 |
.IX Item "2" |
| 2261 |
Parse error\-\-\-for instance, when parsing command-line options, the |
| 2262 |
\&\fB.wgetrc\fR or \fB.netrc\fR... |
| 2263 |
.ie n .IP 3 4 |
| 2264 |
.el .IP \f(CW3\fR 4 |
| 2265 |
.IX Item "3" |
| 2266 |
File I/O error. |
| 2267 |
.ie n .IP 4 4 |
| 2268 |
.el .IP \f(CW4\fR 4 |
| 2269 |
.IX Item "4" |
| 2270 |
Network failure. |
| 2271 |
.ie n .IP 5 4 |
| 2272 |
.el .IP \f(CW5\fR 4 |
| 2273 |
.IX Item "5" |
| 2274 |
SSL verification failure. |
| 2275 |
.ie n .IP 6 4 |
| 2276 |
.el .IP \f(CW6\fR 4 |
| 2277 |
.IX Item "6" |
| 2278 |
Username/password authentication failure. |
| 2279 |
.ie n .IP 7 4 |
| 2280 |
.el .IP \f(CW7\fR 4 |
| 2281 |
.IX Item "7" |
| 2282 |
Protocol errors. |
| 2283 |
.ie n .IP 8 4 |
| 2284 |
.el .IP \f(CW8\fR 4 |
| 2285 |
.IX Item "8" |
| 2286 |
Server issued an error response. |
| 2287 |
.PP |
| 2288 |
With the exceptions of 0 and 1, the lower-numbered exit codes take |
| 2289 |
precedence over higher-numbered ones, when multiple types of errors |
| 2290 |
are encountered. |
| 2291 |
.PP |
| 2292 |
In versions of Wget prior to 1.12, Wget's exit status tended to be |
| 2293 |
unhelpful and inconsistent. Recursive downloads would virtually always |
| 2294 |
return 0 (success), regardless of any issues encountered, and |
| 2295 |
non-recursive fetches only returned the status corresponding to the |
| 2296 |
most recently-attempted download. |
| 2297 |
.SH FILES |
| 2298 |
.IX Header "FILES" |
| 2299 |
.IP \fB/etc/wgetrc\fR 4 |
| 2300 |
.IX Item "/etc/wgetrc" |
| 2301 |
Default location of the \fIglobal\fR startup file. |
| 2302 |
.IP \fB.wgetrc\fR 4 |
| 2303 |
.IX Item ".wgetrc" |
| 2304 |
User startup file. |
| 2305 |
.SH BUGS |
| 2306 |
.IX Header "BUGS" |
| 2307 |
You are welcome to submit bug reports via the GNU Wget bug tracker (see |
| 2308 |
<\fBhttps://savannah.gnu.org/bugs/?func=additem&group=wget\fR>) or to our |
| 2309 |
mailing list <\fBbug\[email protected]\fR>. |
| 2310 |
.PP |
| 2311 |
Visit <\fBhttps://lists.gnu.org/mailman/listinfo/bug\-wget\fR> to |
| 2312 |
get more info (how to subscribe, list archives, ...). |
| 2313 |
.PP |
| 2314 |
Before actually submitting a bug report, please try to follow a few |
| 2315 |
simple guidelines. |
| 2316 |
.IP 1. 4 |
| 2317 |
Please try to ascertain that the behavior you see really is a bug. If |
| 2318 |
Wget crashes, it's a bug. If Wget does not behave as documented, |
| 2319 |
it's a bug. If things work strange, but you are not sure about the way |
| 2320 |
they are supposed to work, it might well be a bug, but you might want to |
| 2321 |
double-check the documentation and the mailing lists. |
| 2322 |
.IP 2. 4 |
| 2323 |
Try to repeat the bug in as simple circumstances as possible. E.g. if |
| 2324 |
Wget crashes while downloading \fBwget \-rl0 \-kKE \-t5 \-\-no\-proxy |
| 2325 |
http://example.com \-o /tmp/log\fR, you should try to see if the crash is |
| 2326 |
repeatable, and if will occur with a simpler set of options. You might |
| 2327 |
even try to start the download at the page where the crash occurred to |
| 2328 |
see if that page somehow triggered the crash. |
| 2329 |
.Sp |
| 2330 |
Also, while I will probably be interested to know the contents of your |
| 2331 |
\&\fI.wgetrc\fR file, just dumping it into the debug message is probably |
| 2332 |
a bad idea. Instead, you should first try to see if the bug repeats |
| 2333 |
with \fI.wgetrc\fR moved out of the way. Only if it turns out that |
| 2334 |
\&\fI.wgetrc\fR settings affect the bug, mail me the relevant parts of |
| 2335 |
the file. |
| 2336 |
.IP 3. 4 |
| 2337 |
Please start Wget with \fB\-d\fR option and send us the resulting |
| 2338 |
output (or relevant parts thereof). If Wget was compiled without |
| 2339 |
debug support, recompile it\-\-\-it is \fImuch\fR easier to trace bugs |
| 2340 |
with debug support on. |
| 2341 |
.Sp |
| 2342 |
Note: please make sure to remove any potentially sensitive information |
| 2343 |
from the debug log before sending it to the bug address. The |
| 2344 |
\&\f(CW\*(C`\-d\*(C'\fR won't go out of its way to collect sensitive information, |
| 2345 |
but the log \fIwill\fR contain a fairly complete transcript of Wget's |
| 2346 |
communication with the server, which may include passwords and pieces |
| 2347 |
of downloaded data. Since the bug address is publicly archived, you |
| 2348 |
may assume that all bug reports are visible to the public. |
| 2349 |
.IP 4. 4 |
| 2350 |
If Wget has crashed, try to run it in a debugger, e.g. \f(CW\*(C`gdb \`which |
| 2351 |
wget\` core\*(C'\fR and type \f(CW\*(C`where\*(C'\fR to get the backtrace. This may not |
| 2352 |
work if the system administrator has disabled core files, but it is |
| 2353 |
safe to try. |
| 2354 |
.SH "SEE ALSO" |
| 2355 |
.IX Header "SEE ALSO" |
| 2356 |
This is \fBnot\fR the complete manual for GNU Wget. |
| 2357 |
For more complete information, including more detailed explanations of |
| 2358 |
some of the options, and a number of commands available |
| 2359 |
for use with \fI.wgetrc\fR files and the \fB\-e\fR option, see the GNU |
| 2360 |
Info entry for \fIwget\fR. |
| 2361 |
.PP |
| 2362 |
Also see \fBwget2\fR\|(1), the updated version of GNU Wget with even better |
| 2363 |
support for recursive downloading and modern protocols like HTTP/2. |
| 2364 |
.SH AUTHOR |
| 2365 |
.IX Header "AUTHOR" |
| 2366 |
Originally written by Hrvoje Nikšić <[email protected]>. |
| 2367 |
Currently maintained by Darshit Shah <[email protected]> and |
| 2368 |
Tim Rühsen <[email protected]>. |
| 2369 |
.SH COPYRIGHT |
| 2370 |
.IX Header "COPYRIGHT" |
| 2371 |
Copyright (c) 1996\-\-2011, 2015, 2018\-\-2024 Free Software |
| 2372 |
Foundation, Inc. |
| 2373 |
.PP |
| 2374 |
Permission is granted to copy, distribute and/or modify this document |
| 2375 |
under the terms of the GNU Free Documentation License, Version 1.3 or |
| 2376 |
any later version published by the Free Software Foundation; with no |
| 2377 |
Invariant Sections, with no Front-Cover Texts, and with no Back-Cover |
| 2378 |
Texts. A copy of the license is included in the section entitled |
| 2379 |
"GNU Free Documentation License". |