| 1 |
# curl tutorial |
| 2 |
|
| 3 |
## Simple Usage |
| 4 |
|
| 5 |
Get the main page from Netscape's web-server: |
| 6 |
|
| 7 |
curl http://www.netscape.com/ |
| 8 |
|
| 9 |
Get the README file the user's home directory at funet's ftp-server: |
| 10 |
|
| 11 |
curl ftp://ftp.funet.fi/README |
| 12 |
|
| 13 |
Get a web page from a server using port 8000: |
| 14 |
|
| 15 |
curl http://www.weirdserver.com:8000/ |
| 16 |
|
| 17 |
Get a directory listing of an FTP site: |
| 18 |
|
| 19 |
curl ftp://cool.haxx.se/ |
| 20 |
|
| 21 |
Get the definition of curl from a dictionary: |
| 22 |
|
| 23 |
curl dict://dict.org/m:curl |
| 24 |
|
| 25 |
Fetch two documents at once: |
| 26 |
|
| 27 |
curl ftp://cool.haxx.se/ http://www.weirdserver.com:8000/ |
| 28 |
|
| 29 |
Get a file off an FTPS server: |
| 30 |
|
| 31 |
curl ftps://files.are.secure.com/secrets.txt |
| 32 |
|
| 33 |
or use the more appropriate FTPS way to get the same file: |
| 34 |
|
| 35 |
curl --ftp-ssl ftp://files.are.secure.com/secrets.txt |
| 36 |
|
| 37 |
Get a file from an SSH server using SFTP: |
| 38 |
|
| 39 |
curl -u username sftp://example.com/etc/issue |
| 40 |
|
| 41 |
Get a file from an SSH server using SCP using a private key (not |
| 42 |
password-protected) to authenticate: |
| 43 |
|
| 44 |
curl -u username: --key ~/.ssh/id_rsa scp://example.com/~/file.txt |
| 45 |
|
| 46 |
Get a file from an SSH server using SCP using a private key |
| 47 |
(password-protected) to authenticate: |
| 48 |
|
| 49 |
curl -u username: --key ~/.ssh/id_rsa --pass private_key_password |
| 50 |
scp://example.com/~/file.txt |
| 51 |
|
| 52 |
Get the main page from an IPv6 web server: |
| 53 |
|
| 54 |
curl "http://[2001:1890:1112:1::20]/" |
| 55 |
|
| 56 |
Get a file from an SMB server: |
| 57 |
|
| 58 |
curl -u "domain\username:passwd" smb://server.example.com/share/file.txt |
| 59 |
|
| 60 |
## Download to a File |
| 61 |
|
| 62 |
Get a web page and store in a local file with a specific name: |
| 63 |
|
| 64 |
curl -o thatpage.html http://www.netscape.com/ |
| 65 |
|
| 66 |
Get a web page and store in a local file, make the local file get the name of |
| 67 |
the remote document (if no file name part is specified in the URL, this will |
| 68 |
fail): |
| 69 |
|
| 70 |
curl -O http://www.netscape.com/index.html |
| 71 |
|
| 72 |
Fetch two files and store them with their remote names: |
| 73 |
|
| 74 |
curl -O www.haxx.se/index.html -O curl.haxx.se/download.html |
| 75 |
|
| 76 |
## Using Passwords |
| 77 |
|
| 78 |
### FTP |
| 79 |
|
| 80 |
To ftp files using name+passwd, include them in the URL like: |
| 81 |
|
| 82 |
curl ftp://name:[email protected]:port/full/path/to/file |
| 83 |
|
| 84 |
or specify them with the -u flag like |
| 85 |
|
| 86 |
curl -u name:passwd ftp://machine.domain:port/full/path/to/file |
| 87 |
|
| 88 |
### FTPS |
| 89 |
|
| 90 |
It is just like for FTP, but you may also want to specify and use SSL-specific |
| 91 |
options for certificates etc. |
| 92 |
|
| 93 |
Note that using `FTPS://` as prefix is the "implicit" way as described in the |
| 94 |
standards while the recommended "explicit" way is done by using FTP:// and the |
| 95 |
`--ftp-ssl` option. |
| 96 |
|
| 97 |
### SFTP / SCP |
| 98 |
|
| 99 |
This is similar to FTP, but you can use the `--key` option to specify a |
| 100 |
private key to use instead of a password. Note that the private key may itself |
| 101 |
be protected by a password that is unrelated to the login password of the |
| 102 |
remote system; this password is specified using the `--pass` option. |
| 103 |
Typically, curl will automatically extract the public key from the private key |
| 104 |
file, but in cases where curl does not have the proper library support, a |
| 105 |
matching public key file must be specified using the `--pubkey` option. |
| 106 |
|
| 107 |
### HTTP |
| 108 |
|
| 109 |
Curl also supports user and password in HTTP URLs, thus you can pick a file |
| 110 |
like: |
| 111 |
|
| 112 |
curl http://name:[email protected]/full/path/to/file |
| 113 |
|
| 114 |
or specify user and password separately like in |
| 115 |
|
| 116 |
curl -u name:passwd http://machine.domain/full/path/to/file |
| 117 |
|
| 118 |
HTTP offers many different methods of authentication and curl supports |
| 119 |
several: Basic, Digest, NTLM and Negotiate (SPNEGO). Without telling which |
| 120 |
method to use, curl defaults to Basic. You can also ask curl to pick the most |
| 121 |
secure ones out of the ones that the server accepts for the given URL, by |
| 122 |
using `--anyauth`. |
| 123 |
|
| 124 |
**Note**! According to the URL specification, HTTP URLs can not contain a user |
| 125 |
and password, so that style will not work when using curl via a proxy, even |
| 126 |
though curl allows it at other times. When using a proxy, you _must_ use the |
| 127 |
`-u` style for user and password. |
| 128 |
|
| 129 |
### HTTPS |
| 130 |
|
| 131 |
Probably most commonly used with private certificates, as explained below. |
| 132 |
|
| 133 |
## Proxy |
| 134 |
|
| 135 |
curl supports both HTTP and SOCKS proxy servers, with optional authentication. |
| 136 |
It does not have special support for FTP proxy servers since there are no |
| 137 |
standards for those, but it can still be made to work with many of them. You |
| 138 |
can also use both HTTP and SOCKS proxies to transfer files to and from FTP |
| 139 |
servers. |
| 140 |
|
| 141 |
Get an ftp file using an HTTP proxy named my-proxy that uses port 888: |
| 142 |
|
| 143 |
curl -x my-proxy:888 ftp://ftp.leachsite.com/README |
| 144 |
|
| 145 |
Get a file from an HTTP server that requires user and password, using the |
| 146 |
same proxy as above: |
| 147 |
|
| 148 |
curl -u user:passwd -x my-proxy:888 http://www.get.this/ |
| 149 |
|
| 150 |
Some proxies require special authentication. Specify by using -U as above: |
| 151 |
|
| 152 |
curl -U user:passwd -x my-proxy:888 http://www.get.this/ |
| 153 |
|
| 154 |
A comma-separated list of hosts and domains which do not use the proxy can be |
| 155 |
specified as: |
| 156 |
|
| 157 |
curl --noproxy localhost,get.this -x my-proxy:888 http://www.get.this/ |
| 158 |
|
| 159 |
If the proxy is specified with `--proxy1.0` instead of `--proxy` or `-x`, then |
| 160 |
curl will use HTTP/1.0 instead of HTTP/1.1 for any `CONNECT` attempts. |
| 161 |
|
| 162 |
curl also supports SOCKS4 and SOCKS5 proxies with `--socks4` and `--socks5`. |
| 163 |
|
| 164 |
See also the environment variables Curl supports that offer further proxy |
| 165 |
control. |
| 166 |
|
| 167 |
Most FTP proxy servers are set up to appear as a normal FTP server from the |
| 168 |
client's perspective, with special commands to select the remote FTP server. |
| 169 |
curl supports the `-u`, `-Q` and `--ftp-account` options that can be used to |
| 170 |
set up transfers through many FTP proxies. For example, a file can be uploaded |
| 171 |
to a remote FTP server using a Blue Coat FTP proxy with the options: |
| 172 |
|
| 173 |
curl -u "[email protected] Proxy-Username:Remote-Pass" |
| 174 |
--ftp-account Proxy-Password --upload-file local-file |
| 175 |
ftp://my-ftp.proxy.server:21/remote/upload/path/ |
| 176 |
|
| 177 |
See the manual for your FTP proxy to determine the form it expects to set up |
| 178 |
transfers, and curl's `-v` option to see exactly what curl is sending. |
| 179 |
|
| 180 |
## Ranges |
| 181 |
|
| 182 |
HTTP 1.1 introduced byte-ranges. Using this, a client can request to get only |
| 183 |
one or more subparts of a specified document. Curl supports this with the `-r` |
| 184 |
flag. |
| 185 |
|
| 186 |
Get the first 100 bytes of a document: |
| 187 |
|
| 188 |
curl -r 0-99 http://www.get.this/ |
| 189 |
|
| 190 |
Get the last 500 bytes of a document: |
| 191 |
|
| 192 |
curl -r -500 http://www.get.this/ |
| 193 |
|
| 194 |
Curl also supports simple ranges for FTP files as well. Then you can only |
| 195 |
specify start and stop position. |
| 196 |
|
| 197 |
Get the first 100 bytes of a document using FTP: |
| 198 |
|
| 199 |
curl -r 0-99 ftp://www.get.this/README |
| 200 |
|
| 201 |
## Uploading |
| 202 |
|
| 203 |
### FTP / FTPS / SFTP / SCP |
| 204 |
|
| 205 |
Upload all data on stdin to a specified server: |
| 206 |
|
| 207 |
curl -T - ftp://ftp.upload.com/myfile |
| 208 |
|
| 209 |
Upload data from a specified file, login with user and password: |
| 210 |
|
| 211 |
curl -T uploadfile -u user:passwd ftp://ftp.upload.com/myfile |
| 212 |
|
| 213 |
Upload a local file to the remote site, and use the local file name at the |
| 214 |
remote site too: |
| 215 |
|
| 216 |
curl -T uploadfile -u user:passwd ftp://ftp.upload.com/ |
| 217 |
|
| 218 |
Upload a local file to get appended to the remote file: |
| 219 |
|
| 220 |
curl -T localfile -a ftp://ftp.upload.com/remotefile |
| 221 |
|
| 222 |
Curl also supports ftp upload through a proxy, but only if the proxy is |
| 223 |
configured to allow that kind of tunneling. If it does, you can run curl in a |
| 224 |
fashion similar to: |
| 225 |
|
| 226 |
curl --proxytunnel -x proxy:port -T localfile ftp.upload.com |
| 227 |
|
| 228 |
### SMB / SMBS |
| 229 |
|
| 230 |
curl -T file.txt -u "domain\username:passwd" |
| 231 |
smb://server.example.com/share/ |
| 232 |
|
| 233 |
### HTTP |
| 234 |
|
| 235 |
Upload all data on stdin to a specified HTTP site: |
| 236 |
|
| 237 |
curl -T - http://www.upload.com/myfile |
| 238 |
|
| 239 |
Note that the HTTP server must have been configured to accept PUT before this |
| 240 |
can be done successfully. |
| 241 |
|
| 242 |
For other ways to do HTTP data upload, see the POST section below. |
| 243 |
|
| 244 |
## Verbose / Debug |
| 245 |
|
| 246 |
If curl fails where it isn't supposed to, if the servers don't let you in, if |
| 247 |
you can't understand the responses: use the `-v` flag to get verbose |
| 248 |
fetching. Curl will output lots of info and what it sends and receives in |
| 249 |
order to let the user see all client-server interaction (but it won't show you |
| 250 |
the actual data). |
| 251 |
|
| 252 |
curl -v ftp://ftp.upload.com/ |
| 253 |
|
| 254 |
To get even more details and information on what curl does, try using the |
| 255 |
`--trace` or `--trace-ascii` options with a given file name to log to, like |
| 256 |
this: |
| 257 |
|
| 258 |
curl --trace trace.txt www.haxx.se |
| 259 |
|
| 260 |
|
| 261 |
## Detailed Information |
| 262 |
|
| 263 |
Different protocols provide different ways of getting detailed information |
| 264 |
about specific files/documents. To get curl to show detailed information about |
| 265 |
a single file, you should use `-I`/`--head` option. It displays all available |
| 266 |
info on a single file for HTTP and FTP. The HTTP information is a lot more |
| 267 |
extensive. |
| 268 |
|
| 269 |
For HTTP, you can get the header information (the same as `-I` would show) |
| 270 |
shown before the data by using `-i`/`--include`. Curl understands the |
| 271 |
`-D`/`--dump-header` option when getting files from both FTP and HTTP, and it |
| 272 |
will then store the headers in the specified file. |
| 273 |
|
| 274 |
Store the HTTP headers in a separate file (headers.txt in the example): |
| 275 |
|
| 276 |
curl --dump-header headers.txt curl.haxx.se |
| 277 |
|
| 278 |
Note that headers stored in a separate file can be very useful at a later time |
| 279 |
if you want curl to use cookies sent by the server. More about that in the |
| 280 |
cookies section. |
| 281 |
|
| 282 |
## POST (HTTP) |
| 283 |
|
| 284 |
It's easy to post data using curl. This is done using the `-d <data>` option. |
| 285 |
The post data must be urlencoded. |
| 286 |
|
| 287 |
Post a simple "name" and "phone" guestbook. |
| 288 |
|
| 289 |
curl -d "name=Rafael%20Sagula&phone=3320780" http://www.where.com/guest.cgi |
| 290 |
|
| 291 |
How to post a form with curl, lesson #1: |
| 292 |
|
| 293 |
Dig out all the `<input>` tags in the form that you want to fill in. |
| 294 |
|
| 295 |
If there's a "normal" post, you use `-d` to post. `-d` takes a full "post |
| 296 |
string", which is in the format |
| 297 |
|
| 298 |
<variable1>=<data1>&<variable2>=<data2>&... |
| 299 |
|
| 300 |
The 'variable' names are the names set with `"name="` in the `<input>` tags, |
| 301 |
and the data is the contents you want to fill in for the inputs. The data |
| 302 |
*must* be properly URL encoded. That means you replace space with + and that |
| 303 |
you replace weird letters with %XX where XX is the hexadecimal representation |
| 304 |
of the letter's ASCII code. |
| 305 |
|
| 306 |
Example: |
| 307 |
|
| 308 |
(page located at `http://www.formpost.com/getthis/`) |
| 309 |
|
| 310 |
<form action="post.cgi" method="post"> |
| 311 |
<input name=user size=10> |
| 312 |
<input name=pass type=password size=10> |
| 313 |
<input name=id type=hidden value="blablabla"> |
| 314 |
<input name=ding value="submit"> |
| 315 |
</form> |
| 316 |
|
| 317 |
We want to enter user 'foobar' with password '12345'. |
| 318 |
|
| 319 |
To post to this, you enter a curl command line like: |
| 320 |
|
| 321 |
curl -d "user=foobar&pass=12345&id=blablabla&ding=submit" |
| 322 |
http://www.formpost.com/getthis/post.cgi |
| 323 |
|
| 324 |
While `-d` uses the application/x-www-form-urlencoded mime-type, generally |
| 325 |
understood by CGI's and similar, curl also supports the more capable |
| 326 |
multipart/form-data type. This latter type supports things like file upload. |
| 327 |
|
| 328 |
`-F` accepts parameters like `-F "name=contents"`. If you want the contents to |
| 329 |
be read from a file, use `@filename` as contents. When specifying a file, you |
| 330 |
can also specify the file content type by appending `;type=<mime type>` to the |
| 331 |
file name. You can also post the contents of several files in one field. For |
| 332 |
example, the field name 'coolfiles' is used to send three files, with |
| 333 |
different content types using the following syntax: |
| 334 |
|
| 335 |
curl -F "[email protected];type=image/gif,fil2.txt,fil3.html" |
| 336 |
http://www.post.com/postit.cgi |
| 337 |
|
| 338 |
If the content-type is not specified, curl will try to guess from the file |
| 339 |
extension (it only knows a few), or use the previously specified type (from an |
| 340 |
earlier file if several files are specified in a list) or else it will use the |
| 341 |
default type 'application/octet-stream'. |
| 342 |
|
| 343 |
Emulate a fill-in form with `-F`. Let's say you fill in three fields in a |
| 344 |
form. One field is a file name which to post, one field is your name and one |
| 345 |
field is a file description. We want to post the file we have written named |
| 346 |
"cooltext.txt". To let curl do the posting of this data instead of your |
| 347 |
favourite browser, you have to read the HTML source of the form page and find |
| 348 |
the names of the input fields. In our example, the input field names are |
| 349 |
'file', 'yourname' and 'filedescription'. |
| 350 |
|
| 351 |
curl -F "[email protected]" -F "yourname=Daniel" |
| 352 |
-F "filedescription=Cool text file with cool text inside" |
| 353 |
http://www.post.com/postit.cgi |
| 354 |
|
| 355 |
To send two files in one post you can do it in two ways: |
| 356 |
|
| 357 |
Send multiple files in a single "field" with a single field name: |
| 358 |
|
| 359 |
curl -F "[email protected],cat.gif" $URL |
| 360 |
|
| 361 |
Send two fields with two field names |
| 362 |
|
| 363 |
curl -F "[email protected]" -F "[email protected]" $URL |
| 364 |
|
| 365 |
To send a field value literally without interpreting a leading `@` or `<`, or |
| 366 |
an embedded `;type=`, use `--form-string` instead of `-F`. This is recommended |
| 367 |
when the value is obtained from a user or some other unpredictable |
| 368 |
source. Under these circumstances, using `-F` instead of `--form-string` could |
| 369 |
allow a user to trick curl into uploading a file. |
| 370 |
|
| 371 |
## Referrer |
| 372 |
|
| 373 |
An HTTP request has the option to include information about which address |
| 374 |
referred it to the actual page. Curl allows you to specify the referrer to be |
| 375 |
used on the command line. It is especially useful to fool or trick stupid |
| 376 |
servers or CGI scripts that rely on that information being available or |
| 377 |
contain certain data. |
| 378 |
|
| 379 |
curl -e www.coolsite.com http://www.showme.com/ |
| 380 |
|
| 381 |
## User Agent |
| 382 |
|
| 383 |
An HTTP request has the option to include information about the browser that |
| 384 |
generated the request. Curl allows it to be specified on the command line. It |
| 385 |
is especially useful to fool or trick stupid servers or CGI scripts that only |
| 386 |
accept certain browsers. |
| 387 |
|
| 388 |
Example: |
| 389 |
|
| 390 |
curl -A 'Mozilla/3.0 (Win95; I)' http://www.nationsbank.com/ |
| 391 |
|
| 392 |
Other common strings: |
| 393 |
|
| 394 |
- `Mozilla/3.0 (Win95; I)` - Netscape Version 3 for Windows 95 |
| 395 |
- `Mozilla/3.04 (Win95; U)` - Netscape Version 3 for Windows 95 |
| 396 |
- `Mozilla/2.02 (OS/2; U)` - Netscape Version 2 for OS/2 |
| 397 |
- `Mozilla/4.04 [en] (X11; U; AIX 4.2; Nav)` - Netscape for AIX |
| 398 |
- `Mozilla/4.05 [en] (X11; U; Linux 2.0.32 i586)` - Netscape for Linux |
| 399 |
|
| 400 |
Note that Internet Explorer tries hard to be compatible in every way: |
| 401 |
|
| 402 |
- `Mozilla/4.0 (compatible; MSIE 4.01; Windows 95)` - MSIE for W95 |
| 403 |
|
| 404 |
Mozilla is not the only possible User-Agent name: |
| 405 |
|
| 406 |
- `Konqueror/1.0` - KDE File Manager desktop client |
| 407 |
- `Lynx/2.7.1 libwww-FM/2.14` - Lynx command line browser |
| 408 |
|
| 409 |
## Cookies |
| 410 |
|
| 411 |
Cookies are generally used by web servers to keep state information at the |
| 412 |
client's side. The server sets cookies by sending a response line in the |
| 413 |
headers that looks like `Set-Cookie: <data>` where the data part then |
| 414 |
typically contains a set of `NAME=VALUE` pairs (separated by semicolons `;` |
| 415 |
like `NAME1=VALUE1; NAME2=VALUE2;`). The server can also specify for what path |
| 416 |
the "cookie" should be used for (by specifying `path=value`), when the cookie |
| 417 |
should expire (`expire=DATE`), for what domain to use it (`domain=NAME`) and |
| 418 |
if it should be used on secure connections only (`secure`). |
| 419 |
|
| 420 |
If you've received a page from a server that contains a header like: |
| 421 |
|
| 422 |
Set-Cookie: sessionid=boo123; path="/foo"; |
| 423 |
|
| 424 |
it means the server wants that first pair passed on when we get anything in a |
| 425 |
path beginning with "/foo". |
| 426 |
|
| 427 |
Example, get a page that wants my name passed in a cookie: |
| 428 |
|
| 429 |
curl -b "name=Daniel" www.sillypage.com |
| 430 |
|
| 431 |
Curl also has the ability to use previously received cookies in following |
| 432 |
sessions. If you get cookies from a server and store them in a file in a |
| 433 |
manner similar to: |
| 434 |
|
| 435 |
curl --dump-header headers www.example.com |
| 436 |
|
| 437 |
... you can then in a second connect to that (or another) site, use the |
| 438 |
cookies from the 'headers' file like: |
| 439 |
|
| 440 |
curl -b headers www.example.com |
| 441 |
|
| 442 |
While saving headers to a file is a working way to store cookies, it is |
| 443 |
however error-prone and not the preferred way to do this. Instead, make curl |
| 444 |
save the incoming cookies using the well-known netscape cookie format like |
| 445 |
this: |
| 446 |
|
| 447 |
curl -c cookies.txt www.example.com |
| 448 |
|
| 449 |
Note that by specifying `-b` you enable the "cookie awareness" and with `-L` |
| 450 |
you can make curl follow a location: (which often is used in combination with |
| 451 |
cookies). So that if a site sends cookies and a location, you can use a |
| 452 |
non-existing file to trigger the cookie awareness like: |
| 453 |
|
| 454 |
curl -L -b empty.txt www.example.com |
| 455 |
|
| 456 |
The file to read cookies from must be formatted using plain HTTP headers OR as |
| 457 |
netscape's cookie file. Curl will determine what kind it is based on the file |
| 458 |
contents. In the above command, curl will parse the header and store the |
| 459 |
cookies received from www.example.com. curl will send to the server the |
| 460 |
stored cookies which match the request as it follows the location. The file |
| 461 |
"empty.txt" may be a nonexistent file. |
| 462 |
|
| 463 |
To read and write cookies from a netscape cookie file, you can set both `-b` |
| 464 |
and `-c` to use the same file: |
| 465 |
|
| 466 |
curl -b cookies.txt -c cookies.txt www.example.com |
| 467 |
|
| 468 |
## Progress Meter |
| 469 |
|
| 470 |
The progress meter exists to show a user that something actually is |
| 471 |
happening. The different fields in the output have the following meaning: |
| 472 |
|
| 473 |
% Total % Received % Xferd Average Speed Time Curr. |
| 474 |
Dload Upload Total Current Left Speed |
| 475 |
0 151M 0 38608 0 0 9406 0 4:41:43 0:00:04 4:41:39 9287 |
| 476 |
|
| 477 |
From left-to-right: |
| 478 |
|
| 479 |
- % - percentage completed of the whole transfer |
| 480 |
- Total - total size of the whole expected transfer |
| 481 |
- % - percentage completed of the download |
| 482 |
- Received - currently downloaded amount of bytes |
| 483 |
- % - percentage completed of the upload |
| 484 |
- Xferd - currently uploaded amount of bytes |
| 485 |
- Average Speed Dload - the average transfer speed of the download |
| 486 |
- Average Speed Upload - the average transfer speed of the upload |
| 487 |
- Time Total - expected time to complete the operation |
| 488 |
- Time Current - time passed since the invoke |
| 489 |
- Time Left - expected time left to completion |
| 490 |
- Curr.Speed - the average transfer speed the last 5 seconds (the first |
| 491 |
5 seconds of a transfer is based on less time of course.) |
| 492 |
|
| 493 |
The `-#` option will display a totally different progress bar that doesn't |
| 494 |
need much explanation! |
| 495 |
|
| 496 |
## Speed Limit |
| 497 |
|
| 498 |
Curl allows the user to set the transfer speed conditions that must be met to |
| 499 |
let the transfer keep going. By using the switch `-y` and `-Y` you can make |
| 500 |
curl abort transfers if the transfer speed is below the specified lowest limit |
| 501 |
for a specified time. |
| 502 |
|
| 503 |
To have curl abort the download if the speed is slower than 3000 bytes per |
| 504 |
second for 1 minute, run: |
| 505 |
|
| 506 |
curl -Y 3000 -y 60 www.far-away-site.com |
| 507 |
|
| 508 |
This can very well be used in combination with the overall time limit, so |
| 509 |
that the above operation must be completed in whole within 30 minutes: |
| 510 |
|
| 511 |
curl -m 1800 -Y 3000 -y 60 www.far-away-site.com |
| 512 |
|
| 513 |
Forcing curl not to transfer data faster than a given rate is also possible, |
| 514 |
which might be useful if you're using a limited bandwidth connection and you |
| 515 |
don't want your transfer to use all of it (sometimes referred to as |
| 516 |
"bandwidth throttle"). |
| 517 |
|
| 518 |
Make curl transfer data no faster than 10 kilobytes per second: |
| 519 |
|
| 520 |
curl --limit-rate 10K www.far-away-site.com |
| 521 |
|
| 522 |
or |
| 523 |
|
| 524 |
curl --limit-rate 10240 www.far-away-site.com |
| 525 |
|
| 526 |
Or prevent curl from uploading data faster than 1 megabyte per second: |
| 527 |
|
| 528 |
curl -T upload --limit-rate 1M ftp://uploadshereplease.com |
| 529 |
|
| 530 |
When using the `--limit-rate` option, the transfer rate is regulated on a |
| 531 |
per-second basis, which will cause the total transfer speed to become lower |
| 532 |
than the given number. Sometimes of course substantially lower, if your |
| 533 |
transfer stalls during periods. |
| 534 |
|
| 535 |
## Config File |
| 536 |
|
| 537 |
Curl automatically tries to read the `.curlrc` file (or `_curlrc` file on |
| 538 |
Microsoft Windows systems) from the user's home dir on startup. |
| 539 |
|
| 540 |
The config file could be made up with normal command line switches, but you |
| 541 |
can also specify the long options without the dashes to make it more |
| 542 |
readable. You can separate the options and the parameter with spaces, or with |
| 543 |
`=` or `:`. Comments can be used within the file. If the first letter on a |
| 544 |
line is a `#`-symbol the rest of the line is treated as a comment. |
| 545 |
|
| 546 |
If you want the parameter to contain spaces, you must enclose the entire |
| 547 |
parameter within double quotes (`"`). Within those quotes, you specify a quote |
| 548 |
as `\"`. |
| 549 |
|
| 550 |
NOTE: You must specify options and their arguments on the same line. |
| 551 |
|
| 552 |
Example, set default time out and proxy in a config file: |
| 553 |
|
| 554 |
# We want a 30 minute timeout: |
| 555 |
-m 1800 |
| 556 |
# ... and we use a proxy for all accesses: |
| 557 |
proxy = proxy.our.domain.com:8080 |
| 558 |
|
| 559 |
White spaces ARE significant at the end of lines, but all white spaces leading |
| 560 |
up to the first characters of each line are ignored. |
| 561 |
|
| 562 |
Prevent curl from reading the default file by using -q as the first command |
| 563 |
line parameter, like: |
| 564 |
|
| 565 |
curl -q www.thatsite.com |
| 566 |
|
| 567 |
Force curl to get and display a local help page in case it is invoked without |
| 568 |
URL by making a config file similar to: |
| 569 |
|
| 570 |
# default url to get |
| 571 |
url = "http://help.with.curl.com/curlhelp.html" |
| 572 |
|
| 573 |
You can specify another config file to be read by using the `-K`/`--config` |
| 574 |
flag. If you set config file name to `-` it'll read the config from stdin, |
| 575 |
which can be handy if you want to hide options from being visible in process |
| 576 |
tables etc: |
| 577 |
|
| 578 |
echo "user = user:passwd" | curl -K - http://that.secret.site.com |
| 579 |
|
| 580 |
## Extra Headers |
| 581 |
|
| 582 |
When using curl in your own very special programs, you may end up needing |
| 583 |
to pass on your own custom headers when getting a web page. You can do |
| 584 |
this by using the `-H` flag. |
| 585 |
|
| 586 |
Example, send the header `X-you-and-me: yes` to the server when getting a |
| 587 |
page: |
| 588 |
|
| 589 |
curl -H "X-you-and-me: yes" www.love.com |
| 590 |
|
| 591 |
This can also be useful in case you want curl to send a different text in a |
| 592 |
header than it normally does. The `-H` header you specify then replaces the |
| 593 |
header curl would normally send. If you replace an internal header with an |
| 594 |
empty one, you prevent that header from being sent. To prevent the `Host:` |
| 595 |
header from being used: |
| 596 |
|
| 597 |
curl -H "Host:" www.server.com |
| 598 |
|
| 599 |
## FTP and Path Names |
| 600 |
|
| 601 |
Do note that when getting files with a `ftp://` URL, the given path is |
| 602 |
relative the directory you enter. To get the file `README` from your home |
| 603 |
directory at your ftp site, do: |
| 604 |
|
| 605 |
curl ftp://user:[email protected]/README |
| 606 |
|
| 607 |
But if you want the README file from the root directory of that very same |
| 608 |
site, you need to specify the absolute file name: |
| 609 |
|
| 610 |
curl ftp://user:[email protected]//README |
| 611 |
|
| 612 |
(I.e with an extra slash in front of the file name.) |
| 613 |
|
| 614 |
## SFTP and SCP and Path Names |
| 615 |
|
| 616 |
With sftp: and scp: URLs, the path name given is the absolute name on the |
| 617 |
server. To access a file relative to the remote user's home directory, prefix |
| 618 |
the file with `/~/` , such as: |
| 619 |
|
| 620 |
curl -u $USER sftp://home.example.com/~/.bashrc |
| 621 |
|
| 622 |
## FTP and Firewalls |
| 623 |
|
| 624 |
The FTP protocol requires one of the involved parties to open a second |
| 625 |
connection as soon as data is about to get transferred. There are two ways to |
| 626 |
do this. |
| 627 |
|
| 628 |
The default way for curl is to issue the PASV command which causes the server |
| 629 |
to open another port and await another connection performed by the |
| 630 |
client. This is good if the client is behind a firewall that doesn't allow |
| 631 |
incoming connections. |
| 632 |
|
| 633 |
curl ftp.download.com |
| 634 |
|
| 635 |
If the server, for example, is behind a firewall that doesn't allow |
| 636 |
connections on ports other than 21 (or if it just doesn't support the `PASV` |
| 637 |
command), the other way to do it is to use the `PORT` command and instruct the |
| 638 |
server to connect to the client on the given IP number and port (as parameters |
| 639 |
to the PORT command). |
| 640 |
|
| 641 |
The `-P` flag to curl supports a few different options. Your machine may have |
| 642 |
several IP-addresses and/or network interfaces and curl allows you to select |
| 643 |
which of them to use. Default address can also be used: |
| 644 |
|
| 645 |
curl -P - ftp.download.com |
| 646 |
|
| 647 |
Download with `PORT` but use the IP address of our `le0` interface (this does |
| 648 |
not work on windows): |
| 649 |
|
| 650 |
curl -P le0 ftp.download.com |
| 651 |
|
| 652 |
Download with `PORT` but use 192.168.0.10 as our IP address to use: |
| 653 |
|
| 654 |
curl -P 192.168.0.10 ftp.download.com |
| 655 |
|
| 656 |
## Network Interface |
| 657 |
|
| 658 |
Get a web page from a server using a specified port for the interface: |
| 659 |
|
| 660 |
curl --interface eth0:1 http://www.netscape.com/ |
| 661 |
|
| 662 |
or |
| 663 |
|
| 664 |
curl --interface 192.168.1.10 http://www.netscape.com/ |
| 665 |
|
| 666 |
## HTTPS |
| 667 |
|
| 668 |
Secure HTTP requires a TLS library to be installed and used when curl is |
| 669 |
built. If that is done, curl is capable of retrieving and posting documents |
| 670 |
using the HTTPS protocol. |
| 671 |
|
| 672 |
Example: |
| 673 |
|
| 674 |
curl https://www.secure-site.com |
| 675 |
|
| 676 |
curl is also capable of using client certificates to get/post files from sites |
| 677 |
that require valid certificates. The only drawback is that the certificate |
| 678 |
needs to be in PEM-format. PEM is a standard and open format to store |
| 679 |
certificates with, but it is not used by the most commonly used browsers. If |
| 680 |
you want curl to use the certificates you use with your (favourite) browser, |
| 681 |
you may need to download/compile a converter that can convert your browser's |
| 682 |
formatted certificates to PEM formatted ones. |
| 683 |
|
| 684 |
Example on how to automatically retrieve a document using a certificate with a |
| 685 |
personal password: |
| 686 |
|
| 687 |
curl -E /path/to/cert.pem:password https://secure.site.com/ |
| 688 |
|
| 689 |
If you neglect to specify the password on the command line, you will be |
| 690 |
prompted for the correct password before any data can be received. |
| 691 |
|
| 692 |
Many older HTTPS servers have problems with specific SSL or TLS versions, |
| 693 |
which newer versions of OpenSSL etc use, therefore it is sometimes useful to |
| 694 |
specify what SSL-version curl should use. Use -3, -2 or -1 to specify that |
| 695 |
exact SSL version to use (for SSLv3, SSLv2 or TLSv1 respectively): |
| 696 |
|
| 697 |
curl -2 https://secure.site.com/ |
| 698 |
|
| 699 |
Otherwise, curl will attempt to use a sensible TLS default version. |
| 700 |
|
| 701 |
## Resuming File Transfers |
| 702 |
|
| 703 |
To continue a file transfer where it was previously aborted, curl supports |
| 704 |
resume on HTTP(S) downloads as well as FTP uploads and downloads. |
| 705 |
|
| 706 |
Continue downloading a document: |
| 707 |
|
| 708 |
curl -C - -o file ftp://ftp.server.com/path/file |
| 709 |
|
| 710 |
Continue uploading a document: |
| 711 |
|
| 712 |
curl -C - -T file ftp://ftp.server.com/path/file |
| 713 |
|
| 714 |
Continue downloading a document from a web server |
| 715 |
|
| 716 |
curl -C - -o file http://www.server.com/ |
| 717 |
|
| 718 |
## Time Conditions |
| 719 |
|
| 720 |
HTTP allows a client to specify a time condition for the document it requests. |
| 721 |
It is `If-Modified-Since` or `If-Unmodified-Since`. curl allows you to specify |
| 722 |
them with the `-z`/`--time-cond` flag. |
| 723 |
|
| 724 |
For example, you can easily make a download that only gets performed if the |
| 725 |
remote file is newer than a local copy. It would be made like: |
| 726 |
|
| 727 |
curl -z local.html http://remote.server.com/remote.html |
| 728 |
|
| 729 |
Or you can download a file only if the local file is newer than the remote |
| 730 |
one. Do this by prepending the date string with a `-`, as in: |
| 731 |
|
| 732 |
curl -z -local.html http://remote.server.com/remote.html |
| 733 |
|
| 734 |
You can specify a "free text" date as condition. Tell curl to only download |
| 735 |
the file if it was updated since January 12, 2012: |
| 736 |
|
| 737 |
curl -z "Jan 12 2012" http://remote.server.com/remote.html |
| 738 |
|
| 739 |
Curl will then accept a wide range of date formats. You always make the date |
| 740 |
check the other way around by prepending it with a dash (`-`). |
| 741 |
|
| 742 |
## DICT |
| 743 |
|
| 744 |
For fun try |
| 745 |
|
| 746 |
curl dict://dict.org/m:curl |
| 747 |
curl dict://dict.org/d:heisenbug:jargon |
| 748 |
curl dict://dict.org/d:daniel:web1913 |
| 749 |
|
| 750 |
Aliases for 'm' are 'match' and 'find', and aliases for 'd' are 'define' and |
| 751 |
'lookup'. For example, |
| 752 |
|
| 753 |
curl dict://dict.org/find:curl |
| 754 |
|
| 755 |
Commands that break the URL description of the RFC (but not the DICT |
| 756 |
protocol) are |
| 757 |
|
| 758 |
curl dict://dict.org/show:db |
| 759 |
curl dict://dict.org/show:strat |
| 760 |
|
| 761 |
Authentication support is still missing |
| 762 |
|
| 763 |
## LDAP |
| 764 |
|
| 765 |
If you have installed the OpenLDAP library, curl can take advantage of it and |
| 766 |
offer `ldap://` support. On Windows, curl will use WinLDAP from Platform SDK |
| 767 |
by default. |
| 768 |
|
| 769 |
Default protocol version used by curl is LDAPv3. LDAPv2 will be used as |
| 770 |
fallback mechanism in case if LDAPv3 will fail to connect. |
| 771 |
|
| 772 |
LDAP is a complex thing and writing an LDAP query is not an easy task. I do |
| 773 |
advise you to dig up the syntax description for that elsewhere. One such place |
| 774 |
might be: [RFC 2255, The LDAP URL |
| 775 |
Format](https://curl.haxx.se/rfc/rfc2255.txt) |
| 776 |
|
| 777 |
To show you an example, this is how I can get all people from my local LDAP |
| 778 |
server that has a certain sub-domain in their email address: |
| 779 |
|
| 780 |
curl -B "ldap://ldap.frontec.se/o=frontec??sub?mail=*sth.frontec.se" |
| 781 |
|
| 782 |
If I want the same info in HTML format, I can get it by not using the `-B` |
| 783 |
(enforce ASCII) flag. |
| 784 |
|
| 785 |
You also can use authentication when accessing LDAP catalog: |
| 786 |
|
| 787 |
curl -u user:passwd "ldap://ldap.frontec.se/o=frontec??sub?mail=*" |
| 788 |
curl "ldap://user:[email protected]/o=frontec??sub?mail=*" |
| 789 |
|
| 790 |
By default, if user and password provided, OpenLDAP/WinLDAP will use basic |
| 791 |
authentication. On Windows you can control this behavior by providing one of |
| 792 |
`--basic`, `--ntlm` or `--digest` option in curl command line |
| 793 |
|
| 794 |
curl --ntlm "ldap://user:[email protected]/o=frontec??sub?mail=*" |
| 795 |
|
| 796 |
On Windows, if no user/password specified, auto-negotiation mechanism will be |
| 797 |
used with current logon credentials (SSPI/SPNEGO). |
| 798 |
|
| 799 |
## Environment Variables |
| 800 |
|
| 801 |
Curl reads and understands the following environment variables: |
| 802 |
|
| 803 |
http_proxy, HTTPS_PROXY, FTP_PROXY |
| 804 |
|
| 805 |
They should be set for protocol-specific proxies. General proxy should be set |
| 806 |
with |
| 807 |
|
| 808 |
ALL_PROXY |
| 809 |
|
| 810 |
A comma-separated list of host names that shouldn't go through any proxy is |
| 811 |
set in (only an asterisk, `*` matches all hosts) |
| 812 |
|
| 813 |
NO_PROXY |
| 814 |
|
| 815 |
If the host name matches one of these strings, or the host is within the |
| 816 |
domain of one of these strings, transactions with that node will not be |
| 817 |
proxied. When a domain is used, it needs to start with a period. A user can |
| 818 |
specify that both www.example.com and foo.example.com should not use a proxy |
| 819 |
by setting `NO_PROXY` to `.example.com`. By including the full name you can |
| 820 |
exclude specific host names, so to make `www.example.com` not use a proxy but |
| 821 |
still have `foo.example.com` do it, set `NO_PROXY` to `www.example.com`. |
| 822 |
|
| 823 |
The usage of the `-x`/`--proxy` flag overrides the environment variables. |
| 824 |
|
| 825 |
## Netrc |
| 826 |
|
| 827 |
Unix introduced the `.netrc` concept a long time ago. It is a way for a user |
| 828 |
to specify name and password for commonly visited FTP sites in a file so that |
| 829 |
you don't have to type them in each time you visit those sites. You realize |
| 830 |
this is a big security risk if someone else gets hold of your passwords, so |
| 831 |
therefore most unix programs won't read this file unless it is only readable |
| 832 |
by yourself (curl doesn't care though). |
| 833 |
|
| 834 |
Curl supports `.netrc` files if told to (using the `-n`/`--netrc` and |
| 835 |
`--netrc-optional` options). This is not restricted to just FTP, so curl can |
| 836 |
use it for all protocols where authentication is used. |
| 837 |
|
| 838 |
A very simple `.netrc` file could look something like: |
| 839 |
|
| 840 |
machine curl.haxx.se login iamdaniel password mysecret |
| 841 |
|
| 842 |
## Custom Output |
| 843 |
|
| 844 |
To better allow script programmers to get to know about the progress of curl, |
| 845 |
the `-w`/`--write-out` option was introduced. Using this, you can specify what |
| 846 |
information from the previous transfer you want to extract. |
| 847 |
|
| 848 |
To display the amount of bytes downloaded together with some text and an |
| 849 |
ending newline: |
| 850 |
|
| 851 |
curl -w 'We downloaded %{size_download} bytes\n' www.download.com |
| 852 |
|
| 853 |
## Kerberos FTP Transfer |
| 854 |
|
| 855 |
Curl supports kerberos4 and kerberos5/GSSAPI for FTP transfers. You need the |
| 856 |
kerberos package installed and used at curl build time for it to be available. |
| 857 |
|
| 858 |
First, get the krb-ticket the normal way, like with the kinit/kauth tool. |
| 859 |
Then use curl in way similar to: |
| 860 |
|
| 861 |
curl --krb private ftp://krb4site.com -u username:fakepwd |
| 862 |
|
| 863 |
There's no use for a password on the `-u` switch, but a blank one will make |
| 864 |
curl ask for one and you already entered the real password to kinit/kauth. |
| 865 |
|
| 866 |
## TELNET |
| 867 |
|
| 868 |
The curl telnet support is basic and very easy to use. Curl passes all data |
| 869 |
passed to it on stdin to the remote server. Connect to a remote telnet server |
| 870 |
using a command line similar to: |
| 871 |
|
| 872 |
curl telnet://remote.server.com |
| 873 |
|
| 874 |
And enter the data to pass to the server on stdin. The result will be sent to |
| 875 |
stdout or to the file you specify with `-o`. |
| 876 |
|
| 877 |
You might want the `-N`/`--no-buffer` option to switch off the buffered output |
| 878 |
for slow connections or similar. |
| 879 |
|
| 880 |
Pass options to the telnet protocol negotiation, by using the `-t` option. To |
| 881 |
tell the server we use a vt100 terminal, try something like: |
| 882 |
|
| 883 |
curl -tTTYPE=vt100 telnet://remote.server.com |
| 884 |
|
| 885 |
Other interesting options for it `-t` include: |
| 886 |
|
| 887 |
- `XDISPLOC=<X display>` Sets the X display location. |
| 888 |
- `NEW_ENV=<var,val>` Sets an environment variable. |
| 889 |
|
| 890 |
NOTE: The telnet protocol does not specify any way to login with a specified |
| 891 |
user and password so curl can't do that automatically. To do that, you need to |
| 892 |
track when the login prompt is received and send the username and password |
| 893 |
accordingly. |
| 894 |
|
| 895 |
## Persistent Connections |
| 896 |
|
| 897 |
Specifying multiple files on a single command line will make curl transfer all |
| 898 |
of them, one after the other in the specified order. |
| 899 |
|
| 900 |
libcurl will attempt to use persistent connections for the transfers so that |
| 901 |
the second transfer to the same host can use the same connection that was |
| 902 |
already initiated and was left open in the previous transfer. This greatly |
| 903 |
decreases connection time for all but the first transfer and it makes a far |
| 904 |
better use of the network. |
| 905 |
|
| 906 |
Note that curl cannot use persistent connections for transfers that are used |
| 907 |
in subsequence curl invokes. Try to stuff as many URLs as possible on the same |
| 908 |
command line if they are using the same host, as that'll make the transfers |
| 909 |
faster. If you use an HTTP proxy for file transfers, practically all transfers |
| 910 |
will be persistent. |
| 911 |
|
| 912 |
## Multiple Transfers With A Single Command Line |
| 913 |
|
| 914 |
As is mentioned above, you can download multiple files with one command line |
| 915 |
by simply adding more URLs. If you want those to get saved to a local file |
| 916 |
instead of just printed to stdout, you need to add one save option for each |
| 917 |
URL you specify. Note that this also goes for the `-O` option (but not |
| 918 |
`--remote-name-all`). |
| 919 |
|
| 920 |
For example: get two files and use `-O` for the first and a custom file |
| 921 |
name for the second: |
| 922 |
|
| 923 |
curl -O http://url.com/file.txt ftp://ftp.com/moo.exe -o moo.jpg |
| 924 |
|
| 925 |
You can also upload multiple files in a similar fashion: |
| 926 |
|
| 927 |
curl -T local1 ftp://ftp.com/moo.exe -T local2 ftp://ftp.com/moo2.txt |
| 928 |
|
| 929 |
## IPv6 |
| 930 |
|
| 931 |
curl will connect to a server with IPv6 when a host lookup returns an IPv6 |
| 932 |
address and fall back to IPv4 if the connection fails. The `--ipv4` and |
| 933 |
`--ipv6` options can specify which address to use when both are |
| 934 |
available. IPv6 addresses can also be specified directly in URLs using the |
| 935 |
syntax: |
| 936 |
|
| 937 |
http://[2001:1890:1112:1::20]/overview.html |
| 938 |
|
| 939 |
When this style is used, the `-g` option must be given to stop curl from |
| 940 |
interpreting the square brackets as special globbing characters. Link local |
| 941 |
and site local addresses including a scope identifier, such as `fe80::1234%1`, |
| 942 |
may also be used, but the scope portion must be numeric or match an existing |
| 943 |
network interface on Linux and the percent character must be URL escaped. The |
| 944 |
previous example in an SFTP URL might look like: |
| 945 |
|
| 946 |
sftp://[fe80::1234%251]/ |
| 947 |
|
| 948 |
IPv6 addresses provided other than in URLs (e.g. to the `--proxy`, |
| 949 |
`--interface` or `--ftp-port` options) should not be URL encoded. |
| 950 |
|
| 951 |
## Metalink |
| 952 |
|
| 953 |
Curl supports Metalink (both version 3 and 4 (RFC 5854) are supported), a way |
| 954 |
to list multiple URIs and hashes for a file. Curl will make use of the mirrors |
| 955 |
listed within for failover if there are errors (such as the file or server not |
| 956 |
being available). It will also verify the hash of the file after the download |
| 957 |
completes. The Metalink file itself is downloaded and processed in memory and |
| 958 |
not stored in the local file system. |
| 959 |
|
| 960 |
Example to use a remote Metalink file: |
| 961 |
|
| 962 |
curl --metalink http://www.example.com/example.metalink |
| 963 |
|
| 964 |
To use a Metalink file in the local file system, use FILE protocol |
| 965 |
(`file://`): |
| 966 |
|
| 967 |
curl --metalink file://example.metalink |
| 968 |
|
| 969 |
Please note that if FILE protocol is disabled, there is no way to use a local |
| 970 |
Metalink file at the time of this writing. Also note that if `--metalink` and |
| 971 |
`--include` are used together, `--include` will be ignored. This is because |
| 972 |
including headers in the response will break Metalink parser and if the |
| 973 |
headers are included in the file described in Metalink file, hash check will |
| 974 |
fail. |
| 975 |
|
| 976 |
## Mailing Lists |
| 977 |
|
| 978 |
For your convenience, we have several open mailing lists to discuss curl, its |
| 979 |
development and things relevant to this. Get all info at |
| 980 |
https://curl.haxx.se/mail/. |
| 981 |
|
| 982 |
Please direct curl questions, feature requests and trouble reports to one of |
| 983 |
these mailing lists instead of mailing any individual. |
| 984 |
|
| 985 |
Available lists include: |
| 986 |
|
| 987 |
### curl-users |
| 988 |
|
| 989 |
Users of the command line tool. How to use it, what doesn't work, new |
| 990 |
features, related tools, questions, news, installations, compilations, |
| 991 |
running, porting etc. |
| 992 |
|
| 993 |
### curl-library |
| 994 |
|
| 995 |
Developers using or developing libcurl. Bugs, extensions, improvements. |
| 996 |
|
| 997 |
### curl-announce |
| 998 |
|
| 999 |
Low-traffic. Only receives announcements of new public versions. At worst, |
| 1000 |
that makes something like one or two mails per month, but usually only one |
| 1001 |
mail every second month. |
| 1002 |
|
| 1003 |
### curl-and-php |
| 1004 |
|
| 1005 |
Using the curl functions in PHP. Everything curl with a PHP angle. Or PHP with |
| 1006 |
a curl angle. |
| 1007 |
|
| 1008 |
### curl-and-python |
| 1009 |
|
| 1010 |
Python hackers using curl with or without the python binding pycurl. |
| 1011 |
|