| 1 |
.. _urllib-howto: |
| 2 |
|
| 3 |
*********************************************************** |
| 4 |
HOWTO Fetch Internet Resources Using The urllib Package |
| 5 |
*********************************************************** |
| 6 |
|
| 7 |
:Author: `Michael Foord <https://agileabstractions.com/>`_ |
| 8 |
|
| 9 |
|
| 10 |
Introduction |
| 11 |
============ |
| 12 |
|
| 13 |
.. sidebar:: Related Articles |
| 14 |
|
| 15 |
You may also find useful the following article on fetching web resources |
| 16 |
with Python: |
| 17 |
|
| 18 |
* `Basic Authentication <https://web.archive.org/web/20201215133350/http://www.voidspace.org.uk/python/articles/authentication.shtml>`__ |
| 19 |
|
| 20 |
A tutorial on *Basic Authentication*, with examples in Python. |
| 21 |
|
| 22 |
**urllib.request** is a Python module for fetching URLs |
| 23 |
(Uniform Resource Locators). It offers a very simple interface, in the form of |
| 24 |
the *urlopen* function. This is capable of fetching URLs using a variety of |
| 25 |
different protocols. It also offers a slightly more complex interface for |
| 26 |
handling common situations - like basic authentication, cookies, proxies and so |
| 27 |
on. These are provided by objects called handlers and openers. |
| 28 |
|
| 29 |
urllib.request supports fetching URLs for many "URL schemes" (identified by the string |
| 30 |
before the ``":"`` in URL - for example ``"ftp"`` is the URL scheme of |
| 31 |
``"ftp://python.org/"``) using their associated network protocols (e.g. FTP, HTTP). |
| 32 |
This tutorial focuses on the most common case, HTTP. |
| 33 |
|
| 34 |
For straightforward situations *urlopen* is very easy to use. But as soon as you |
| 35 |
encounter errors or non-trivial cases when opening HTTP URLs, you will need some |
| 36 |
understanding of the HyperText Transfer Protocol. The most comprehensive and |
| 37 |
authoritative reference to HTTP is :rfc:`2616`. This is a technical document and |
| 38 |
not intended to be easy to read. This HOWTO aims to illustrate using *urllib*, |
| 39 |
with enough detail about HTTP to help you through. It is not intended to replace |
| 40 |
the :mod:`urllib.request` docs, but is supplementary to them. |
| 41 |
|
| 42 |
|
| 43 |
Fetching URLs |
| 44 |
============= |
| 45 |
|
| 46 |
The simplest way to use urllib.request is as follows:: |
| 47 |
|
| 48 |
import urllib.request |
| 49 |
with urllib.request.urlopen('http://python.org/') as response: |
| 50 |
html = response.read() |
| 51 |
|
| 52 |
If you wish to retrieve a resource via URL and store it in a temporary |
| 53 |
location, you can do so via the :func:`shutil.copyfileobj` and |
| 54 |
:func:`tempfile.NamedTemporaryFile` functions:: |
| 55 |
|
| 56 |
import shutil |
| 57 |
import tempfile |
| 58 |
import urllib.request |
| 59 |
|
| 60 |
with urllib.request.urlopen('http://python.org/') as response: |
| 61 |
with tempfile.NamedTemporaryFile(delete=False) as tmp_file: |
| 62 |
shutil.copyfileobj(response, tmp_file) |
| 63 |
|
| 64 |
with open(tmp_file.name) as html: |
| 65 |
pass |
| 66 |
|
| 67 |
Many uses of urllib will be that simple (note that instead of an 'http:' URL we |
| 68 |
could have used a URL starting with 'ftp:', 'file:', etc.). However, it's the |
| 69 |
purpose of this tutorial to explain the more complicated cases, concentrating on |
| 70 |
HTTP. |
| 71 |
|
| 72 |
HTTP is based on requests and responses - the client makes requests and servers |
| 73 |
send responses. urllib.request mirrors this with a ``Request`` object which represents |
| 74 |
the HTTP request you are making. In its simplest form you create a Request |
| 75 |
object that specifies the URL you want to fetch. Calling ``urlopen`` with this |
| 76 |
Request object returns a response object for the URL requested. This response is |
| 77 |
a file-like object, which means you can for example call ``.read()`` on the |
| 78 |
response:: |
| 79 |
|
| 80 |
import urllib.request |
| 81 |
|
| 82 |
req = urllib.request.Request('http://python.org/') |
| 83 |
with urllib.request.urlopen(req) as response: |
| 84 |
the_page = response.read() |
| 85 |
|
| 86 |
Note that urllib.request makes use of the same Request interface to handle all URL |
| 87 |
schemes. For example, you can make an FTP request like so:: |
| 88 |
|
| 89 |
req = urllib.request.Request('ftp://example.com/') |
| 90 |
|
| 91 |
In the case of HTTP, there are two extra things that Request objects allow you |
| 92 |
to do: First, you can pass data to be sent to the server. Second, you can pass |
| 93 |
extra information ("metadata") *about* the data or about the request itself, to |
| 94 |
the server - this information is sent as HTTP "headers". Let's look at each of |
| 95 |
these in turn. |
| 96 |
|
| 97 |
Data |
| 98 |
---- |
| 99 |
|
| 100 |
Sometimes you want to send data to a URL (often the URL will refer to a CGI |
| 101 |
(Common Gateway Interface) script or other web application). With HTTP, |
| 102 |
this is often done using what's known as a **POST** request. This is often what |
| 103 |
your browser does when you submit a HTML form that you filled in on the web. Not |
| 104 |
all POSTs have to come from forms: you can use a POST to transmit arbitrary data |
| 105 |
to your own application. In the common case of HTML forms, the data needs to be |
| 106 |
encoded in a standard way, and then passed to the Request object as the ``data`` |
| 107 |
argument. The encoding is done using a function from the :mod:`urllib.parse` |
| 108 |
library. :: |
| 109 |
|
| 110 |
import urllib.parse |
| 111 |
import urllib.request |
| 112 |
|
| 113 |
url = 'http://www.someserver.com/cgi-bin/register.cgi' |
| 114 |
values = {'name' : 'Michael Foord', |
| 115 |
'location' : 'Northampton', |
| 116 |
'language' : 'Python' } |
| 117 |
|
| 118 |
data = urllib.parse.urlencode(values) |
| 119 |
data = data.encode('ascii') # data should be bytes |
| 120 |
req = urllib.request.Request(url, data) |
| 121 |
with urllib.request.urlopen(req) as response: |
| 122 |
the_page = response.read() |
| 123 |
|
| 124 |
Note that other encodings are sometimes required (e.g. for file upload from HTML |
| 125 |
forms - see `HTML Specification, Form Submission |
| 126 |
<https://www.w3.org/TR/REC-html40/interact/forms.html#h-17.13>`_ for more |
| 127 |
details). |
| 128 |
|
| 129 |
If you do not pass the ``data`` argument, urllib uses a **GET** request. One |
| 130 |
way in which GET and POST requests differ is that POST requests often have |
| 131 |
"side-effects": they change the state of the system in some way (for example by |
| 132 |
placing an order with the website for a hundredweight of tinned spam to be |
| 133 |
delivered to your door). Though the HTTP standard makes it clear that POSTs are |
| 134 |
intended to *always* cause side-effects, and GET requests *never* to cause |
| 135 |
side-effects, nothing prevents a GET request from having side-effects, nor a |
| 136 |
POST requests from having no side-effects. Data can also be passed in an HTTP |
| 137 |
GET request by encoding it in the URL itself. |
| 138 |
|
| 139 |
This is done as follows:: |
| 140 |
|
| 141 |
>>> import urllib.request |
| 142 |
>>> import urllib.parse |
| 143 |
>>> data = {} |
| 144 |
>>> data['name'] = 'Somebody Here' |
| 145 |
>>> data['location'] = 'Northampton' |
| 146 |
>>> data['language'] = 'Python' |
| 147 |
>>> url_values = urllib.parse.urlencode(data) |
| 148 |
>>> print(url_values) # The order may differ from below. #doctest: +SKIP |
| 149 |
name=Somebody+Here&language=Python&location=Northampton |
| 150 |
>>> url = 'http://www.example.com/example.cgi' |
| 151 |
>>> full_url = url + '?' + url_values |
| 152 |
>>> data = urllib.request.urlopen(full_url) |
| 153 |
|
| 154 |
Notice that the full URL is created by adding a ``?`` to the URL, followed by |
| 155 |
the encoded values. |
| 156 |
|
| 157 |
Headers |
| 158 |
------- |
| 159 |
|
| 160 |
We'll discuss here one particular HTTP header, to illustrate how to add headers |
| 161 |
to your HTTP request. |
| 162 |
|
| 163 |
Some websites [#]_ dislike being browsed by programs, or send different versions |
| 164 |
to different browsers [#]_. By default urllib identifies itself as |
| 165 |
``Python-urllib/x.y`` (where ``x`` and ``y`` are the major and minor version |
| 166 |
numbers of the Python release, |
| 167 |
e.g. ``Python-urllib/2.5``), which may confuse the site, or just plain |
| 168 |
not work. The way a browser identifies itself is through the |
| 169 |
``User-Agent`` header [#]_. When you create a Request object you can |
| 170 |
pass a dictionary of headers in. The following example makes the same |
| 171 |
request as above, but identifies itself as a version of Internet |
| 172 |
Explorer [#]_. :: |
| 173 |
|
| 174 |
import urllib.parse |
| 175 |
import urllib.request |
| 176 |
|
| 177 |
url = 'http://www.someserver.com/cgi-bin/register.cgi' |
| 178 |
user_agent = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64)' |
| 179 |
values = {'name': 'Michael Foord', |
| 180 |
'location': 'Northampton', |
| 181 |
'language': 'Python' } |
| 182 |
headers = {'User-Agent': user_agent} |
| 183 |
|
| 184 |
data = urllib.parse.urlencode(values) |
| 185 |
data = data.encode('ascii') |
| 186 |
req = urllib.request.Request(url, data, headers) |
| 187 |
with urllib.request.urlopen(req) as response: |
| 188 |
the_page = response.read() |
| 189 |
|
| 190 |
The response also has two useful methods. See the section on `info and geturl`_ |
| 191 |
which comes after we have a look at what happens when things go wrong. |
| 192 |
|
| 193 |
|
| 194 |
Handling Exceptions |
| 195 |
=================== |
| 196 |
|
| 197 |
*urlopen* raises :exc:`~urllib.error.URLError` when it cannot handle a response (though as |
| 198 |
usual with Python APIs, built-in exceptions such as :exc:`ValueError`, |
| 199 |
:exc:`TypeError` etc. may also be raised). |
| 200 |
|
| 201 |
:exc:`~urllib.error.HTTPError` is the subclass of :exc:`~urllib.error.URLError` raised in the specific case of |
| 202 |
HTTP URLs. |
| 203 |
|
| 204 |
The exception classes are exported from the :mod:`urllib.error` module. |
| 205 |
|
| 206 |
URLError |
| 207 |
-------- |
| 208 |
|
| 209 |
Often, URLError is raised because there is no network connection (no route to |
| 210 |
the specified server), or the specified server doesn't exist. In this case, the |
| 211 |
exception raised will have a 'reason' attribute, which is a tuple containing an |
| 212 |
error code and a text error message. |
| 213 |
|
| 214 |
e.g. :: |
| 215 |
|
| 216 |
>>> req = urllib.request.Request('http://www.pretend_server.org') |
| 217 |
>>> try: urllib.request.urlopen(req) |
| 218 |
... except urllib.error.URLError as e: |
| 219 |
... print(e.reason) #doctest: +SKIP |
| 220 |
... |
| 221 |
(4, 'getaddrinfo failed') |
| 222 |
|
| 223 |
|
| 224 |
HTTPError |
| 225 |
--------- |
| 226 |
|
| 227 |
Every HTTP response from the server contains a numeric "status code". Sometimes |
| 228 |
the status code indicates that the server is unable to fulfil the request. The |
| 229 |
default handlers will handle some of these responses for you (for example, if |
| 230 |
the response is a "redirection" that requests the client fetch the document from |
| 231 |
a different URL, urllib will handle that for you). For those it can't handle, |
| 232 |
urlopen will raise an :exc:`~urllib.error.HTTPError`. Typical errors include '404' (page not |
| 233 |
found), '403' (request forbidden), and '401' (authentication required). |
| 234 |
|
| 235 |
See section 10 of :rfc:`2616` for a reference on all the HTTP error codes. |
| 236 |
|
| 237 |
The :exc:`~urllib.error.HTTPError` instance raised will have an integer 'code' attribute, which |
| 238 |
corresponds to the error sent by the server. |
| 239 |
|
| 240 |
Error Codes |
| 241 |
~~~~~~~~~~~ |
| 242 |
|
| 243 |
Because the default handlers handle redirects (codes in the 300 range), and |
| 244 |
codes in the 100--299 range indicate success, you will usually only see error |
| 245 |
codes in the 400--599 range. |
| 246 |
|
| 247 |
:attr:`http.server.BaseHTTPRequestHandler.responses` is a useful dictionary of |
| 248 |
response codes that shows all the response codes used by :rfc:`2616`. |
| 249 |
An excerpt from the dictionary is shown below :: |
| 250 |
|
| 251 |
responses = { |
| 252 |
... |
| 253 |
<HTTPStatus.OK: 200>: ('OK', 'Request fulfilled, document follows'), |
| 254 |
... |
| 255 |
<HTTPStatus.FORBIDDEN: 403>: ('Forbidden', |
| 256 |
'Request forbidden -- authorization will ' |
| 257 |
'not help'), |
| 258 |
<HTTPStatus.NOT_FOUND: 404>: ('Not Found', |
| 259 |
'Nothing matches the given URI'), |
| 260 |
... |
| 261 |
<HTTPStatus.IM_A_TEAPOT: 418>: ("I'm a Teapot", |
| 262 |
'Server refuses to brew coffee because ' |
| 263 |
'it is a teapot'), |
| 264 |
... |
| 265 |
<HTTPStatus.SERVICE_UNAVAILABLE: 503>: ('Service Unavailable', |
| 266 |
'The server cannot process the ' |
| 267 |
'request due to a high load'), |
| 268 |
... |
| 269 |
} |
| 270 |
|
| 271 |
When an error is raised the server responds by returning an HTTP error code |
| 272 |
*and* an error page. You can use the :exc:`~urllib.error.HTTPError` instance as a response on the |
| 273 |
page returned. This means that as well as the code attribute, it also has read, |
| 274 |
geturl, and info, methods as returned by the ``urllib.response`` module:: |
| 275 |
|
| 276 |
>>> req = urllib.request.Request('http://www.python.org/fish.html') |
| 277 |
>>> try: |
| 278 |
... urllib.request.urlopen(req) |
| 279 |
... except urllib.error.HTTPError as e: |
| 280 |
... print(e.code) |
| 281 |
... print(e.read()) #doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE |
| 282 |
... |
| 283 |
404 |
| 284 |
b'<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" |
| 285 |
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\n\n\n<html |
| 286 |
... |
| 287 |
<title>Page Not Found</title>\n |
| 288 |
... |
| 289 |
|
| 290 |
Wrapping it Up |
| 291 |
-------------- |
| 292 |
|
| 293 |
So if you want to be prepared for :exc:`~urllib.error.HTTPError` *or* :exc:`~urllib.error.URLError` there are two |
| 294 |
basic approaches. I prefer the second approach. |
| 295 |
|
| 296 |
Number 1 |
| 297 |
~~~~~~~~ |
| 298 |
|
| 299 |
:: |
| 300 |
|
| 301 |
|
| 302 |
from urllib.request import Request, urlopen |
| 303 |
from urllib.error import URLError, HTTPError |
| 304 |
req = Request(someurl) |
| 305 |
try: |
| 306 |
response = urlopen(req) |
| 307 |
except HTTPError as e: |
| 308 |
print('The server couldn\'t fulfill the request.') |
| 309 |
print('Error code: ', e.code) |
| 310 |
except URLError as e: |
| 311 |
print('We failed to reach a server.') |
| 312 |
print('Reason: ', e.reason) |
| 313 |
else: |
| 314 |
# everything is fine |
| 315 |
|
| 316 |
|
| 317 |
.. note:: |
| 318 |
|
| 319 |
The ``except HTTPError`` *must* come first, otherwise ``except URLError`` |
| 320 |
will *also* catch an :exc:`~urllib.error.HTTPError`. |
| 321 |
|
| 322 |
Number 2 |
| 323 |
~~~~~~~~ |
| 324 |
|
| 325 |
:: |
| 326 |
|
| 327 |
from urllib.request import Request, urlopen |
| 328 |
from urllib.error import URLError |
| 329 |
req = Request(someurl) |
| 330 |
try: |
| 331 |
response = urlopen(req) |
| 332 |
except URLError as e: |
| 333 |
if hasattr(e, 'reason'): |
| 334 |
print('We failed to reach a server.') |
| 335 |
print('Reason: ', e.reason) |
| 336 |
elif hasattr(e, 'code'): |
| 337 |
print('The server couldn\'t fulfill the request.') |
| 338 |
print('Error code: ', e.code) |
| 339 |
else: |
| 340 |
# everything is fine |
| 341 |
|
| 342 |
|
| 343 |
info and geturl |
| 344 |
=============== |
| 345 |
|
| 346 |
The response returned by urlopen (or the :exc:`~urllib.error.HTTPError` instance) has two |
| 347 |
useful methods :meth:`!info` and :meth:`!geturl` and is defined in the module |
| 348 |
:mod:`urllib.response`. |
| 349 |
|
| 350 |
* **geturl** - this returns the real URL of the page fetched. This is useful |
| 351 |
because ``urlopen`` (or the opener object used) may have followed a |
| 352 |
redirect. The URL of the page fetched may not be the same as the URL requested. |
| 353 |
|
| 354 |
* **info** - this returns a dictionary-like object that describes the page |
| 355 |
fetched, particularly the headers sent by the server. It is currently an |
| 356 |
:class:`http.client.HTTPMessage` instance. |
| 357 |
|
| 358 |
Typical headers include 'Content-length', 'Content-type', and so on. See the |
| 359 |
`Quick Reference to HTTP Headers <https://jkorpela.fi/http.html>`_ |
| 360 |
for a useful listing of HTTP headers with brief explanations of their meaning |
| 361 |
and use. |
| 362 |
|
| 363 |
|
| 364 |
Openers and Handlers |
| 365 |
==================== |
| 366 |
|
| 367 |
When you fetch a URL you use an opener (an instance of the perhaps |
| 368 |
confusingly named :class:`urllib.request.OpenerDirector`). Normally we have been using |
| 369 |
the default opener - via ``urlopen`` - but you can create custom |
| 370 |
openers. Openers use handlers. All the "heavy lifting" is done by the |
| 371 |
handlers. Each handler knows how to open URLs for a particular URL scheme (http, |
| 372 |
ftp, etc.), or how to handle an aspect of URL opening, for example HTTP |
| 373 |
redirections or HTTP cookies. |
| 374 |
|
| 375 |
You will want to create openers if you want to fetch URLs with specific handlers |
| 376 |
installed, for example to get an opener that handles cookies, or to get an |
| 377 |
opener that does not handle redirections. |
| 378 |
|
| 379 |
To create an opener, instantiate an ``OpenerDirector``, and then call |
| 380 |
``.add_handler(some_handler_instance)`` repeatedly. |
| 381 |
|
| 382 |
Alternatively, you can use ``build_opener``, which is a convenience function for |
| 383 |
creating opener objects with a single function call. ``build_opener`` adds |
| 384 |
several handlers by default, but provides a quick way to add more and/or |
| 385 |
override the default handlers. |
| 386 |
|
| 387 |
Other sorts of handlers you might want to can handle proxies, authentication, |
| 388 |
and other common but slightly specialised situations. |
| 389 |
|
| 390 |
``install_opener`` can be used to make an ``opener`` object the (global) default |
| 391 |
opener. This means that calls to ``urlopen`` will use the opener you have |
| 392 |
installed. |
| 393 |
|
| 394 |
Opener objects have an ``open`` method, which can be called directly to fetch |
| 395 |
urls in the same way as the ``urlopen`` function: there's no need to call |
| 396 |
``install_opener``, except as a convenience. |
| 397 |
|
| 398 |
|
| 399 |
Basic Authentication |
| 400 |
==================== |
| 401 |
|
| 402 |
To illustrate creating and installing a handler we will use the |
| 403 |
``HTTPBasicAuthHandler``. For a more detailed discussion of this subject -- |
| 404 |
including an explanation of how Basic Authentication works - see the `Basic |
| 405 |
Authentication Tutorial |
| 406 |
<https://web.archive.org/web/20201215133350/http://www.voidspace.org.uk/python/articles/authentication.shtml>`__. |
| 407 |
|
| 408 |
When authentication is required, the server sends a header (as well as the 401 |
| 409 |
error code) requesting authentication. This specifies the authentication scheme |
| 410 |
and a 'realm'. The header looks like: ``WWW-Authenticate: SCHEME |
| 411 |
realm="REALM"``. |
| 412 |
|
| 413 |
e.g. |
| 414 |
|
| 415 |
.. code-block:: none |
| 416 |
|
| 417 |
WWW-Authenticate: Basic realm="cPanel Users" |
| 418 |
|
| 419 |
|
| 420 |
The client should then retry the request with the appropriate name and password |
| 421 |
for the realm included as a header in the request. This is 'basic |
| 422 |
authentication'. In order to simplify this process we can create an instance of |
| 423 |
``HTTPBasicAuthHandler`` and an opener to use this handler. |
| 424 |
|
| 425 |
The ``HTTPBasicAuthHandler`` uses an object called a password manager to handle |
| 426 |
the mapping of URLs and realms to passwords and usernames. If you know what the |
| 427 |
realm is (from the authentication header sent by the server), then you can use a |
| 428 |
``HTTPPasswordMgr``. Frequently one doesn't care what the realm is. In that |
| 429 |
case, it is convenient to use ``HTTPPasswordMgrWithDefaultRealm``. This allows |
| 430 |
you to specify a default username and password for a URL. This will be supplied |
| 431 |
in the absence of you providing an alternative combination for a specific |
| 432 |
realm. We indicate this by providing ``None`` as the realm argument to the |
| 433 |
``add_password`` method. |
| 434 |
|
| 435 |
The top-level URL is the first URL that requires authentication. URLs "deeper" |
| 436 |
than the URL you pass to .add_password() will also match. :: |
| 437 |
|
| 438 |
# create a password manager |
| 439 |
password_mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm() |
| 440 |
|
| 441 |
# Add the username and password. |
| 442 |
# If we knew the realm, we could use it instead of None. |
| 443 |
top_level_url = "http://example.com/foo/" |
| 444 |
password_mgr.add_password(None, top_level_url, username, password) |
| 445 |
|
| 446 |
handler = urllib.request.HTTPBasicAuthHandler(password_mgr) |
| 447 |
|
| 448 |
# create "opener" (OpenerDirector instance) |
| 449 |
opener = urllib.request.build_opener(handler) |
| 450 |
|
| 451 |
# use the opener to fetch a URL |
| 452 |
opener.open(a_url) |
| 453 |
|
| 454 |
# Install the opener. |
| 455 |
# Now all calls to urllib.request.urlopen use our opener. |
| 456 |
urllib.request.install_opener(opener) |
| 457 |
|
| 458 |
.. note:: |
| 459 |
|
| 460 |
In the above example we only supplied our ``HTTPBasicAuthHandler`` to |
| 461 |
``build_opener``. By default openers have the handlers for normal situations |
| 462 |
-- ``ProxyHandler`` (if a proxy setting such as an :envvar:`!http_proxy` |
| 463 |
environment variable is set), ``UnknownHandler``, ``HTTPHandler``, |
| 464 |
``HTTPDefaultErrorHandler``, ``HTTPRedirectHandler``, ``FTPHandler``, |
| 465 |
``FileHandler``, ``DataHandler``, ``HTTPErrorProcessor``. |
| 466 |
|
| 467 |
``top_level_url`` is in fact *either* a full URL (including the 'http:' scheme |
| 468 |
component and the hostname and optionally the port number) |
| 469 |
e.g. ``"http://example.com/"`` *or* an "authority" (i.e. the hostname, |
| 470 |
optionally including the port number) e.g. ``"example.com"`` or ``"example.com:8080"`` |
| 471 |
(the latter example includes a port number). The authority, if present, must |
| 472 |
NOT contain the "userinfo" component - for example ``"joe:[email protected]"`` is |
| 473 |
not correct. |
| 474 |
|
| 475 |
|
| 476 |
Proxies |
| 477 |
======= |
| 478 |
|
| 479 |
**urllib** will auto-detect your proxy settings and use those. This is through |
| 480 |
the ``ProxyHandler``, which is part of the normal handler chain when a proxy |
| 481 |
setting is detected. Normally that's a good thing, but there are occasions |
| 482 |
when it may not be helpful [#]_. One way to do this is to setup our own |
| 483 |
``ProxyHandler``, with no proxies defined. This is done using similar steps to |
| 484 |
setting up a `Basic Authentication`_ handler: :: |
| 485 |
|
| 486 |
>>> proxy_support = urllib.request.ProxyHandler({}) |
| 487 |
>>> opener = urllib.request.build_opener(proxy_support) |
| 488 |
>>> urllib.request.install_opener(opener) |
| 489 |
|
| 490 |
.. note:: |
| 491 |
|
| 492 |
Currently ``urllib.request`` *does not* support fetching of ``https`` locations |
| 493 |
through a proxy. However, this can be enabled by extending urllib.request as |
| 494 |
shown in the recipe [#]_. |
| 495 |
|
| 496 |
.. note:: |
| 497 |
|
| 498 |
``HTTP_PROXY`` will be ignored if a variable ``REQUEST_METHOD`` is set; see |
| 499 |
the documentation on :func:`~urllib.request.getproxies`. |
| 500 |
|
| 501 |
|
| 502 |
Sockets and Layers |
| 503 |
================== |
| 504 |
|
| 505 |
The Python support for fetching resources from the web is layered. urllib uses |
| 506 |
the :mod:`http.client` library, which in turn uses the socket library. |
| 507 |
|
| 508 |
As of Python 2.3 you can specify how long a socket should wait for a response |
| 509 |
before timing out. This can be useful in applications which have to fetch web |
| 510 |
pages. By default the socket module has *no timeout* and can hang. Currently, |
| 511 |
the socket timeout is not exposed at the http.client or urllib.request levels. |
| 512 |
However, you can set the default timeout globally for all sockets using :: |
| 513 |
|
| 514 |
import socket |
| 515 |
import urllib.request |
| 516 |
|
| 517 |
# timeout in seconds |
| 518 |
timeout = 10 |
| 519 |
socket.setdefaulttimeout(timeout) |
| 520 |
|
| 521 |
# this call to urllib.request.urlopen now uses the default timeout |
| 522 |
# we have set in the socket module |
| 523 |
req = urllib.request.Request('http://www.voidspace.org.uk') |
| 524 |
response = urllib.request.urlopen(req) |
| 525 |
|
| 526 |
|
| 527 |
------- |
| 528 |
|
| 529 |
|
| 530 |
Footnotes |
| 531 |
========= |
| 532 |
|
| 533 |
This document was reviewed and revised by John Lee. |
| 534 |
|
| 535 |
.. [#] Google for example. |
| 536 |
.. [#] Browser sniffing is a very bad practice for website design - building |
| 537 |
sites using web standards is much more sensible. Unfortunately a lot of |
| 538 |
sites still send different versions to different browsers. |
| 539 |
.. [#] The user agent for MSIE 6 is |
| 540 |
*'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)'* |
| 541 |
.. [#] For details of more HTTP request headers, see |
| 542 |
`Quick Reference to HTTP Headers`_. |
| 543 |
.. [#] In my case I have to use a proxy to access the internet at work. If you |
| 544 |
attempt to fetch *localhost* URLs through this proxy it blocks them. IE |
| 545 |
is set to use the proxy, which urllib picks up on. In order to test |
| 546 |
scripts with a localhost server, I have to prevent urllib from using |
| 547 |
the proxy. |
| 548 |
.. [#] urllib opener for SSL proxy (CONNECT method): `ASPN Cookbook Recipe |
| 549 |
<https://code.activestate.com/recipes/456195-urrlib2-opener-for-ssl-proxy-connect-method/>`_. |
| 550 |
|