Debian Codegraph @

Doc/howto/urllib2.rst

raw
1 .. _urllib-howto:
2
3 ***********************************************************
4 HOWTO Fetch Internet Resources Using The urllib Package
5 ***********************************************************
6
7 :Author: `Michael Foord <https://agileabstractions.com/>`_
8
9
10 Introduction
11 ============
12
13 .. sidebar:: Related Articles
14
15 You may also find useful the following article on fetching web resources
16 with Python:
17
18 * `Basic Authentication <https://web.archive.org/web/20201215133350/http://www.voidspace.org.uk/python/articles/authentication.shtml>`__
19
20 A tutorial on *Basic Authentication*, with examples in Python.
21
22 **urllib.request** is a Python module for fetching URLs
23 (Uniform Resource Locators). It offers a very simple interface, in the form of
24 the *urlopen* function. This is capable of fetching URLs using a variety of
25 different protocols. It also offers a slightly more complex interface for
26 handling common situations - like basic authentication, cookies, proxies and so
27 on. These are provided by objects called handlers and openers.
28
29 urllib.request supports fetching URLs for many "URL schemes" (identified by the string
30 before the ``":"`` in URL - for example ``"ftp"`` is the URL scheme of
31 ``"ftp://python.org/"``) using their associated network protocols (e.g. FTP, HTTP).
32 This tutorial focuses on the most common case, HTTP.
33
34 For straightforward situations *urlopen* is very easy to use. But as soon as you
35 encounter errors or non-trivial cases when opening HTTP URLs, you will need some
36 understanding of the HyperText Transfer Protocol. The most comprehensive and
37 authoritative reference to HTTP is :rfc:`2616`. This is a technical document and
38 not intended to be easy to read. This HOWTO aims to illustrate using *urllib*,
39 with enough detail about HTTP to help you through. It is not intended to replace
40 the :mod:`urllib.request` docs, but is supplementary to them.
41
42
43 Fetching URLs
44 =============
45
46 The simplest way to use urllib.request is as follows::
47
48 import urllib.request
49 with urllib.request.urlopen('http://python.org/') as response:
50 html = response.read()
51
52 If you wish to retrieve a resource via URL and store it in a temporary
53 location, you can do so via the :func:`shutil.copyfileobj` and
54 :func:`tempfile.NamedTemporaryFile` functions::
55
56 import shutil
57 import tempfile
58 import urllib.request
59
60 with urllib.request.urlopen('http://python.org/') as response:
61 with tempfile.NamedTemporaryFile(delete=False) as tmp_file:
62 shutil.copyfileobj(response, tmp_file)
63
64 with open(tmp_file.name) as html:
65 pass
66
67 Many uses of urllib will be that simple (note that instead of an 'http:' URL we
68 could have used a URL starting with 'ftp:', 'file:', etc.). However, it's the
69 purpose of this tutorial to explain the more complicated cases, concentrating on
70 HTTP.
71
72 HTTP is based on requests and responses - the client makes requests and servers
73 send responses. urllib.request mirrors this with a ``Request`` object which represents
74 the HTTP request you are making. In its simplest form you create a Request
75 object that specifies the URL you want to fetch. Calling ``urlopen`` with this
76 Request object returns a response object for the URL requested. This response is
77 a file-like object, which means you can for example call ``.read()`` on the
78 response::
79
80 import urllib.request
81
82 req = urllib.request.Request('http://python.org/')
83 with urllib.request.urlopen(req) as response:
84 the_page = response.read()
85
86 Note that urllib.request makes use of the same Request interface to handle all URL
87 schemes. For example, you can make an FTP request like so::
88
89 req = urllib.request.Request('ftp://example.com/')
90
91 In the case of HTTP, there are two extra things that Request objects allow you
92 to do: First, you can pass data to be sent to the server. Second, you can pass
93 extra information ("metadata") *about* the data or about the request itself, to
94 the server - this information is sent as HTTP "headers". Let's look at each of
95 these in turn.
96
97 Data
98 ----
99
100 Sometimes you want to send data to a URL (often the URL will refer to a CGI
101 (Common Gateway Interface) script or other web application). With HTTP,
102 this is often done using what's known as a **POST** request. This is often what
103 your browser does when you submit a HTML form that you filled in on the web. Not
104 all POSTs have to come from forms: you can use a POST to transmit arbitrary data
105 to your own application. In the common case of HTML forms, the data needs to be
106 encoded in a standard way, and then passed to the Request object as the ``data``
107 argument. The encoding is done using a function from the :mod:`urllib.parse`
108 library. ::
109
110 import urllib.parse
111 import urllib.request
112
113 url = 'http://www.someserver.com/cgi-bin/register.cgi'
114 values = {'name' : 'Michael Foord',
115 'location' : 'Northampton',
116 'language' : 'Python' }
117
118 data = urllib.parse.urlencode(values)
119 data = data.encode('ascii') # data should be bytes
120 req = urllib.request.Request(url, data)
121 with urllib.request.urlopen(req) as response:
122 the_page = response.read()
123
124 Note that other encodings are sometimes required (e.g. for file upload from HTML
125 forms - see `HTML Specification, Form Submission
126 <https://www.w3.org/TR/REC-html40/interact/forms.html#h-17.13>`_ for more
127 details).
128
129 If you do not pass the ``data`` argument, urllib uses a **GET** request. One
130 way in which GET and POST requests differ is that POST requests often have
131 "side-effects": they change the state of the system in some way (for example by
132 placing an order with the website for a hundredweight of tinned spam to be
133 delivered to your door). Though the HTTP standard makes it clear that POSTs are
134 intended to *always* cause side-effects, and GET requests *never* to cause
135 side-effects, nothing prevents a GET request from having side-effects, nor a
136 POST requests from having no side-effects. Data can also be passed in an HTTP
137 GET request by encoding it in the URL itself.
138
139 This is done as follows::
140
141 >>> import urllib.request
142 >>> import urllib.parse
143 >>> data = {}
144 >>> data['name'] = 'Somebody Here'
145 >>> data['location'] = 'Northampton'
146 >>> data['language'] = 'Python'
147 >>> url_values = urllib.parse.urlencode(data)
148 >>> print(url_values) # The order may differ from below. #doctest: +SKIP
149 name=Somebody+Here&language=Python&location=Northampton
150 >>> url = 'http://www.example.com/example.cgi'
151 >>> full_url = url + '?' + url_values
152 >>> data = urllib.request.urlopen(full_url)
153
154 Notice that the full URL is created by adding a ``?`` to the URL, followed by
155 the encoded values.
156
157 Headers
158 -------
159
160 We'll discuss here one particular HTTP header, to illustrate how to add headers
161 to your HTTP request.
162
163 Some websites [#]_ dislike being browsed by programs, or send different versions
164 to different browsers [#]_. By default urllib identifies itself as
165 ``Python-urllib/x.y`` (where ``x`` and ``y`` are the major and minor version
166 numbers of the Python release,
167 e.g. ``Python-urllib/2.5``), which may confuse the site, or just plain
168 not work. The way a browser identifies itself is through the
169 ``User-Agent`` header [#]_. When you create a Request object you can
170 pass a dictionary of headers in. The following example makes the same
171 request as above, but identifies itself as a version of Internet
172 Explorer [#]_. ::
173
174 import urllib.parse
175 import urllib.request
176
177 url = 'http://www.someserver.com/cgi-bin/register.cgi'
178 user_agent = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64)'
179 values = {'name': 'Michael Foord',
180 'location': 'Northampton',
181 'language': 'Python' }
182 headers = {'User-Agent': user_agent}
183
184 data = urllib.parse.urlencode(values)
185 data = data.encode('ascii')
186 req = urllib.request.Request(url, data, headers)
187 with urllib.request.urlopen(req) as response:
188 the_page = response.read()
189
190 The response also has two useful methods. See the section on `info and geturl`_
191 which comes after we have a look at what happens when things go wrong.
192
193
194 Handling Exceptions
195 ===================
196
197 *urlopen* raises :exc:`~urllib.error.URLError` when it cannot handle a response (though as
198 usual with Python APIs, built-in exceptions such as :exc:`ValueError`,
199 :exc:`TypeError` etc. may also be raised).
200
201 :exc:`~urllib.error.HTTPError` is the subclass of :exc:`~urllib.error.URLError` raised in the specific case of
202 HTTP URLs.
203
204 The exception classes are exported from the :mod:`urllib.error` module.
205
206 URLError
207 --------
208
209 Often, URLError is raised because there is no network connection (no route to
210 the specified server), or the specified server doesn't exist. In this case, the
211 exception raised will have a 'reason' attribute, which is a tuple containing an
212 error code and a text error message.
213
214 e.g. ::
215
216 >>> req = urllib.request.Request('http://www.pretend_server.org')
217 >>> try: urllib.request.urlopen(req)
218 ... except urllib.error.URLError as e:
219 ... print(e.reason) #doctest: +SKIP
220 ...
221 (4, 'getaddrinfo failed')
222
223
224 HTTPError
225 ---------
226
227 Every HTTP response from the server contains a numeric "status code". Sometimes
228 the status code indicates that the server is unable to fulfil the request. The
229 default handlers will handle some of these responses for you (for example, if
230 the response is a "redirection" that requests the client fetch the document from
231 a different URL, urllib will handle that for you). For those it can't handle,
232 urlopen will raise an :exc:`~urllib.error.HTTPError`. Typical errors include '404' (page not
233 found), '403' (request forbidden), and '401' (authentication required).
234
235 See section 10 of :rfc:`2616` for a reference on all the HTTP error codes.
236
237 The :exc:`~urllib.error.HTTPError` instance raised will have an integer 'code' attribute, which
238 corresponds to the error sent by the server.
239
240 Error Codes
241 ~~~~~~~~~~~
242
243 Because the default handlers handle redirects (codes in the 300 range), and
244 codes in the 100--299 range indicate success, you will usually only see error
245 codes in the 400--599 range.
246
247 :attr:`http.server.BaseHTTPRequestHandler.responses` is a useful dictionary of
248 response codes that shows all the response codes used by :rfc:`2616`.
249 An excerpt from the dictionary is shown below ::
250
251 responses = {
252 ...
253 <HTTPStatus.OK: 200>: ('OK', 'Request fulfilled, document follows'),
254 ...
255 <HTTPStatus.FORBIDDEN: 403>: ('Forbidden',
256 'Request forbidden -- authorization will '
257 'not help'),
258 <HTTPStatus.NOT_FOUND: 404>: ('Not Found',
259 'Nothing matches the given URI'),
260 ...
261 <HTTPStatus.IM_A_TEAPOT: 418>: ("I'm a Teapot",
262 'Server refuses to brew coffee because '
263 'it is a teapot'),
264 ...
265 <HTTPStatus.SERVICE_UNAVAILABLE: 503>: ('Service Unavailable',
266 'The server cannot process the '
267 'request due to a high load'),
268 ...
269 }
270
271 When an error is raised the server responds by returning an HTTP error code
272 *and* an error page. You can use the :exc:`~urllib.error.HTTPError` instance as a response on the
273 page returned. This means that as well as the code attribute, it also has read,
274 geturl, and info, methods as returned by the ``urllib.response`` module::
275
276 >>> req = urllib.request.Request('http://www.python.org/fish.html')
277 >>> try:
278 ... urllib.request.urlopen(req)
279 ... except urllib.error.HTTPError as e:
280 ... print(e.code)
281 ... print(e.read()) #doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
282 ...
283 404
284 b'<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
285 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\n\n\n<html
286 ...
287 <title>Page Not Found</title>\n
288 ...
289
290 Wrapping it Up
291 --------------
292
293 So if you want to be prepared for :exc:`~urllib.error.HTTPError` *or* :exc:`~urllib.error.URLError` there are two
294 basic approaches. I prefer the second approach.
295
296 Number 1
297 ~~~~~~~~
298
299 ::
300
301
302 from urllib.request import Request, urlopen
303 from urllib.error import URLError, HTTPError
304 req = Request(someurl)
305 try:
306 response = urlopen(req)
307 except HTTPError as e:
308 print('The server couldn\'t fulfill the request.')
309 print('Error code: ', e.code)
310 except URLError as e:
311 print('We failed to reach a server.')
312 print('Reason: ', e.reason)
313 else:
314 # everything is fine
315
316
317 .. note::
318
319 The ``except HTTPError`` *must* come first, otherwise ``except URLError``
320 will *also* catch an :exc:`~urllib.error.HTTPError`.
321
322 Number 2
323 ~~~~~~~~
324
325 ::
326
327 from urllib.request import Request, urlopen
328 from urllib.error import URLError
329 req = Request(someurl)
330 try:
331 response = urlopen(req)
332 except URLError as e:
333 if hasattr(e, 'reason'):
334 print('We failed to reach a server.')
335 print('Reason: ', e.reason)
336 elif hasattr(e, 'code'):
337 print('The server couldn\'t fulfill the request.')
338 print('Error code: ', e.code)
339 else:
340 # everything is fine
341
342
343 info and geturl
344 ===============
345
346 The response returned by urlopen (or the :exc:`~urllib.error.HTTPError` instance) has two
347 useful methods :meth:`!info` and :meth:`!geturl` and is defined in the module
348 :mod:`urllib.response`.
349
350 * **geturl** - this returns the real URL of the page fetched. This is useful
351 because ``urlopen`` (or the opener object used) may have followed a
352 redirect. The URL of the page fetched may not be the same as the URL requested.
353
354 * **info** - this returns a dictionary-like object that describes the page
355 fetched, particularly the headers sent by the server. It is currently an
356 :class:`http.client.HTTPMessage` instance.
357
358 Typical headers include 'Content-length', 'Content-type', and so on. See the
359 `Quick Reference to HTTP Headers <https://jkorpela.fi/http.html>`_
360 for a useful listing of HTTP headers with brief explanations of their meaning
361 and use.
362
363
364 Openers and Handlers
365 ====================
366
367 When you fetch a URL you use an opener (an instance of the perhaps
368 confusingly named :class:`urllib.request.OpenerDirector`). Normally we have been using
369 the default opener - via ``urlopen`` - but you can create custom
370 openers. Openers use handlers. All the "heavy lifting" is done by the
371 handlers. Each handler knows how to open URLs for a particular URL scheme (http,
372 ftp, etc.), or how to handle an aspect of URL opening, for example HTTP
373 redirections or HTTP cookies.
374
375 You will want to create openers if you want to fetch URLs with specific handlers
376 installed, for example to get an opener that handles cookies, or to get an
377 opener that does not handle redirections.
378
379 To create an opener, instantiate an ``OpenerDirector``, and then call
380 ``.add_handler(some_handler_instance)`` repeatedly.
381
382 Alternatively, you can use ``build_opener``, which is a convenience function for
383 creating opener objects with a single function call. ``build_opener`` adds
384 several handlers by default, but provides a quick way to add more and/or
385 override the default handlers.
386
387 Other sorts of handlers you might want to can handle proxies, authentication,
388 and other common but slightly specialised situations.
389
390 ``install_opener`` can be used to make an ``opener`` object the (global) default
391 opener. This means that calls to ``urlopen`` will use the opener you have
392 installed.
393
394 Opener objects have an ``open`` method, which can be called directly to fetch
395 urls in the same way as the ``urlopen`` function: there's no need to call
396 ``install_opener``, except as a convenience.
397
398
399 Basic Authentication
400 ====================
401
402 To illustrate creating and installing a handler we will use the
403 ``HTTPBasicAuthHandler``. For a more detailed discussion of this subject --
404 including an explanation of how Basic Authentication works - see the `Basic
405 Authentication Tutorial
406 <https://web.archive.org/web/20201215133350/http://www.voidspace.org.uk/python/articles/authentication.shtml>`__.
407
408 When authentication is required, the server sends a header (as well as the 401
409 error code) requesting authentication. This specifies the authentication scheme
410 and a 'realm'. The header looks like: ``WWW-Authenticate: SCHEME
411 realm="REALM"``.
412
413 e.g.
414
415 .. code-block:: none
416
417 WWW-Authenticate: Basic realm="cPanel Users"
418
419
420 The client should then retry the request with the appropriate name and password
421 for the realm included as a header in the request. This is 'basic
422 authentication'. In order to simplify this process we can create an instance of
423 ``HTTPBasicAuthHandler`` and an opener to use this handler.
424
425 The ``HTTPBasicAuthHandler`` uses an object called a password manager to handle
426 the mapping of URLs and realms to passwords and usernames. If you know what the
427 realm is (from the authentication header sent by the server), then you can use a
428 ``HTTPPasswordMgr``. Frequently one doesn't care what the realm is. In that
429 case, it is convenient to use ``HTTPPasswordMgrWithDefaultRealm``. This allows
430 you to specify a default username and password for a URL. This will be supplied
431 in the absence of you providing an alternative combination for a specific
432 realm. We indicate this by providing ``None`` as the realm argument to the
433 ``add_password`` method.
434
435 The top-level URL is the first URL that requires authentication. URLs "deeper"
436 than the URL you pass to .add_password() will also match. ::
437
438 # create a password manager
439 password_mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()
440
441 # Add the username and password.
442 # If we knew the realm, we could use it instead of None.
443 top_level_url = "http://example.com/foo/"
444 password_mgr.add_password(None, top_level_url, username, password)
445
446 handler = urllib.request.HTTPBasicAuthHandler(password_mgr)
447
448 # create "opener" (OpenerDirector instance)
449 opener = urllib.request.build_opener(handler)
450
451 # use the opener to fetch a URL
452 opener.open(a_url)
453
454 # Install the opener.
455 # Now all calls to urllib.request.urlopen use our opener.
456 urllib.request.install_opener(opener)
457
458 .. note::
459
460 In the above example we only supplied our ``HTTPBasicAuthHandler`` to
461 ``build_opener``. By default openers have the handlers for normal situations
462 -- ``ProxyHandler`` (if a proxy setting such as an :envvar:`!http_proxy`
463 environment variable is set), ``UnknownHandler``, ``HTTPHandler``,
464 ``HTTPDefaultErrorHandler``, ``HTTPRedirectHandler``, ``FTPHandler``,
465 ``FileHandler``, ``DataHandler``, ``HTTPErrorProcessor``.
466
467 ``top_level_url`` is in fact *either* a full URL (including the 'http:' scheme
468 component and the hostname and optionally the port number)
469 e.g. ``"http://example.com/"`` *or* an "authority" (i.e. the hostname,
470 optionally including the port number) e.g. ``"example.com"`` or ``"example.com:8080"``
471 (the latter example includes a port number). The authority, if present, must
472 NOT contain the "userinfo" component - for example ``"joe:[email protected]"`` is
473 not correct.
474
475
476 Proxies
477 =======
478
479 **urllib** will auto-detect your proxy settings and use those. This is through
480 the ``ProxyHandler``, which is part of the normal handler chain when a proxy
481 setting is detected. Normally that's a good thing, but there are occasions
482 when it may not be helpful [#]_. One way to do this is to setup our own
483 ``ProxyHandler``, with no proxies defined. This is done using similar steps to
484 setting up a `Basic Authentication`_ handler: ::
485
486 >>> proxy_support = urllib.request.ProxyHandler({})
487 >>> opener = urllib.request.build_opener(proxy_support)
488 >>> urllib.request.install_opener(opener)
489
490 .. note::
491
492 Currently ``urllib.request`` *does not* support fetching of ``https`` locations
493 through a proxy. However, this can be enabled by extending urllib.request as
494 shown in the recipe [#]_.
495
496 .. note::
497
498 ``HTTP_PROXY`` will be ignored if a variable ``REQUEST_METHOD`` is set; see
499 the documentation on :func:`~urllib.request.getproxies`.
500
501
502 Sockets and Layers
503 ==================
504
505 The Python support for fetching resources from the web is layered. urllib uses
506 the :mod:`http.client` library, which in turn uses the socket library.
507
508 As of Python 2.3 you can specify how long a socket should wait for a response
509 before timing out. This can be useful in applications which have to fetch web
510 pages. By default the socket module has *no timeout* and can hang. Currently,
511 the socket timeout is not exposed at the http.client or urllib.request levels.
512 However, you can set the default timeout globally for all sockets using ::
513
514 import socket
515 import urllib.request
516
517 # timeout in seconds
518 timeout = 10
519 socket.setdefaulttimeout(timeout)
520
521 # this call to urllib.request.urlopen now uses the default timeout
522 # we have set in the socket module
523 req = urllib.request.Request('http://www.voidspace.org.uk')
524 response = urllib.request.urlopen(req)
525
526
527 -------
528
529
530 Footnotes
531 =========
532
533 This document was reviewed and revised by John Lee.
534
535 .. [#] Google for example.
536 .. [#] Browser sniffing is a very bad practice for website design - building
537 sites using web standards is much more sensible. Unfortunately a lot of
538 sites still send different versions to different browsers.
539 .. [#] The user agent for MSIE 6 is
540 *'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)'*
541 .. [#] For details of more HTTP request headers, see
542 `Quick Reference to HTTP Headers`_.
543 .. [#] In my case I have to use a proxy to access the internet at work. If you
544 attempt to fetch *localhost* URLs through this proxy it blocks them. IE
545 is set to use the proxy, which urllib picks up on. In order to test
546 scripts with a localhost server, I have to prevent urllib from using
547 the proxy.
548 .. [#] urllib opener for SSL proxy (CONNECT method): `ASPN Cookbook Recipe
549 <https://code.activestate.com/recipes/456195-urrlib2-opener-for-ssl-proxy-connect-method/>`_.
550