Debian Codegraph @

Doc/library/urllib.request.rst

raw
1 :mod:`!urllib.request` --- Extensible library for opening URLs
2 ==============================================================
3
4 .. module:: urllib.request
5 :synopsis: Extensible library for opening URLs.
6
7 .. moduleauthor:: Jeremy Hylton <[email protected]>
8 .. sectionauthor:: Moshe Zadka <[email protected]>
9 .. sectionauthor:: Senthil Kumaran <[email protected]>
10
11 **Source code:** :source:`Lib/urllib/request.py`
12
13 --------------
14
15 The :mod:`!urllib.request` module defines functions and classes which help in
16 opening URLs (mostly HTTP) in a complex world --- basic and digest
17 authentication, redirections, cookies and more.
18
19 .. seealso::
20
21 The `Requests package <https://requests.readthedocs.io/en/master/>`_
22 is recommended for a higher-level HTTP client interface.
23
24 .. warning::
25
26 On macOS it is unsafe to use this module in programs using
27 :func:`os.fork` because the :func:`getproxies` implementation for
28 macOS uses a higher-level system API. Set the environment variable
29 ``no_proxy`` to ``*`` to avoid this problem
30 (e.g. ``os.environ["no_proxy"] = "*"``).
31
32 .. include:: ../includes/wasm-notavail.rst
33
34 The :mod:`!urllib.request` module defines the following functions:
35
36
37 .. function:: urlopen(url, data=None[, timeout], *, context=None)
38
39 Open *url*, which can be either a string containing a valid, properly
40 encoded URL, or a :class:`Request` object.
41
42 *data* must be an object specifying additional data to be sent to the
43 server, or ``None`` if no such data is needed. See :class:`Request`
44 for details.
45
46 urllib.request module uses HTTP/1.1 and includes ``Connection:close`` header
47 in its HTTP requests.
48
49 The optional *timeout* parameter specifies a timeout in seconds for
50 blocking operations like the connection attempt (if not specified,
51 the global default timeout setting will be used). This actually
52 only works for HTTP, HTTPS and FTP connections.
53
54 If *context* is specified, it must be a :class:`ssl.SSLContext` instance
55 describing the various SSL options. See :class:`~http.client.HTTPSConnection`
56 for more details.
57
58 This function always returns an object which can work as a
59 :term:`context manager` and has the properties *url*, *headers*, and *status*.
60 See :class:`urllib.response.addinfourl` for more detail on these properties.
61
62 For HTTP and HTTPS URLs, this function returns a
63 :class:`http.client.HTTPResponse` object slightly modified. In addition
64 to the three new methods above, the msg attribute contains the
65 same information as the :attr:`~http.client.HTTPResponse.reason`
66 attribute --- the reason phrase returned by server --- instead of
67 the response headers as it is specified in the documentation for
68 :class:`~http.client.HTTPResponse`.
69
70 For FTP, file, and data URLs, this function
71 returns a :class:`urllib.response.addinfourl` object.
72
73 Raises :exc:`~urllib.error.URLError` on protocol errors.
74
75 Note that ``None`` may be returned if no handler handles the request (though
76 the default installed global :class:`OpenerDirector` uses
77 :class:`UnknownHandler` to ensure this never happens).
78
79 In addition, if proxy settings are detected (for example, when a ``*_proxy``
80 environment variable like :envvar:`!http_proxy` is set),
81 :class:`ProxyHandler` is default installed and makes sure the requests are
82 handled through the proxy.
83
84 The legacy ``urllib.urlopen`` function from Python 2.6 and earlier has been
85 discontinued; :func:`urllib.request.urlopen` corresponds to the old
86 ``urllib2.urlopen``. Proxy handling, which was done by passing a dictionary
87 parameter to ``urllib.urlopen``, can be obtained by using
88 :class:`ProxyHandler` objects.
89
90 .. audit-event:: urllib.Request fullurl,data,headers,method urllib.request.urlopen
91
92 The default opener raises an :ref:`auditing event <auditing>`
93 ``urllib.Request`` with arguments ``fullurl``, ``data``, ``headers``,
94 ``method`` taken from the request object.
95
96 .. versionchanged:: 3.2
97 *cafile* and *capath* were added.
98
99 HTTPS virtual hosts are now supported if possible (that is, if
100 :const:`ssl.HAS_SNI` is true).
101
102 *data* can be an iterable object.
103
104 .. versionchanged:: 3.3
105 *cadefault* was added.
106
107 .. versionchanged:: 3.4.3
108 *context* was added.
109
110 .. versionchanged:: 3.10
111 HTTPS connection now send an ALPN extension with protocol indicator
112 ``http/1.1`` when no *context* is given. Custom *context* should set
113 ALPN protocols with :meth:`~ssl.SSLContext.set_alpn_protocols`.
114
115 .. versionchanged:: 3.13
116 Remove *cafile*, *capath* and *cadefault* parameters: use the *context*
117 parameter instead.
118
119
120 .. function:: install_opener(opener)
121
122 Install an :class:`OpenerDirector` instance as the default global opener.
123 Installing an opener is only necessary if you want urlopen to use that
124 opener; otherwise, simply call :meth:`OpenerDirector.open` instead of
125 :func:`~urllib.request.urlopen`. The code does not check for a real
126 :class:`OpenerDirector`, and any class with the appropriate interface will
127 work.
128
129
130 .. function:: build_opener([handler, ...])
131
132 Return an :class:`OpenerDirector` instance, which chains the handlers in the
133 order given. *handler*\s can be either instances of :class:`BaseHandler`, or
134 subclasses of :class:`BaseHandler` (in which case it must be possible to call
135 the constructor without any parameters). Instances of the following classes
136 will be in front of the *handler*\s, unless the *handler*\s contain them,
137 instances of them or subclasses of them: :class:`ProxyHandler` (if proxy
138 settings are detected), :class:`UnknownHandler`, :class:`HTTPHandler`,
139 :class:`HTTPDefaultErrorHandler`, :class:`HTTPRedirectHandler`,
140 :class:`FTPHandler`, :class:`FileHandler`, :class:`HTTPErrorProcessor`.
141
142 If the Python installation has SSL support (i.e., if the :mod:`ssl` module
143 can be imported), :class:`HTTPSHandler` will also be added.
144
145 A :class:`BaseHandler` subclass may also change its :attr:`handler_order`
146 attribute to modify its position in the handlers list.
147
148
149 .. function:: pathname2url(path, *, add_scheme=False)
150
151 Convert the given local path to a ``file:`` URL. This function uses
152 :func:`~urllib.parse.quote` function to encode the path.
153
154 If *add_scheme* is false (the default), the return value omits the
155 ``file:`` scheme prefix. Set *add_scheme* to true to return a complete URL.
156
157 This example shows the function being used on Windows::
158
159 >>> from urllib.request import pathname2url
160 >>> path = 'C:\\Program Files'
161 >>> pathname2url(path, add_scheme=True)
162 'file:///C:/Program%20Files'
163
164 .. versionchanged:: 3.14
165 Windows drive letters are no longer converted to uppercase, and ``:``
166 characters not following a drive letter no longer cause an
167 :exc:`OSError` exception to be raised on Windows.
168
169 .. versionchanged:: 3.14
170 Paths beginning with a slash are converted to URLs with authority
171 sections. For example, the path ``/etc/hosts`` is converted to
172 the URL ``///etc/hosts``.
173
174 .. versionchanged:: 3.14
175 The *add_scheme* parameter was added.
176
177
178 .. function:: url2pathname(url, *, require_scheme=False, resolve_host=False)
179
180 Convert the given ``file:`` URL to a local path. This function uses
181 :func:`~urllib.parse.unquote` to decode the URL.
182
183 If *require_scheme* is false (the default), the given value should omit a
184 ``file:`` scheme prefix. If *require_scheme* is set to true, the given
185 value should include the prefix; a :exc:`~urllib.error.URLError` is raised
186 if it doesn't.
187
188 The URL authority is discarded if it is empty, ``localhost``, or the local
189 hostname. Otherwise, if *resolve_host* is set to true, the authority is
190 resolved using :func:`socket.gethostbyname` and discarded if it matches a
191 local IP address (as per :rfc:`RFC 8089 §3 <8089#section-3>`). If the
192 authority is still unhandled, then on Windows a UNC path is returned, and
193 on other platforms a :exc:`~urllib.error.URLError` is raised.
194
195 This example shows the function being used on Windows::
196
197 >>> from urllib.request import url2pathname
198 >>> url = 'file:///C:/Program%20Files'
199 >>> url2pathname(url, require_scheme=True)
200 'C:\\Program Files'
201
202 .. versionchanged:: 3.14
203 Windows drive letters are no longer converted to uppercase, and ``:``
204 characters not following a drive letter no longer cause an
205 :exc:`OSError` exception to be raised on Windows.
206
207 .. versionchanged:: 3.14
208 The URL authority is discarded if it matches the local hostname.
209 Otherwise, if the authority isn't empty or ``localhost``, then on
210 Windows a UNC path is returned (as before), and on other platforms a
211 :exc:`~urllib.error.URLError` is raised.
212
213 .. versionchanged:: 3.14
214 The URL query and fragment components are discarded if present.
215
216 .. versionchanged:: 3.14
217 The *require_scheme* and *resolve_host* parameters were added.
218
219
220 .. function:: getproxies()
221
222 This helper function returns a dictionary of scheme to proxy server URL
223 mappings. It scans the environment for variables named ``<scheme>_proxy``,
224 in a case insensitive approach, for all operating systems first, and when it
225 cannot find it, looks for proxy information from System
226 Configuration for macOS and Windows Systems Registry for Windows.
227 If both lowercase and uppercase environment variables exist (and disagree),
228 lowercase is preferred.
229
230 .. note::
231
232 If the environment variable ``REQUEST_METHOD`` is set, which usually
233 indicates your script is running in a CGI environment, the environment
234 variable ``HTTP_PROXY`` (uppercase ``_PROXY``) will be ignored. This is
235 because that variable can be injected by a client using the "Proxy:" HTTP
236 header. If you need to use an HTTP proxy in a CGI environment, either use
237 ``ProxyHandler`` explicitly, or make sure the variable name is in
238 lowercase (or at least the ``_proxy`` suffix).
239
240
241 The following classes are provided:
242
243 .. class:: Request(url, data=None, headers={}, origin_req_host=None, unverifiable=False, method=None)
244
245 This class is an abstraction of a URL request.
246
247 *url* should be a string containing a valid, properly encoded URL.
248
249 *data* must be an object specifying additional data to send to the
250 server, or ``None`` if no such data is needed. Currently HTTP
251 requests are the only ones that use *data*. The supported object
252 types include bytes, file-like objects, and iterables of bytes-like objects.
253 If no ``Content-Length`` nor ``Transfer-Encoding`` header field
254 has been provided, :class:`HTTPHandler` will set these headers according
255 to the type of *data*. ``Content-Length`` will be used to send
256 bytes objects, while ``Transfer-Encoding: chunked`` as specified in
257 :rfc:`7230`, Section 3.3.1 will be used to send files and other iterables.
258
259 For an HTTP POST request method, *data* should be a buffer in the
260 standard :mimetype:`application/x-www-form-urlencoded` format. The
261 :func:`urllib.parse.urlencode` function takes a mapping or sequence
262 of 2-tuples and returns an ASCII string in this format. It should
263 be encoded to bytes before being used as the *data* parameter.
264
265 *headers* should be a dictionary, and will be treated as if
266 :meth:`add_header` was called with each key and value as arguments.
267 This is often used to "spoof" the ``User-Agent`` header value, which is
268 used by a browser to identify itself -- some HTTP servers only
269 allow requests coming from common browsers as opposed to scripts.
270 For example, Mozilla Firefox may identify itself as ``"Mozilla/5.0
271 (X11; U; Linux i686) Gecko/20071127 Firefox/2.0.0.11"``, while
272 :mod:`urllib`'s default user agent string is
273 ``"Python-urllib/2.6"`` (on Python 2.6).
274 All header keys are sent in camel case.
275
276 An appropriate ``Content-Type`` header should be included if the *data*
277 argument is present. If this header has not been provided and *data*
278 is not ``None``, ``Content-Type: application/x-www-form-urlencoded`` will
279 be added as a default.
280
281 The next two arguments are only of interest for correct handling
282 of third-party HTTP cookies:
283
284 *origin_req_host* should be the request-host of the origin
285 transaction, as defined by :rfc:`2965`. It defaults to
286 ``http.cookiejar.request_host(self)``. This is the host name or IP
287 address of the original request that was initiated by the user.
288 For example, if the request is for an image in an HTML document,
289 this should be the request-host of the request for the page
290 containing the image.
291
292 *unverifiable* should indicate whether the request is unverifiable,
293 as defined by :rfc:`2965`. It defaults to ``False``. An unverifiable
294 request is one whose URL the user did not have the option to
295 approve. For example, if the request is for an image in an HTML
296 document, and the user had no option to approve the automatic
297 fetching of the image, this should be true.
298
299 *method* should be a string that indicates the HTTP request method that
300 will be used (e.g. ``'HEAD'``). If provided, its value is stored in the
301 :attr:`~Request.method` attribute and is used by :meth:`get_method`.
302 The default is ``'GET'`` if *data* is ``None`` or ``'POST'`` otherwise.
303 Subclasses may indicate a different default method by setting the
304 :attr:`~Request.method` attribute in the class itself.
305
306 .. note::
307 The request will not work as expected if the data object is unable
308 to deliver its content more than once (e.g. a file or an iterable
309 that can produce the content only once) and the request is retried
310 for HTTP redirects or authentication. The *data* is sent to the
311 HTTP server right away after the headers. There is no support for
312 a 100-continue expectation in the library.
313
314 .. versionchanged:: 3.3
315 :attr:`Request.method` argument is added to the Request class.
316
317 .. versionchanged:: 3.4
318 Default :attr:`Request.method` may be indicated at the class level.
319
320 .. versionchanged:: 3.6
321 Do not raise an error if the ``Content-Length`` has not been
322 provided and *data* is neither ``None`` nor a bytes object.
323 Fall back to use chunked transfer encoding instead.
324
325 .. class:: OpenerDirector()
326
327 The :class:`OpenerDirector` class opens URLs via :class:`BaseHandler`\ s chained
328 together. It manages the chaining of handlers, and recovery from errors.
329
330
331 .. class:: BaseHandler()
332
333 This is the base class for all registered handlers --- and handles only the
334 simple mechanics of registration.
335
336
337 .. class:: HTTPDefaultErrorHandler()
338
339 A class which defines a default handler for HTTP error responses; all responses
340 are turned into :exc:`~urllib.error.HTTPError` exceptions.
341
342
343 .. class:: HTTPRedirectHandler()
344
345 A class to handle redirections.
346
347
348 .. class:: HTTPCookieProcessor(cookiejar=None)
349
350 A class to handle HTTP Cookies.
351
352
353 .. class:: ProxyHandler(proxies=None)
354
355 Cause requests to go through a proxy. If *proxies* is given, it must be a
356 dictionary mapping protocol names to URLs of proxies. The default is to read
357 the list of proxies from the environment variables
358 ``<protocol>_proxy``. If no proxy environment variables are set, then
359 in a Windows environment proxy settings are obtained from the registry's
360 Internet Settings section, and in a macOS environment proxy information
361 is retrieved from the System Configuration Framework.
362
363 To disable autodetected proxy pass an empty dictionary.
364
365 The :envvar:`no_proxy` environment variable can be used to specify hosts
366 which shouldn't be reached via proxy; if set, it should be a comma-separated
367 list of hostname suffixes, optionally with ``:port`` appended, for example
368 ``cern.ch,ncsa.uiuc.edu,some.host:8080``.
369
370 .. note::
371
372 ``HTTP_PROXY`` will be ignored if a variable ``REQUEST_METHOD`` is set;
373 see the documentation on :func:`~urllib.request.getproxies`.
374
375
376 .. class:: HTTPPasswordMgr()
377
378 Keep a database of ``(realm, uri) -> (user, password)`` mappings.
379
380
381 .. class:: HTTPPasswordMgrWithDefaultRealm()
382
383 Keep a database of ``(realm, uri) -> (user, password)`` mappings. A realm of
384 ``None`` is considered a catch-all realm, which is searched if no other realm
385 fits.
386
387
388 .. class:: HTTPPasswordMgrWithPriorAuth()
389
390 A variant of :class:`HTTPPasswordMgrWithDefaultRealm` that also has a
391 database of ``uri -> is_authenticated`` mappings. Can be used by a
392 BasicAuth handler to determine when to send authentication credentials
393 immediately instead of waiting for a ``401`` response first.
394
395 .. versionadded:: 3.5
396
397
398 .. class:: AbstractBasicAuthHandler(password_mgr=None)
399
400 This is a mixin class that helps with HTTP authentication, both to the remote
401 host and to a proxy. *password_mgr*, if given, should be something that is
402 compatible with :class:`HTTPPasswordMgr`; refer to section
403 :ref:`http-password-mgr` for information on the interface that must be
404 supported. If *passwd_mgr* also provides ``is_authenticated`` and
405 ``update_authenticated`` methods (see
406 :ref:`http-password-mgr-with-prior-auth`), then the handler will use the
407 ``is_authenticated`` result for a given URI to determine whether or not to
408 send authentication credentials with the request. If ``is_authenticated``
409 returns ``True`` for the URI, credentials are sent. If ``is_authenticated``
410 is ``False``, credentials are not sent, and then if a ``401`` response is
411 received the request is re-sent with the authentication credentials. If
412 authentication succeeds, ``update_authenticated`` is called to set
413 ``is_authenticated`` ``True`` for the URI, so that subsequent requests to
414 the URI or any of its super-URIs will automatically include the
415 authentication credentials.
416
417 .. versionadded:: 3.5
418 Added ``is_authenticated`` support.
419
420
421 .. class:: HTTPBasicAuthHandler(password_mgr=None)
422
423 Handle authentication with the remote host. *password_mgr*, if given, should
424 be something that is compatible with :class:`HTTPPasswordMgr`; refer to
425 section :ref:`http-password-mgr` for information on the interface that must
426 be supported. HTTPBasicAuthHandler will raise a :exc:`ValueError` when
427 presented with a wrong Authentication scheme.
428
429
430 .. class:: ProxyBasicAuthHandler(password_mgr=None)
431
432 Handle authentication with the proxy. *password_mgr*, if given, should be
433 something that is compatible with :class:`HTTPPasswordMgr`; refer to section
434 :ref:`http-password-mgr` for information on the interface that must be
435 supported.
436
437
438 .. class:: AbstractDigestAuthHandler(password_mgr=None)
439
440 This is a mixin class that helps with HTTP authentication, both to the remote
441 host and to a proxy. *password_mgr*, if given, should be something that is
442 compatible with :class:`HTTPPasswordMgr`; refer to section
443 :ref:`http-password-mgr` for information on the interface that must be
444 supported.
445
446 .. versionchanged:: 3.14
447 Added support for HTTP digest authentication algorithm ``SHA-256``.
448
449
450 .. class:: HTTPDigestAuthHandler(password_mgr=None)
451
452 Handle authentication with the remote host. *password_mgr*, if given, should
453 be something that is compatible with :class:`HTTPPasswordMgr`; refer to
454 section :ref:`http-password-mgr` for information on the interface that must
455 be supported. When both Digest Authentication Handler and Basic
456 Authentication Handler are both added, Digest Authentication is always tried
457 first. If the Digest Authentication returns a 40x response again, it is sent
458 to Basic Authentication handler to Handle. This Handler method will raise a
459 :exc:`ValueError` when presented with an authentication scheme other than
460 Digest or Basic.
461
462 .. versionchanged:: 3.3
463 Raise :exc:`ValueError` on unsupported Authentication Scheme.
464
465
466
467 .. class:: ProxyDigestAuthHandler(password_mgr=None)
468
469 Handle authentication with the proxy. *password_mgr*, if given, should be
470 something that is compatible with :class:`HTTPPasswordMgr`; refer to section
471 :ref:`http-password-mgr` for information on the interface that must be
472 supported.
473
474
475 .. class:: HTTPHandler()
476
477 A class to handle opening of HTTP URLs.
478
479
480 .. class:: HTTPSHandler(debuglevel=0, context=None, check_hostname=None)
481
482 A class to handle opening of HTTPS URLs. *context* and *check_hostname*
483 have the same meaning as in :class:`http.client.HTTPSConnection`.
484
485 .. versionchanged:: 3.2
486 *context* and *check_hostname* were added.
487
488
489 .. class:: FileHandler()
490
491 Open local files.
492
493 .. class:: DataHandler()
494
495 Open data URLs.
496
497 .. versionadded:: 3.4
498
499 .. class:: FTPHandler()
500
501 Open FTP URLs.
502
503
504 .. class:: CacheFTPHandler()
505
506 Open FTP URLs, keeping a cache of open FTP connections to minimize delays.
507
508
509 .. class:: UnknownHandler()
510
511 A catch-all class to handle unknown URLs.
512
513
514 .. class:: HTTPErrorProcessor()
515
516 Process HTTP error responses.
517
518
519 .. _request-objects:
520
521 Request Objects
522 ---------------
523
524 The following methods describe :class:`Request`'s public interface,
525 and so all may be overridden in subclasses. It also defines several
526 public attributes that can be used by clients to inspect the parsed
527 request.
528
529 .. attribute:: Request.full_url
530
531 The original URL passed to the constructor.
532
533 .. versionchanged:: 3.4
534
535 Request.full_url is a property with setter, getter and a deleter. Getting
536 :attr:`~Request.full_url` returns the original request URL with the
537 fragment, if it was present.
538
539 .. attribute:: Request.type
540
541 The URI scheme.
542
543 .. attribute:: Request.host
544
545 The URI authority, typically a host, but may also contain a port
546 separated by a colon.
547
548 .. attribute:: Request.origin_req_host
549
550 The original host for the request, without port.
551
552 .. attribute:: Request.selector
553
554 The URI path. If the :class:`Request` uses a proxy, then selector
555 will be the full URL that is passed to the proxy.
556
557 .. attribute:: Request.data
558
559 The entity body for the request, or ``None`` if not specified.
560
561 .. versionchanged:: 3.4
562 Changing value of :attr:`Request.data` now deletes "Content-Length"
563 header if it was previously set or calculated.
564
565 .. attribute:: Request.unverifiable
566
567 boolean, indicates whether the request is unverifiable as defined
568 by :rfc:`2965`.
569
570 .. attribute:: Request.method
571
572 The HTTP request method to use. By default its value is :const:`None`,
573 which means that :meth:`~Request.get_method` will do its normal computation
574 of the method to be used. Its value can be set (thus overriding the default
575 computation in :meth:`~Request.get_method`) either by providing a default
576 value by setting it at the class level in a :class:`Request` subclass, or by
577 passing a value in to the :class:`Request` constructor via the *method*
578 argument.
579
580 .. versionadded:: 3.3
581
582 .. versionchanged:: 3.4
583 A default value can now be set in subclasses; previously it could only
584 be set via the constructor argument.
585
586
587 .. method:: Request.get_method()
588
589 Return a string indicating the HTTP request method. If
590 :attr:`Request.method` is not ``None``, return its value, otherwise return
591 ``'GET'`` if :attr:`Request.data` is ``None``, or ``'POST'`` if it's not.
592 This is only meaningful for HTTP requests.
593
594 .. versionchanged:: 3.3
595 get_method now looks at the value of :attr:`Request.method`.
596
597
598 .. method:: Request.add_header(key, val)
599
600 Add another header to the request. Headers are currently ignored by all
601 handlers except HTTP handlers, where they are added to the list of headers sent
602 to the server. Note that there cannot be more than one header with the same
603 name, and later calls will overwrite previous calls in case the *key* collides.
604 Currently, this is no loss of HTTP functionality, since all headers which have
605 meaning when used more than once have a (header-specific) way of gaining the
606 same functionality using only one header. Note that headers added using
607 this method are also added to redirected requests.
608
609
610 .. method:: Request.add_unredirected_header(key, header)
611
612 Add a header that will not be added to a redirected request.
613
614
615 .. method:: Request.has_header(header)
616
617 Return whether the instance has the named header (checks both regular and
618 unredirected).
619
620
621 .. method:: Request.remove_header(header)
622
623 Remove named header from the request instance (both from regular and
624 unredirected headers).
625
626 .. versionadded:: 3.4
627
628
629 .. method:: Request.get_full_url()
630
631 Return the URL given in the constructor.
632
633 .. versionchanged:: 3.4
634
635 Returns :attr:`Request.full_url`
636
637
638 .. method:: Request.set_proxy(host, type)
639
640 Prepare the request by connecting to a proxy server. The *host* and *type* will
641 replace those of the instance, and the instance's selector will be the original
642 URL given in the constructor.
643
644
645 .. method:: Request.get_header(header_name, default=None)
646
647 Return the value of the given header. If the header is not present, return
648 the default value.
649
650
651 .. method:: Request.header_items()
652
653 Return a list of tuples (header_name, header_value) of the Request headers.
654
655 .. versionchanged:: 3.4
656 The request methods add_data, has_data, get_data, get_type, get_host,
657 get_selector, get_origin_req_host and is_unverifiable that were deprecated
658 since 3.3 have been removed.
659
660
661 .. _opener-director-objects:
662
663 OpenerDirector Objects
664 ----------------------
665
666 :class:`OpenerDirector` instances have the following methods:
667
668
669 .. method:: OpenerDirector.add_handler(handler)
670
671 *handler* should be an instance of :class:`BaseHandler`. The following methods
672 are searched, and added to the possible chains (note that HTTP errors are a
673 special case). Note that, in the following, *protocol* should be replaced
674 with the actual protocol to handle, for example :meth:`http_response` would
675 be the HTTP protocol response handler. Also *type* should be replaced with
676 the actual HTTP code, for example :meth:`http_error_404` would handle HTTP
677 404 errors.
678
679 * :meth:`!<protocol>_open` --- signal that the handler knows how to open *protocol*
680 URLs.
681
682 See |protocol_open|_ for more information.
683
684 * :meth:`!http_error_\<type\>` --- signal that the handler knows how to handle HTTP
685 errors with HTTP error code *type*.
686
687 See |http_error_nnn|_ for more information.
688
689 * :meth:`!<protocol>_error` --- signal that the handler knows how to handle errors
690 from (non-\ ``http``) *protocol*.
691
692 * :meth:`!<protocol>_request` --- signal that the handler knows how to pre-process
693 *protocol* requests.
694
695 See |protocol_request|_ for more information.
696
697 * :meth:`!<protocol>_response` --- signal that the handler knows how to
698 post-process *protocol* responses.
699
700 See |protocol_response|_ for more information.
701
702 .. |protocol_open| replace:: :meth:`BaseHandler.<protocol>_open`
703 .. |http_error_nnn| replace:: :meth:`BaseHandler.http_error_\<nnn\>`
704 .. |protocol_request| replace:: :meth:`BaseHandler.<protocol>_request`
705 .. |protocol_response| replace:: :meth:`BaseHandler.<protocol>_response`
706
707 .. method:: OpenerDirector.open(url, data=None[, timeout])
708
709 Open the given *url* (which can be a request object or a string), optionally
710 passing the given *data*. Arguments, return values and exceptions raised are
711 the same as those of :func:`urlopen` (which simply calls the :meth:`open`
712 method on the currently installed global :class:`OpenerDirector`). The
713 optional *timeout* parameter specifies a timeout in seconds for blocking
714 operations like the connection attempt (if not specified, the global default
715 timeout setting will be used). The timeout feature actually works only for
716 HTTP, HTTPS and FTP connections.
717
718
719 .. method:: OpenerDirector.error(proto, *args)
720
721 Handle an error of the given protocol. This will call the registered error
722 handlers for the given protocol with the given arguments (which are protocol
723 specific). The HTTP protocol is a special case which uses the HTTP response
724 code to determine the specific error handler; refer to the :meth:`!http_error_\<type\>`
725 methods of the handler classes.
726
727 Return values and exceptions raised are the same as those of :func:`urlopen`.
728
729 OpenerDirector objects open URLs in three stages:
730
731 The order in which these methods are called within each stage is determined by
732 sorting the handler instances.
733
734 #. Every handler with a method named like :meth:`!<protocol>_request` has that
735 method called to pre-process the request.
736
737 #. Handlers with a method named like :meth:`!<protocol>_open` are called to handle
738 the request. This stage ends when a handler either returns a non-\ :const:`None`
739 value (ie. a response), or raises an exception (usually
740 :exc:`~urllib.error.URLError`). Exceptions are allowed to propagate.
741
742 In fact, the above algorithm is first tried for methods named
743 :meth:`~BaseHandler.default_open`. If all such methods return :const:`None`, the algorithm
744 is repeated for methods named like :meth:`!<protocol>_open`. If all such methods
745 return :const:`None`, the algorithm is repeated for methods named
746 :meth:`~BaseHandler.unknown_open`.
747
748 Note that the implementation of these methods may involve calls of the parent
749 :class:`OpenerDirector` instance's :meth:`~OpenerDirector.open` and
750 :meth:`~OpenerDirector.error` methods.
751
752 #. Every handler with a method named like :meth:`!<protocol>_response` has that
753 method called to post-process the response.
754
755
756 .. _base-handler-objects:
757
758 BaseHandler Objects
759 -------------------
760
761 :class:`BaseHandler` objects provide a couple of methods that are directly
762 useful, and others that are meant to be used by derived classes. These are
763 intended for direct use:
764
765
766 .. method:: BaseHandler.add_parent(director)
767
768 Add a director as parent.
769
770
771 .. method:: BaseHandler.close()
772
773 Remove any parents.
774
775 The following attribute and methods should only be used by classes derived from
776 :class:`BaseHandler`.
777
778 .. note::
779
780 The convention has been adopted that subclasses defining
781 :meth:`!<protocol>_request` or :meth:`!<protocol>_response` methods are named
782 :class:`!\*Processor`; all others are named :class:`!\*Handler`.
783
784
785 .. attribute:: BaseHandler.parent
786
787 A valid :class:`OpenerDirector`, which can be used to open using a different
788 protocol, or handle errors.
789
790
791 .. method:: BaseHandler.default_open(req)
792
793 This method is *not* defined in :class:`BaseHandler`, but subclasses should
794 define it if they want to catch all URLs.
795
796 This method, if implemented, will be called by the parent
797 :class:`OpenerDirector`. It should return a file-like object as described in
798 the return value of the :meth:`~OpenerDirector.open` method of :class:`OpenerDirector`, or ``None``.
799 It should raise :exc:`~urllib.error.URLError`, unless a truly exceptional
800 thing happens (for example, :exc:`MemoryError` should not be mapped to
801 :exc:`~urllib.error.URLError`).
802
803 This method will be called before any protocol-specific open method.
804
805
806 .. _protocol_open:
807 .. method:: BaseHandler.<protocol>_open(req)
808 :noindex:
809
810 This method is *not* defined in :class:`BaseHandler`, but subclasses should
811 define it if they want to handle URLs with the given protocol.
812
813 This method, if defined, will be called by the parent :class:`OpenerDirector`.
814 Return values should be the same as for :meth:`~BaseHandler.default_open`.
815
816
817 .. method:: BaseHandler.unknown_open(req)
818
819 This method is *not* defined in :class:`BaseHandler`, but subclasses should
820 define it if they want to catch all URLs with no specific registered handler to
821 open it.
822
823 This method, if implemented, will be called by the :attr:`parent`
824 :class:`OpenerDirector`. Return values should be the same as for
825 :meth:`default_open`.
826
827
828 .. method:: BaseHandler.http_error_default(req, fp, code, msg, hdrs)
829
830 This method is *not* defined in :class:`BaseHandler`, but subclasses should
831 override it if they intend to provide a catch-all for otherwise unhandled HTTP
832 errors. It will be called automatically by the :class:`OpenerDirector` getting
833 the error, and should not normally be called in other circumstances.
834
835 :class:`OpenerDirector` will call this method with five positional arguments:
836
837 1. a :class:`Request` object,
838 #. a file-like object with the HTTP error body,
839 #. the three-digit code of the error, as a string,
840 #. the user-visible explanation of the code, as a string, and
841 #. the headers of the error, as a mapping object.
842
843 Return values and exceptions raised should be the same as those of
844 :func:`urlopen`.
845
846
847 .. _http_error_nnn:
848 .. method:: BaseHandler.http_error_<nnn>(req, fp, code, msg, hdrs)
849
850 *nnn* should be a three-digit HTTP error code. This method is also not defined
851 in :class:`BaseHandler`, but will be called, if it exists, on an instance of a
852 subclass, when an HTTP error with code *nnn* occurs.
853
854 Subclasses should override this method to handle specific HTTP errors.
855
856 Arguments, return values and exceptions raised should be the same as for
857 :meth:`~BaseHandler.http_error_default`.
858
859
860 .. _protocol_request:
861 .. method:: BaseHandler.<protocol>_request(req)
862 :noindex:
863
864 This method is *not* defined in :class:`BaseHandler`, but subclasses should
865 define it if they want to pre-process requests of the given protocol.
866
867 This method, if defined, will be called by the parent :class:`OpenerDirector`.
868 *req* will be a :class:`Request` object. The return value should be a
869 :class:`Request` object.
870
871
872 .. _protocol_response:
873 .. method:: BaseHandler.<protocol>_response(req, response)
874 :noindex:
875
876 This method is *not* defined in :class:`BaseHandler`, but subclasses should
877 define it if they want to post-process responses of the given protocol.
878
879 This method, if defined, will be called by the parent :class:`OpenerDirector`.
880 *req* will be a :class:`Request` object. *response* will be an object
881 implementing the same interface as the return value of :func:`urlopen`. The
882 return value should implement the same interface as the return value of
883 :func:`urlopen`.
884
885
886 .. _http-redirect-handler:
887
888 HTTPRedirectHandler Objects
889 ---------------------------
890
891 .. note::
892
893 Some HTTP redirections require action from this module's client code. If this
894 is the case, :exc:`~urllib.error.HTTPError` is raised. See :rfc:`2616` for
895 details of the precise meanings of the various redirection codes.
896
897 An :exc:`~urllib.error.HTTPError` exception raised as a security consideration if the
898 HTTPRedirectHandler is presented with a redirected URL which is not an HTTP,
899 HTTPS or FTP URL.
900
901
902 .. method:: HTTPRedirectHandler.redirect_request(req, fp, code, msg, hdrs, newurl)
903
904 Return a :class:`Request` or ``None`` in response to a redirect. This is called
905 by the default implementations of the :meth:`!http_error_30\*` methods when a
906 redirection is received from the server. If a redirection should take place,
907 return a new :class:`Request` to allow :meth:`!http_error_30\*` to perform the
908 redirect to *newurl*. Otherwise, raise :exc:`~urllib.error.HTTPError` if
909 no other handler should try to handle this URL, or return ``None`` if you
910 can't but another handler might.
911
912 .. note::
913
914 The default implementation of this method does not strictly follow :rfc:`2616`,
915 which says that 301 and 302 responses to ``POST`` requests must not be
916 automatically redirected without confirmation by the user. In reality, browsers
917 do allow automatic redirection of these responses, changing the POST to a
918 ``GET``, and the default implementation reproduces this behavior.
919
920
921 .. method:: HTTPRedirectHandler.http_error_301(req, fp, code, msg, hdrs)
922
923 Redirect to the ``Location:`` or ``URI:`` URL. This method is called by the
924 parent :class:`OpenerDirector` when getting an HTTP 'moved permanently' response.
925
926
927 .. method:: HTTPRedirectHandler.http_error_302(req, fp, code, msg, hdrs)
928
929 The same as :meth:`http_error_301`, but called for the 'found' response.
930
931
932 .. method:: HTTPRedirectHandler.http_error_303(req, fp, code, msg, hdrs)
933
934 The same as :meth:`http_error_301`, but called for the 'see other' response.
935
936
937 .. method:: HTTPRedirectHandler.http_error_307(req, fp, code, msg, hdrs)
938
939 The same as :meth:`http_error_301`, but called for the 'temporary redirect'
940 response. It does not allow changing the request method from ``POST``
941 to ``GET``.
942
943
944 .. method:: HTTPRedirectHandler.http_error_308(req, fp, code, msg, hdrs)
945
946 The same as :meth:`http_error_301`, but called for the 'permanent redirect'
947 response. It does not allow changing the request method from ``POST``
948 to ``GET``.
949
950 .. versionadded:: 3.11
951
952
953 .. _http-cookie-processor:
954
955 HTTPCookieProcessor Objects
956 ---------------------------
957
958 :class:`HTTPCookieProcessor` instances have one attribute:
959
960 .. attribute:: HTTPCookieProcessor.cookiejar
961
962 The :class:`http.cookiejar.CookieJar` in which cookies are stored.
963
964
965 .. _proxy-handler:
966
967 ProxyHandler Objects
968 --------------------
969
970
971 .. method:: ProxyHandler.<protocol>_open(request)
972 :noindex:
973
974 The :class:`ProxyHandler` will have a method :meth:`!<protocol>_open` for every
975 *protocol* which has a proxy in the *proxies* dictionary given in the
976 constructor. The method will modify requests to go through the proxy, by
977 calling ``request.set_proxy()``, and call the next handler in the chain to
978 actually execute the protocol.
979
980
981 .. _http-password-mgr:
982
983 HTTPPasswordMgr Objects
984 -----------------------
985
986 These methods are available on :class:`HTTPPasswordMgr` and
987 :class:`HTTPPasswordMgrWithDefaultRealm` objects.
988
989
990 .. method:: HTTPPasswordMgr.add_password(realm, uri, user, passwd)
991
992 *uri* can be either a single URI, or a sequence of URIs. *realm*, *user* and
993 *passwd* must be strings. This causes ``(user, passwd)`` to be used as
994 authentication tokens when authentication for *realm* and a super-URI of any of
995 the given URIs is given.
996
997
998 .. method:: HTTPPasswordMgr.find_user_password(realm, authuri)
999
1000 Get user/password for given realm and URI, if any. This method will return
1001 ``(None, None)`` if there is no matching user/password.
1002
1003 For :class:`HTTPPasswordMgrWithDefaultRealm` objects, the realm ``None`` will be
1004 searched if the given *realm* has no matching user/password.
1005
1006
1007 .. _http-password-mgr-with-prior-auth:
1008
1009 HTTPPasswordMgrWithPriorAuth Objects
1010 ------------------------------------
1011
1012 This password manager extends :class:`HTTPPasswordMgrWithDefaultRealm` to support
1013 tracking URIs for which authentication credentials should always be sent.
1014
1015
1016 .. method:: HTTPPasswordMgrWithPriorAuth.add_password(realm, uri, user, \
1017 passwd, is_authenticated=False)
1018
1019 *realm*, *uri*, *user*, *passwd* are as for
1020 :meth:`HTTPPasswordMgr.add_password`. *is_authenticated* sets the initial
1021 value of the ``is_authenticated`` flag for the given URI or list of URIs.
1022 If *is_authenticated* is specified as ``True``, *realm* is ignored.
1023
1024
1025 .. method:: HTTPPasswordMgrWithPriorAuth.find_user_password(realm, authuri)
1026
1027 Same as for :class:`HTTPPasswordMgrWithDefaultRealm` objects
1028
1029
1030 .. method:: HTTPPasswordMgrWithPriorAuth.update_authenticated(self, uri, \
1031 is_authenticated=False)
1032
1033 Update the ``is_authenticated`` flag for the given *uri* or list
1034 of URIs.
1035
1036
1037 .. method:: HTTPPasswordMgrWithPriorAuth.is_authenticated(self, authuri)
1038
1039 Returns the current state of the ``is_authenticated`` flag for
1040 the given URI.
1041
1042
1043 .. _abstract-basic-auth-handler:
1044
1045 AbstractBasicAuthHandler Objects
1046 --------------------------------
1047
1048
1049 .. method:: AbstractBasicAuthHandler.http_error_auth_reqed(authreq, host, req, headers)
1050
1051 Handle an authentication request by getting a user/password pair, and re-trying
1052 the request. *authreq* should be the name of the header where the information
1053 about the realm is included in the request, *host* specifies the URL and path to
1054 authenticate for, *req* should be the (failed) :class:`Request` object, and
1055 *headers* should be the error headers.
1056
1057 *host* is either an authority (e.g. ``"python.org"``) or a URL containing an
1058 authority component (e.g. ``"https://python.org/"``). In either case, the
1059 authority must not contain a userinfo component (so, ``"python.org"`` and
1060 ``"python.org:80"`` are fine, ``"joe:[email protected]"`` is not).
1061
1062
1063 .. _http-basic-auth-handler:
1064
1065 HTTPBasicAuthHandler Objects
1066 ----------------------------
1067
1068
1069 .. method:: HTTPBasicAuthHandler.http_error_401(req, fp, code, msg, hdrs)
1070
1071 Retry the request with authentication information, if available.
1072
1073
1074 .. _proxy-basic-auth-handler:
1075
1076 ProxyBasicAuthHandler Objects
1077 -----------------------------
1078
1079
1080 .. method:: ProxyBasicAuthHandler.http_error_407(req, fp, code, msg, hdrs)
1081
1082 Retry the request with authentication information, if available.
1083
1084
1085 .. _abstract-digest-auth-handler:
1086
1087 AbstractDigestAuthHandler Objects
1088 ---------------------------------
1089
1090
1091 .. method:: AbstractDigestAuthHandler.http_error_auth_reqed(authreq, host, req, headers)
1092
1093 *authreq* should be the name of the header where the information about the realm
1094 is included in the request, *host* should be the host to authenticate to, *req*
1095 should be the (failed) :class:`Request` object, and *headers* should be the
1096 error headers.
1097
1098
1099 .. _http-digest-auth-handler:
1100
1101 HTTPDigestAuthHandler Objects
1102 -----------------------------
1103
1104
1105 .. method:: HTTPDigestAuthHandler.http_error_401(req, fp, code, msg, hdrs)
1106
1107 Retry the request with authentication information, if available.
1108
1109
1110 .. _proxy-digest-auth-handler:
1111
1112 ProxyDigestAuthHandler Objects
1113 ------------------------------
1114
1115
1116 .. method:: ProxyDigestAuthHandler.http_error_407(req, fp, code, msg, hdrs)
1117
1118 Retry the request with authentication information, if available.
1119
1120
1121 .. _http-handler-objects:
1122
1123 HTTPHandler Objects
1124 -------------------
1125
1126
1127 .. method:: HTTPHandler.http_open(req)
1128
1129 Send an HTTP request, which can be either GET or POST, depending on
1130 ``req.data``.
1131
1132
1133 .. _https-handler-objects:
1134
1135 HTTPSHandler Objects
1136 --------------------
1137
1138
1139 .. method:: HTTPSHandler.https_open(req)
1140
1141 Send an HTTPS request, which can be either GET or POST, depending on
1142 ``req.data``.
1143
1144
1145 .. _file-handler-objects:
1146
1147 FileHandler Objects
1148 -------------------
1149
1150
1151 .. method:: FileHandler.file_open(req)
1152
1153 Open the file locally, if there is no host name, or the host name is
1154 ``'localhost'``.
1155
1156 .. versionchanged:: 3.2
1157 This method is applicable only for local hostnames. When a remote
1158 hostname is given, a :exc:`~urllib.error.URLError` is raised.
1159
1160
1161 .. _data-handler-objects:
1162
1163 DataHandler Objects
1164 -------------------
1165
1166 .. method:: DataHandler.data_open(req)
1167
1168 Read a data URL. This kind of URL contains the content encoded in the URL
1169 itself. The data URL syntax is specified in :rfc:`2397`. This implementation
1170 ignores white spaces in base64 encoded data URLs so the URL may be wrapped
1171 in whatever source file it comes from. But even though some browsers don't
1172 mind about a missing padding at the end of a base64 encoded data URL, this
1173 implementation will raise a :exc:`ValueError` in that case.
1174
1175
1176 .. _ftp-handler-objects:
1177
1178 FTPHandler Objects
1179 ------------------
1180
1181
1182 .. method:: FTPHandler.ftp_open(req)
1183
1184 Open the FTP file indicated by *req*. The login is always done with empty
1185 username and password.
1186
1187
1188 .. _cacheftp-handler-objects:
1189
1190 CacheFTPHandler Objects
1191 -----------------------
1192
1193 :class:`CacheFTPHandler` objects are :class:`FTPHandler` objects with the
1194 following additional methods:
1195
1196
1197 .. method:: CacheFTPHandler.setTimeout(t)
1198
1199 Set timeout of connections to *t* seconds.
1200
1201
1202 .. method:: CacheFTPHandler.setMaxConns(m)
1203
1204 Set maximum number of cached connections to *m*.
1205
1206
1207 .. _unknown-handler-objects:
1208
1209 UnknownHandler Objects
1210 ----------------------
1211
1212
1213 .. method:: UnknownHandler.unknown_open()
1214
1215 Raise a :exc:`~urllib.error.URLError` exception.
1216
1217
1218 .. _http-error-processor-objects:
1219
1220 HTTPErrorProcessor Objects
1221 --------------------------
1222
1223 .. method:: HTTPErrorProcessor.http_response(request, response)
1224
1225 Process HTTP error responses.
1226
1227 For 200 error codes, the response object is returned immediately.
1228
1229 For non-200 error codes, this simply passes the job on to the
1230 :meth:`!http_error_\<type\>` handler methods, via :meth:`OpenerDirector.error`.
1231 Eventually, :class:`HTTPDefaultErrorHandler` will raise an
1232 :exc:`~urllib.error.HTTPError` if no other handler handles the error.
1233
1234
1235 .. method:: HTTPErrorProcessor.https_response(request, response)
1236
1237 Process HTTPS error responses.
1238
1239 The behavior is same as :meth:`http_response`.
1240
1241
1242 .. _urllib-request-examples:
1243
1244 Examples
1245 --------
1246
1247 In addition to the examples below, more examples are given in
1248 :ref:`urllib-howto`.
1249
1250 This example gets the python.org main page and displays the first 300 bytes of
1251 it::
1252
1253 >>> import urllib.request
1254 >>> with urllib.request.urlopen('https://www.python.org/') as f:
1255 ... # The response may be compressed (for example, 'gzip').
1256 ... print(f.headers.get('Content-Encoding'))
1257 ... data = f.read()
1258 ... if f.headers.get('Content-Encoding') == 'gzip':
1259 ... import gzip
1260 ... data = gzip.decompress(data)
1261 ... print(data[:300].decode('utf-8', errors='replace'))
1262
1263 Note that urlopen returns a bytes object. This is because there is no way
1264 for urlopen to automatically determine the encoding of the byte stream
1265 it receives from the HTTP server. In general, a program will decode
1266 the returned bytes object to string once it determines or guesses
1267 the appropriate encoding.
1268
1269 The following HTML spec document, https://html.spec.whatwg.org/#charset, lists
1270 the various ways in which an HTML or an XML document could have specified its
1271 encoding information.
1272
1273 For additional information, see the W3C document: https://www.w3.org/International/questions/qa-html-encoding-declarations.
1274
1275 As the python.org website uses *utf-8* encoding as specified in its meta tag, we
1276 will use the same for decoding the bytes object::
1277
1278 >>> with urllib.request.urlopen('https://www.python.org/') as f:
1279 ... # Check for compression and decode appropriately.
1280 ... enc = f.headers.get('Content-Encoding')
1281 ... data = f.read()
1282 ... if enc == 'gzip':
1283 ... import gzip
1284 ... data = gzip.decompress(data)
1285 ... print(data[:100].decode('utf-8', errors='replace'))
1286 ...
1287
1288 It is also possible to achieve the same result without using the
1289 :term:`context manager` approach::
1290
1291 >>> import urllib.request
1292 >>> f = urllib.request.urlopen('https://www.python.org/')
1293 >>> try:
1294 ... enc = f.headers.get('Content-Encoding')
1295 ... data = f.read()
1296 ... if enc == 'gzip':
1297 ... import gzip
1298 ... data = gzip.decompress(data)
1299 ... print(data[:100].decode('utf-8', errors='replace'))
1300 ... finally:
1301 ... f.close()
1302
1303 In the following example, we are sending a data-stream to the stdin of a CGI
1304 and reading the data it returns to us. Note that this example will only work
1305 when the Python installation supports SSL. ::
1306
1307 >>> import urllib.request
1308 >>> req = urllib.request.Request(url='https://localhost/cgi-bin/test.cgi',
1309 ... data=b'This data is passed to stdin of the CGI')
1310 >>> with urllib.request.urlopen(req) as f:
1311 ... print(f.read().decode('utf-8'))
1312 ...
1313 Got Data: "This data is passed to stdin of the CGI"
1314
1315 The code for the sample CGI used in the above example is::
1316
1317 #!/usr/bin/env python
1318 import sys
1319 data = sys.stdin.read()
1320 print('Content-type: text/plain\n\nGot Data: "%s"' % data)
1321
1322 Here is an example of doing a ``PUT`` request using :class:`Request`::
1323
1324 import urllib.request
1325 DATA = b'some data'
1326 req = urllib.request.Request(url='http://localhost:8080', data=DATA, method='PUT')
1327 with urllib.request.urlopen(req) as f:
1328 pass
1329 print(f.status)
1330 print(f.reason)
1331
1332 Use of Basic HTTP Authentication::
1333
1334 import urllib.request
1335 # Create an OpenerDirector with support for Basic HTTP Authentication...
1336 auth_handler = urllib.request.HTTPBasicAuthHandler()
1337 auth_handler.add_password(realm='PDQ Application',
1338 uri='https://mahler:8092/site-updates.py',
1339 user='klem',
1340 passwd='kadidd!ehopper')
1341 opener = urllib.request.build_opener(auth_handler)
1342 # ...and install it globally so it can be used with urlopen.
1343 urllib.request.install_opener(opener)
1344 with urllib.request.urlopen('http://www.example.com/login.html') as f:
1345 print(f.read().decode('utf-8'))
1346
1347 :func:`build_opener` provides many handlers by default, including a
1348 :class:`ProxyHandler`. By default, :class:`ProxyHandler` uses the environment
1349 variables named ``<scheme>_proxy``, where ``<scheme>`` is the URL scheme
1350 involved. For example, the :envvar:`!http_proxy` environment variable is read to
1351 obtain the HTTP proxy's URL.
1352
1353 This example replaces the default :class:`ProxyHandler` with one that uses
1354 programmatically supplied proxy URLs, and adds proxy authorization support with
1355 :class:`ProxyBasicAuthHandler`. ::
1356
1357 proxy_handler = urllib.request.ProxyHandler({'http': 'http://www.example.com:3128/'})
1358 proxy_auth_handler = urllib.request.ProxyBasicAuthHandler()
1359 proxy_auth_handler.add_password('realm', 'host', 'username', 'password')
1360
1361 opener = urllib.request.build_opener(proxy_handler, proxy_auth_handler)
1362 # This time, rather than install the OpenerDirector, we use it directly:
1363 with opener.open('http://www.example.com/login.html') as f:
1364 print(f.read().decode('utf-8'))
1365
1366 Adding HTTP headers:
1367
1368 Use the *headers* argument to the :class:`Request` constructor, or::
1369
1370 import urllib.request
1371 req = urllib.request.Request('http://www.example.com/')
1372 req.add_header('Referer', 'https://www.python.org/')
1373 # Customize the default User-Agent header value:
1374 req.add_header('User-Agent', 'urllib-example/0.1 (Contact: . . .)')
1375 with urllib.request.urlopen(req) as f:
1376 print(f.read().decode('utf-8'))
1377
1378
1379 :class:`OpenerDirector` automatically adds a :mailheader:`User-Agent` header to
1380 every :class:`Request`. To change this::
1381
1382 import urllib.request
1383 opener = urllib.request.build_opener()
1384 opener.addheaders = [('User-agent', 'Mozilla/5.0')]
1385 with opener.open('http://www.example.com/') as f:
1386 print(f.read().decode('utf-8'))
1387
1388 Also, remember that a few standard headers (:mailheader:`Content-Length`,
1389 :mailheader:`Content-Type` and :mailheader:`Host`)
1390 are added when the :class:`Request` is passed to :func:`urlopen` (or
1391 :meth:`OpenerDirector.open`).
1392
1393 .. _urllib-examples:
1394
1395 Here is an example session that uses the ``GET`` method to retrieve a URL
1396 containing parameters::
1397
1398 >>> import urllib.request
1399 >>> import urllib.parse
1400 >>> params = urllib.parse.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
1401 >>> url = "https://www.python.org/?%s" % params
1402 >>> with urllib.request.urlopen(url) as f:
1403 ... print(f.read().decode('utf-8'))
1404 ...
1405
1406 The following example uses the ``POST`` method instead. Note that params output
1407 from urlencode is encoded to bytes before it is sent to urlopen as data::
1408
1409 >>> import urllib.request
1410 >>> import urllib.parse
1411 >>> data = urllib.parse.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
1412 >>> data = data.encode('ascii')
1413 >>> with urllib.request.urlopen("https://httpbin.org/post", data) as f:
1414 ... print(f.read().decode('utf-8'))
1415 ...
1416
1417 The following example uses an explicitly specified HTTP proxy, overriding
1418 environment settings::
1419
1420 >>> import urllib.request
1421 >>> proxies = {'http': 'http://proxy.example.com:8080/'}
1422 >>> opener = urllib.request.build_opener(urllib.request.ProxyHandler(proxies))
1423 >>> with opener.open("https://www.python.org") as f:
1424 ... f.read().decode('utf-8')
1425 ...
1426
1427 The following example uses no proxies at all, overriding environment settings::
1428
1429 >>> import urllib.request
1430 >>> opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
1431 >>> with opener.open("https://www.python.org/") as f:
1432 ... f.read().decode('utf-8')
1433 ...
1434
1435
1436 Legacy interface
1437 ----------------
1438
1439 The following functions and classes are ported from the Python 2 module
1440 ``urllib`` (as opposed to ``urllib2``). They might become deprecated at
1441 some point in the future.
1442
1443 .. function:: urlretrieve(url, filename=None, reporthook=None, data=None)
1444
1445 Copy a network object denoted by a URL to a local file. If the URL
1446 points to a local file, the object will not be copied unless filename is supplied.
1447 Return a tuple ``(filename, headers)`` where *filename* is the
1448 local file name under which the object can be found, and *headers* is whatever
1449 the :meth:`!info` method of the object returned by :func:`urlopen` returned (for
1450 a remote object). Exceptions are the same as for :func:`urlopen`.
1451
1452 The second argument, if present, specifies the file location to copy to (if
1453 absent, the location will be a tempfile with a generated name). The third
1454 argument, if present, is a callable that will be called once on
1455 establishment of the network connection and once after each block read
1456 thereafter. The callable will be passed three arguments; a count of blocks
1457 transferred so far, a block size in bytes, and the total size of the file. The
1458 third argument may be ``-1`` on older FTP servers which do not return a file
1459 size in response to a retrieval request.
1460
1461 The following example illustrates the most common usage scenario::
1462
1463 >>> import urllib.request
1464 >>> local_filename, headers = urllib.request.urlretrieve('https://python.org/')
1465 >>> html = open(local_filename)
1466 >>> html.close()
1467
1468 If the *url* uses the :file:`http:` scheme identifier, the optional *data*
1469 argument may be given to specify a ``POST`` request (normally the request
1470 type is ``GET``). The *data* argument must be a bytes object in standard
1471 :mimetype:`application/x-www-form-urlencoded` format; see the
1472 :func:`urllib.parse.urlencode` function.
1473
1474 :func:`urlretrieve` will raise :exc:`~urllib.error.ContentTooShortError` when it detects that
1475 the amount of data available was less than the expected amount (which is the
1476 size reported by a *Content-Length* header). This can occur, for example, when
1477 the download is interrupted.
1478
1479 The *Content-Length* is treated as a lower bound: if there's more data to read,
1480 urlretrieve reads more data, but if less data is available, it raises the
1481 exception.
1482
1483 You can still retrieve the downloaded data in this case, it is stored in the
1484 :attr:`!content` attribute of the exception instance.
1485
1486 If no *Content-Length* header was supplied, urlretrieve can not check the size
1487 of the data it has downloaded, and just returns it. In this case you just have
1488 to assume that the download was successful.
1489
1490 .. function:: urlcleanup()
1491
1492 Cleans up temporary files that may have been left behind by previous
1493 calls to :func:`urlretrieve`.
1494
1495
1496 :mod:`!urllib.request` Restrictions
1497 -----------------------------------
1498
1499 .. index::
1500 pair: HTTP; protocol
1501 pair: FTP; protocol
1502
1503 * Currently, only the following protocols are supported: HTTP (versions 0.9 and
1504 1.0), FTP, local files, and data URLs.
1505
1506 .. versionchanged:: 3.4 Added support for data URLs.
1507
1508 * The caching feature of :func:`urlretrieve` has been disabled until someone
1509 finds the time to hack proper processing of Expiration time headers.
1510
1511 * There should be a function to query whether a particular URL is in the cache.
1512
1513 * For backward compatibility, if a URL appears to point to a local file but the
1514 file can't be opened, the URL is re-interpreted using the FTP protocol. This
1515 can sometimes cause confusing error messages.
1516
1517 * The :func:`urlopen` and :func:`urlretrieve` functions can cause arbitrarily
1518 long delays while waiting for a network connection to be set up. This means
1519 that it is difficult to build an interactive web client using these functions
1520 without using threads.
1521
1522 .. index::
1523 single: HTML
1524 pair: HTTP; protocol
1525
1526 * The data returned by :func:`urlopen` or :func:`urlretrieve` is the raw data
1527 returned by the server. This may be binary data (such as an image), plain text
1528 or (for example) HTML. The HTTP protocol provides type information in the reply
1529 header, which can be inspected by looking at the :mailheader:`Content-Type`
1530 header. If the returned data is HTML, you can use the module
1531 :mod:`html.parser` to parse it.
1532
1533 .. index:: single: FTP
1534
1535 * The code handling the FTP protocol cannot differentiate between a file and a
1536 directory. This can lead to unexpected behavior when attempting to read a URL
1537 that points to a file that is not accessible. If the URL ends in a ``/``, it is
1538 assumed to refer to a directory and will be handled accordingly. But if an
1539 attempt to read a file leads to a 550 error (meaning the URL cannot be found or
1540 is not accessible, often for permission reasons), then the path is treated as a
1541 directory in order to handle the case when a directory is specified by a URL but
1542 the trailing ``/`` has been left off. This can cause misleading results when
1543 you try to fetch a file whose read permissions make it inaccessible; the FTP
1544 code will try to read it, fail with a 550 error, and then perform a directory
1545 listing for the unreadable file. If fine-grained control is needed, consider
1546 using the :mod:`ftplib` module.
1547
1548
1549
1550 :mod:`!urllib.response` --- Response classes used by urllib
1551 ===========================================================
1552
1553 .. module:: urllib.response
1554 :synopsis: Response classes used by urllib.
1555
1556 The :mod:`!urllib.response` module defines functions and classes which define a
1557 minimal file-like interface, including ``read()`` and ``readline()``.
1558 Functions defined by this module are used internally by the :mod:`!urllib.request` module.
1559 The typical response object is a :class:`urllib.response.addinfourl` instance:
1560
1561 .. class:: addinfourl
1562
1563 .. attribute:: url
1564
1565 URL of the resource retrieved, commonly used to determine if a redirect was followed.
1566
1567 .. attribute:: headers
1568
1569 Returns the headers of the response in the form of an :class:`~email.message.EmailMessage` instance.
1570
1571 .. attribute:: status
1572
1573 .. versionadded:: 3.9
1574
1575 Status code returned by server.
1576
1577 .. method:: geturl()
1578
1579 .. deprecated:: 3.9
1580 Deprecated in favor of :attr:`~addinfourl.url`.
1581
1582 .. method:: info()
1583
1584 .. deprecated:: 3.9
1585 Deprecated in favor of :attr:`~addinfourl.headers`.
1586
1587 .. attribute:: code
1588
1589 .. deprecated:: 3.9
1590 Deprecated in favor of :attr:`~addinfourl.status`.
1591
1592 .. method:: getcode()
1593
1594 .. deprecated:: 3.9
1595 Deprecated in favor of :attr:`~addinfourl.status`.