Debian Codegraph @

Doc/library/urllib.request.rst

raw
1 :mod:`!urllib.request` --- Extensible library for opening URLs
2 ==============================================================
3
4 .. module:: urllib.request
5 :synopsis: Extensible library for opening URLs.
6
7 .. moduleauthor:: Jeremy Hylton <[email protected]>
8 .. sectionauthor:: Moshe Zadka <[email protected]>
9 .. sectionauthor:: Senthil Kumaran <[email protected]>
10
11 **Source code:** :source:`Lib/urllib/request.py`
12
13 --------------
14
15 The :mod:`!urllib.request` module defines functions and classes which help in
16 opening URLs (mostly HTTP) in a complex world --- basic and digest
17 authentication, redirections, cookies and more.
18
19 .. seealso::
20
21 The `Requests package <https://requests.readthedocs.io/en/master/>`_
22 is recommended for a higher-level HTTP client interface.
23
24 .. warning::
25
26 On macOS it is unsafe to use this module in programs using
27 :func:`os.fork` because the :func:`getproxies` implementation for
28 macOS uses a higher-level system API. Set the environment variable
29 ``no_proxy`` to ``*`` to avoid this problem
30 (e.g. ``os.environ["no_proxy"] = "*"``).
31
32 .. include:: ../includes/wasm-notavail.rst
33
34 The :mod:`!urllib.request` module defines the following functions:
35
36
37 .. function:: urlopen(url, data=None[, timeout], *, context=None)
38
39 Open *url*, which can be either a string containing a valid, properly
40 encoded URL, or a :class:`Request` object.
41
42 *data* must be an object specifying additional data to be sent to the
43 server, or ``None`` if no such data is needed. See :class:`Request`
44 for details.
45
46 urllib.request module uses HTTP/1.1 and includes ``Connection:close`` header
47 in its HTTP requests.
48
49 The optional *timeout* parameter specifies a timeout in seconds for
50 blocking operations like the connection attempt (if not specified,
51 the global default timeout setting will be used). This actually
52 only works for HTTP, HTTPS and FTP connections.
53
54 If *context* is specified, it must be a :class:`ssl.SSLContext` instance
55 describing the various SSL options. See :class:`~http.client.HTTPSConnection`
56 for more details.
57
58 This function always returns an object which can work as a
59 :term:`context manager` and has the properties *url*, *headers*, and *status*.
60 See :class:`urllib.response.addinfourl` for more detail on these properties.
61
62 For HTTP and HTTPS URLs, this function returns a
63 :class:`http.client.HTTPResponse` object slightly modified. In addition
64 to the three new methods above, the msg attribute contains the
65 same information as the :attr:`~http.client.HTTPResponse.reason`
66 attribute --- the reason phrase returned by server --- instead of
67 the response headers as it is specified in the documentation for
68 :class:`~http.client.HTTPResponse`.
69
70 For FTP, file, and data URLs, this function
71 returns a :class:`urllib.response.addinfourl` object.
72
73 Raises :exc:`~urllib.error.URLError` on protocol errors.
74
75 Note that ``None`` may be returned if no handler handles the request (though
76 the default installed global :class:`OpenerDirector` uses
77 :class:`UnknownHandler` to ensure this never happens).
78
79 In addition, if proxy settings are detected (for example, when a ``*_proxy``
80 environment variable like :envvar:`!http_proxy` is set),
81 :class:`ProxyHandler` is default installed and makes sure the requests are
82 handled through the proxy.
83
84 The legacy ``urllib.urlopen`` function from Python 2.6 and earlier has been
85 discontinued; :func:`urllib.request.urlopen` corresponds to the old
86 ``urllib2.urlopen``. Proxy handling, which was done by passing a dictionary
87 parameter to ``urllib.urlopen``, can be obtained by using
88 :class:`ProxyHandler` objects.
89
90 .. audit-event:: urllib.Request fullurl,data,headers,method urllib.request.urlopen
91
92 The default opener raises an :ref:`auditing event <auditing>`
93 ``urllib.Request`` with arguments ``fullurl``, ``data``, ``headers``,
94 ``method`` taken from the request object.
95
96 .. versionchanged:: 3.2
97 *cafile* and *capath* were added.
98
99 HTTPS virtual hosts are now supported if possible (that is, if
100 :const:`ssl.HAS_SNI` is true).
101
102 *data* can be an iterable object.
103
104 .. versionchanged:: 3.3
105 *cadefault* was added.
106
107 .. versionchanged:: 3.4.3
108 *context* was added.
109
110 .. versionchanged:: 3.10
111 HTTPS connection now send an ALPN extension with protocol indicator
112 ``http/1.1`` when no *context* is given. Custom *context* should set
113 ALPN protocols with :meth:`~ssl.SSLContext.set_alpn_protocols`.
114
115 .. versionchanged:: 3.13
116 Remove *cafile*, *capath* and *cadefault* parameters: use the *context*
117 parameter instead.
118
119
120 .. function:: install_opener(opener)
121
122 Install an :class:`OpenerDirector` instance as the default global opener.
123 Installing an opener is only necessary if you want urlopen to use that
124 opener; otherwise, simply call :meth:`OpenerDirector.open` instead of
125 :func:`~urllib.request.urlopen`. The code does not check for a real
126 :class:`OpenerDirector`, and any class with the appropriate interface will
127 work.
128
129
130 .. function:: build_opener([handler, ...])
131
132 Return an :class:`OpenerDirector` instance, which chains the handlers in the
133 order given. *handler*\s can be either instances of :class:`BaseHandler`, or
134 subclasses of :class:`BaseHandler` (in which case it must be possible to call
135 the constructor without any parameters). Instances of the following classes
136 will be in front of the *handler*\s, unless the *handler*\s contain them,
137 instances of them or subclasses of them: :class:`ProxyHandler` (if proxy
138 settings are detected), :class:`UnknownHandler`, :class:`HTTPHandler`,
139 :class:`HTTPDefaultErrorHandler`, :class:`HTTPRedirectHandler`,
140 :class:`FTPHandler`, :class:`FileHandler`, :class:`HTTPErrorProcessor`.
141
142 If the Python installation has SSL support (i.e., if the :mod:`ssl` module
143 can be imported), :class:`HTTPSHandler` will also be added.
144
145 A :class:`BaseHandler` subclass may also change its :attr:`handler_order`
146 attribute to modify its position in the handlers list.
147
148
149 .. function:: pathname2url(path, *, add_scheme=False)
150
151 Convert the given local path to a ``file:`` URL. This function uses
152 :func:`~urllib.parse.quote` function to encode the path.
153
154 If *add_scheme* is false (the default), the return value omits the
155 ``file:`` scheme prefix. Set *add_scheme* to true to return a complete URL.
156
157 This example shows the function being used on Windows::
158
159 >>> from urllib.request import pathname2url
160 >>> path = 'C:\\Program Files'
161 >>> pathname2url(path, add_scheme=True)
162 'file:///C:/Program%20Files'
163
164 .. versionchanged:: 3.14
165 Windows drive letters are no longer converted to uppercase, and ``:``
166 characters not following a drive letter no longer cause an
167 :exc:`OSError` exception to be raised on Windows.
168
169 .. versionchanged:: 3.14
170 Paths beginning with a slash are converted to URLs with authority
171 sections. For example, the path ``/etc/hosts`` is converted to
172 the URL ``///etc/hosts``.
173
174 .. versionchanged:: 3.14
175 The *add_scheme* parameter was added.
176
177
178 .. function:: url2pathname(url, *, require_scheme=False, resolve_host=False)
179
180 Convert the given ``file:`` URL to a local path. This function uses
181 :func:`~urllib.parse.unquote` to decode the URL.
182
183 If *require_scheme* is false (the default), the given value should omit a
184 ``file:`` scheme prefix. If *require_scheme* is set to true, the given
185 value should include the prefix; a :exc:`~urllib.error.URLError` is raised
186 if it doesn't.
187
188 The URL authority is discarded if it is empty, ``localhost``, or the local
189 hostname. Otherwise, if *resolve_host* is set to true, the authority is
190 resolved using :func:`socket.gethostbyname` and discarded if it matches a
191 local IP address (as per :rfc:`RFC 8089 §3 <8089#section-3>`). If the
192 authority is still unhandled, then on Windows a UNC path is returned, and
193 on other platforms a :exc:`~urllib.error.URLError` is raised.
194
195 This example shows the function being used on Windows::
196
197 >>> from urllib.request import url2pathname
198 >>> url = 'file:///C:/Program%20Files'
199 >>> url2pathname(url, require_scheme=True)
200 'C:\\Program Files'
201
202 .. versionchanged:: 3.14
203 Windows drive letters are no longer converted to uppercase, and ``:``
204 characters not following a drive letter no longer cause an
205 :exc:`OSError` exception to be raised on Windows.
206
207 .. versionchanged:: 3.14
208 The URL authority is discarded if it matches the local hostname.
209 Otherwise, if the authority isn't empty or ``localhost``, then on
210 Windows a UNC path is returned (as before), and on other platforms a
211 :exc:`~urllib.error.URLError` is raised.
212
213 .. versionchanged:: 3.14
214 The URL query and fragment components are discarded if present.
215
216 .. versionchanged:: 3.14
217 The *require_scheme* and *resolve_host* parameters were added.
218
219
220 .. function:: getproxies()
221
222 This helper function returns a dictionary of scheme to proxy server URL
223 mappings. It scans the environment for variables named ``<scheme>_proxy``,
224 in a case insensitive approach, for all operating systems first, and when it
225 cannot find it, looks for proxy information from System
226 Configuration for macOS and Windows Systems Registry for Windows.
227 If both lowercase and uppercase environment variables exist (and disagree),
228 lowercase is preferred.
229
230 .. note::
231
232 If the environment variable ``REQUEST_METHOD`` is set, which usually
233 indicates your script is running in a CGI environment, the environment
234 variable ``HTTP_PROXY`` (uppercase ``_PROXY``) will be ignored. This is
235 because that variable can be injected by a client using the "Proxy:" HTTP
236 header. If you need to use an HTTP proxy in a CGI environment, either use
237 ``ProxyHandler`` explicitly, or make sure the variable name is in
238 lowercase (or at least the ``_proxy`` suffix).
239
240
241 The following classes are provided:
242
243 .. class:: Request(url, data=None, headers={}, origin_req_host=None, unverifiable=False, method=None)
244
245 This class is an abstraction of a URL request.
246
247 *url* should be a string containing a valid, properly encoded URL.
248
249 *data* must be an object specifying additional data to send to the
250 server, or ``None`` if no such data is needed. Currently HTTP
251 requests are the only ones that use *data*. The supported object
252 types include bytes, file-like objects, and iterables of bytes-like objects.
253 If no ``Content-Length`` nor ``Transfer-Encoding`` header field
254 has been provided, :class:`HTTPHandler` will set these headers according
255 to the type of *data*. ``Content-Length`` will be used to send
256 bytes objects, while ``Transfer-Encoding: chunked`` as specified in
257 :rfc:`7230`, Section 3.3.1 will be used to send files and other iterables.
258
259 For an HTTP POST request method, *data* should be a buffer in the
260 standard :mimetype:`application/x-www-form-urlencoded` format. The
261 :func:`urllib.parse.urlencode` function takes a mapping or sequence
262 of 2-tuples and returns an ASCII string in this format. It should
263 be encoded to bytes before being used as the *data* parameter.
264
265 *headers* should be a dictionary, and will be treated as if
266 :meth:`add_header` was called with each key and value as arguments.
267 This is often used to "spoof" the ``User-Agent`` header value, which is
268 used by a browser to identify itself -- some HTTP servers only
269 allow requests coming from common browsers as opposed to scripts.
270 For example, Mozilla Firefox may identify itself as ``"Mozilla/5.0
271 (X11; U; Linux i686) Gecko/20071127 Firefox/2.0.0.11"``, while
272 :mod:`urllib`'s default user agent string is
273 ``"Python-urllib/2.6"`` (on Python 2.6).
274 All header keys are sent in camel case.
275
276 An appropriate ``Content-Type`` header should be included if the *data*
277 argument is present. If this header has not been provided and *data*
278 is not ``None``, ``Content-Type: application/x-www-form-urlencoded`` will
279 be added as a default.
280
281 The next two arguments are only of interest for correct handling
282 of third-party HTTP cookies:
283
284 *origin_req_host* should be the request-host of the origin
285 transaction, as defined by :rfc:`2965`. It defaults to
286 ``http.cookiejar.request_host(self)``. This is the host name or IP
287 address of the original request that was initiated by the user.
288 For example, if the request is for an image in an HTML document,
289 this should be the request-host of the request for the page
290 containing the image.
291
292 *unverifiable* should indicate whether the request is unverifiable,
293 as defined by :rfc:`2965`. It defaults to ``False``. An unverifiable
294 request is one whose URL the user did not have the option to
295 approve. For example, if the request is for an image in an HTML
296 document, and the user had no option to approve the automatic
297 fetching of the image, this should be true.
298
299 *method* should be a string that indicates the HTTP request method that
300 will be used (e.g. ``'HEAD'``). If provided, its value is stored in the
301 :attr:`~Request.method` attribute and is used by :meth:`get_method`.
302 The default is ``'GET'`` if *data* is ``None`` or ``'POST'`` otherwise.
303 Subclasses may indicate a different default method by setting the
304 :attr:`~Request.method` attribute in the class itself.
305
306 .. note::
307 The request will not work as expected if the data object is unable
308 to deliver its content more than once (e.g. a file or an iterable
309 that can produce the content only once) and the request is retried
310 for HTTP redirects or authentication. The *data* is sent to the
311 HTTP server right away after the headers. There is no support for
312 a 100-continue expectation in the library.
313
314 .. versionchanged:: 3.3
315 :attr:`Request.method` argument is added to the Request class.
316
317 .. versionchanged:: 3.4
318 Default :attr:`Request.method` may be indicated at the class level.
319
320 .. versionchanged:: 3.6
321 Do not raise an error if the ``Content-Length`` has not been
322 provided and *data* is neither ``None`` nor a bytes object.
323 Fall back to use chunked transfer encoding instead.
324
325 .. class:: OpenerDirector()
326
327 The :class:`OpenerDirector` class opens URLs via :class:`BaseHandler`\ s chained
328 together. It manages the chaining of handlers, and recovery from errors.
329
330
331 .. class:: BaseHandler()
332
333 This is the base class for all registered handlers --- and handles only the
334 simple mechanics of registration.
335
336
337 .. class:: HTTPDefaultErrorHandler()
338
339 A class which defines a default handler for HTTP error responses; all responses
340 are turned into :exc:`~urllib.error.HTTPError` exceptions.
341
342
343 .. class:: HTTPRedirectHandler()
344
345 A class to handle redirections.
346
347
348 .. class:: HTTPCookieProcessor(cookiejar=None)
349
350 A class to handle HTTP Cookies.
351
352
353 .. class:: ProxyHandler(proxies=None)
354
355 Cause requests to go through a proxy. If *proxies* is given, it must be a
356 dictionary mapping protocol names to URLs of proxies. The default is to read
357 the list of proxies from the environment variables
358 ``<protocol>_proxy``. If no proxy environment variables are set, then
359 in a Windows environment proxy settings are obtained from the registry's
360 Internet Settings section, and in a macOS environment proxy information
361 is retrieved from the System Configuration Framework.
362
363 To disable autodetected proxy pass an empty dictionary.
364
365 The :envvar:`no_proxy` environment variable can be used to specify hosts
366 which shouldn't be reached via proxy; if set, it should be a comma-separated
367 list of hostname suffixes, optionally with ``:port`` appended, for example
368 ``cern.ch,ncsa.uiuc.edu,some.host:8080``.
369
370 .. note::
371
372 ``HTTP_PROXY`` will be ignored if a variable ``REQUEST_METHOD`` is set;
373 see the documentation on :func:`~urllib.request.getproxies`.
374
375
376 .. class:: HTTPPasswordMgr()
377
378 Keep a database of ``(realm, uri) -> (user, password)`` mappings.
379
380
381 .. class:: HTTPPasswordMgrWithDefaultRealm()
382
383 Keep a database of ``(realm, uri) -> (user, password)`` mappings. A realm of
384 ``None`` is considered a catch-all realm, which is searched if no other realm
385 fits.
386
387
388 .. class:: HTTPPasswordMgrWithPriorAuth()
389
390 A variant of :class:`HTTPPasswordMgrWithDefaultRealm` that also has a
391 database of ``uri -> is_authenticated`` mappings. Can be used by a
392 BasicAuth handler to determine when to send authentication credentials
393 immediately instead of waiting for a ``401`` response first.
394
395 .. versionadded:: 3.5
396
397
398 .. class:: AbstractBasicAuthHandler(password_mgr=None)
399
400 This is a mixin class that helps with HTTP authentication, both to the remote
401 host and to a proxy. *password_mgr*, if given, should be something that is
402 compatible with :class:`HTTPPasswordMgr`; refer to section
403 :ref:`http-password-mgr` for information on the interface that must be
404 supported. If *passwd_mgr* also provides ``is_authenticated`` and
405 ``update_authenticated`` methods (see
406 :ref:`http-password-mgr-with-prior-auth`), then the handler will use the
407 ``is_authenticated`` result for a given URI to determine whether or not to
408 send authentication credentials with the request. If ``is_authenticated``
409 returns ``True`` for the URI, credentials are sent. If ``is_authenticated``
410 is ``False``, credentials are not sent, and then if a ``401`` response is
411 received the request is re-sent with the authentication credentials. If
412 authentication succeeds, ``update_authenticated`` is called to set
413 ``is_authenticated`` ``True`` for the URI, so that subsequent requests to
414 the URI or any of its super-URIs will automatically include the
415 authentication credentials.
416
417 .. versionadded:: 3.5
418 Added ``is_authenticated`` support.
419
420
421 .. class:: HTTPBasicAuthHandler(password_mgr=None)
422
423 Handle authentication with the remote host. *password_mgr*, if given, should
424 be something that is compatible with :class:`HTTPPasswordMgr`; refer to
425 section :ref:`http-password-mgr` for information on the interface that must
426 be supported. HTTPBasicAuthHandler will raise a :exc:`ValueError` when
427 presented with a wrong Authentication scheme.
428
429
430 .. class:: ProxyBasicAuthHandler(password_mgr=None)
431
432 Handle authentication with the proxy. *password_mgr*, if given, should be
433 something that is compatible with :class:`HTTPPasswordMgr`; refer to section
434 :ref:`http-password-mgr` for information on the interface that must be
435 supported.
436
437
438 .. class:: AbstractDigestAuthHandler(password_mgr=None)
439
440 This is a mixin class that helps with HTTP authentication, both to the remote
441 host and to a proxy. *password_mgr*, if given, should be something that is
442 compatible with :class:`HTTPPasswordMgr`; refer to section
443 :ref:`http-password-mgr` for information on the interface that must be
444 supported.
445
446 .. versionchanged:: 3.14
447 Added support for HTTP digest authentication algorithm ``SHA-256``.
448
449
450 .. class:: HTTPDigestAuthHandler(password_mgr=None)
451
452 Handle authentication with the remote host. *password_mgr*, if given, should
453 be something that is compatible with :class:`HTTPPasswordMgr`; refer to
454 section :ref:`http-password-mgr` for information on the interface that must
455 be supported. When both Digest Authentication Handler and Basic
456 Authentication Handler are both added, Digest Authentication is always tried
457 first. If the Digest Authentication returns a 40x response again, it is sent
458 to Basic Authentication handler to Handle. This Handler method will raise a
459 :exc:`ValueError` when presented with an authentication scheme other than
460 Digest or Basic.
461
462 .. versionchanged:: 3.3
463 Raise :exc:`ValueError` on unsupported Authentication Scheme.
464
465
466
467 .. class:: ProxyDigestAuthHandler(password_mgr=None)
468
469 Handle authentication with the proxy. *password_mgr*, if given, should be
470 something that is compatible with :class:`HTTPPasswordMgr`; refer to section
471 :ref:`http-password-mgr` for information on the interface that must be
472 supported.
473
474
475 .. class:: HTTPHandler()
476
477 A class to handle opening of HTTP URLs.
478
479
480 .. class:: HTTPSHandler(debuglevel=0, context=None, check_hostname=None)
481
482 A class to handle opening of HTTPS URLs. *context* and *check_hostname*
483 have the same meaning as in :class:`http.client.HTTPSConnection`.
484
485 .. versionchanged:: 3.2
486 *context* and *check_hostname* were added.
487
488
489 .. class:: FileHandler()
490
491 Open local files.
492
493 .. class:: DataHandler()
494
495 Open data URLs.
496
497 .. versionadded:: 3.4
498
499 .. class:: FTPHandler()
500
501 Open FTP URLs.
502
503
504 .. class:: CacheFTPHandler()
505
506 Open FTP URLs, keeping a cache of open FTP connections to minimize delays.
507
508
509 .. class:: UnknownHandler()
510
511 A catch-all class to handle unknown URLs.
512
513
514 .. class:: HTTPErrorProcessor()
515
516 Process HTTP error responses.
517
518
519 .. _request-objects:
520
521 Request Objects
522 ---------------
523
524 The following methods describe :class:`Request`'s public interface,
525 and so all may be overridden in subclasses. It also defines several
526 public attributes that can be used by clients to inspect the parsed
527 request.
528
529 .. attribute:: Request.full_url
530
531 The original URL passed to the constructor.
532
533 .. versionchanged:: 3.4
534
535 Request.full_url is a property with setter, getter and a deleter. Getting
536 :attr:`~Request.full_url` returns the original request URL with the
537 fragment, if it was present.
538
539 .. attribute:: Request.type
540
541 The URI scheme.
542
543 .. attribute:: Request.host
544
545 The URI authority, typically a host, but may also contain a port
546 separated by a colon.
547
548 .. attribute:: Request.origin_req_host
549
550 The original host for the request, without port.
551
552 .. attribute:: Request.selector
553
554 The URI path. If the :class:`Request` uses a proxy, then selector
555 will be the full URL that is passed to the proxy.
556
557 .. attribute:: Request.data
558
559 The entity body for the request, or ``None`` if not specified.
560
561 .. versionchanged:: 3.4
562 Changing value of :attr:`Request.data` now deletes "Content-Length"
563 header if it was previously set or calculated.
564
565 .. attribute:: Request.unverifiable
566
567 boolean, indicates whether the request is unverifiable as defined
568 by :rfc:`2965`.
569
570 .. attribute:: Request.method
571
572 The HTTP request method to use. By default its value is :const:`None`,
573 which means that :meth:`~Request.get_method` will do its normal computation
574 of the method to be used. Its value can be set (thus overriding the default
575 computation in :meth:`~Request.get_method`) either by providing a default
576 value by setting it at the class level in a :class:`Request` subclass, or by
577 passing a value in to the :class:`Request` constructor via the *method*
578 argument.
579
580 .. versionadded:: 3.3
581
582 .. versionchanged:: 3.4
583 A default value can now be set in subclasses; previously it could only
584 be set via the constructor argument.
585
586
587 .. method:: Request.get_method()
588
589 Return a string indicating the HTTP request method. If
590 :attr:`Request.method` is not ``None``, return its value, otherwise return
591 ``'GET'`` if :attr:`Request.data` is ``None``, or ``'POST'`` if it's not.
592 This is only meaningful for HTTP requests.
593
594 .. versionchanged:: 3.3
595 get_method now looks at the value of :attr:`Request.method`.
596
597
598 .. method:: Request.add_header(key, val)
599
600 Add another header to the request. Headers are currently ignored by all
601 handlers except HTTP handlers, where they are added to the list of headers sent
602 to the server. Note that there cannot be more than one header with the same
603 name, and later calls will overwrite previous calls in case the *key* collides.
604 Currently, this is no loss of HTTP functionality, since all headers which have
605 meaning when used more than once have a (header-specific) way of gaining the
606 same functionality using only one header. Note that headers added using
607 this method are also added to redirected requests.
608
609
610 .. method:: Request.add_unredirected_header(key, header)
611
612 Add a header that will not be added to a redirected request.
613
614
615 .. method:: Request.has_header(header)
616
617 Return whether the instance has the named header (checks both regular and
618 unredirected).
619
620
621 .. method:: Request.remove_header(header)
622
623 Remove named header from the request instance (both from regular and
624 unredirected headers).
625
626 .. versionadded:: 3.4
627
628
629 .. method:: Request.get_full_url()
630
631 Return the URL given in the constructor.
632
633 .. versionchanged:: 3.4
634
635 Returns :attr:`Request.full_url`
636
637
638 .. method:: Request.set_proxy(host, type)
639
640 Prepare the request by connecting to a proxy server. The *host* and *type* will
641 replace those of the instance, and the instance's selector will be the original
642 URL given in the constructor.
643
644
645 .. method:: Request.get_header(header_name, default=None)
646
647 Return the value of the given header. If the header is not present, return
648 the default value.
649
650
651 .. method:: Request.header_items()
652
653 Return a list of tuples (header_name, header_value) of the Request headers.
654
655 .. versionchanged:: 3.4
656 The request methods add_data, has_data, get_data, get_type, get_host,
657 get_selector, get_origin_req_host and is_unverifiable that were deprecated
658 since 3.3 have been removed.
659
660
661 .. _opener-director-objects:
662
663 OpenerDirector Objects
664 ----------------------
665
666 :class:`OpenerDirector` instances have the following methods:
667
668
669 .. method:: OpenerDirector.add_handler(handler)
670
671 *handler* should be an instance of :class:`BaseHandler`. The following methods
672 are searched, and added to the possible chains (note that HTTP errors are a
673 special case). Note that, in the following, *protocol* should be replaced
674 with the actual protocol to handle, for example :meth:`http_response` would
675 be the HTTP protocol response handler. Also *type* should be replaced with
676 the actual HTTP code, for example :meth:`http_error_404` would handle HTTP
677 404 errors.
678
679 * :meth:`!<protocol>_open` --- signal that the handler knows how to open *protocol*
680 URLs.
681
682 See |protocol_open|_ for more information.
683
684 * :meth:`!http_error_\<type\>` --- signal that the handler knows how to handle HTTP
685 errors with HTTP error code *type*.
686
687 See |http_error_nnn|_ for more information.
688
689 * :meth:`!<protocol>_error` --- signal that the handler knows how to handle errors
690 from (non-\ ``http``) *protocol*.
691
692 * :meth:`!<protocol>_request` --- signal that the handler knows how to pre-process
693 *protocol* requests.
694
695 See |protocol_request|_ for more information.
696
697 * :meth:`!<protocol>_response` --- signal that the handler knows how to
698 post-process *protocol* responses.
699
700 See |protocol_response|_ for more information.
701
702 .. |protocol_open| replace:: :meth:`BaseHandler.<protocol>_open`
703 .. |http_error_nnn| replace:: :meth:`BaseHandler.http_error_\<nnn\>`
704 .. |protocol_request| replace:: :meth:`BaseHandler.<protocol>_request`
705 .. |protocol_response| replace:: :meth:`BaseHandler.<protocol>_response`
706
707 .. method:: OpenerDirector.open(url, data=None[, timeout])
708
709 Open the given *url* (which can be a request object or a string), optionally
710 passing the given *data*. Arguments, return values and exceptions raised are
711 the same as those of :func:`urlopen` (which simply calls the :meth:`open`
712 method on the currently installed global :class:`OpenerDirector`). The
713 optional *timeout* parameter specifies a timeout in seconds for blocking
714 operations like the connection attempt (if not specified, the global default
715 timeout setting will be used). The timeout feature actually works only for
716 HTTP, HTTPS and FTP connections.
717
718
719 .. method:: OpenerDirector.error(proto, *args)
720
721 Handle an error of the given protocol. This will call the registered error
722 handlers for the given protocol with the given arguments (which are protocol
723 specific). The HTTP protocol is a special case which uses the HTTP response
724 code to determine the specific error handler; refer to the :meth:`!http_error_\<type\>`
725 methods of the handler classes.
726
727 Return values and exceptions raised are the same as those of :func:`urlopen`.
728
729 OpenerDirector objects open URLs in three stages:
730
731 The order in which these methods are called within each stage is determined by
732 sorting the handler instances.
733
734 #. Every handler with a method named like :meth:`!<protocol>_request` has that
735 method called to pre-process the request.
736
737 #. Handlers with a method named like :meth:`!<protocol>_open` are called to handle
738 the request. This stage ends when a handler either returns a non-\ :const:`None`
739 value (ie. a response), or raises an exception (usually
740 :exc:`~urllib.error.URLError`). Exceptions are allowed to propagate.
741
742 In fact, the above algorithm is first tried for methods named
743 :meth:`~BaseHandler.default_open`. If all such methods return :const:`None`, the algorithm
744 is repeated for methods named like :meth:`!<protocol>_open`. If all such methods
745 return :const:`None`, the algorithm is repeated for methods named
746 :meth:`~BaseHandler.unknown_open`.
747
748 Note that the implementation of these methods may involve calls of the parent
749 :class:`OpenerDirector` instance's :meth:`~OpenerDirector.open` and
750 :meth:`~OpenerDirector.error` methods.
751
752 #. Every handler with a method named like :meth:`!<protocol>_response` has that
753 method called to post-process the response.
754
755
756 .. _base-handler-objects:
757
758 BaseHandler Objects
759 -------------------
760
761 :class:`BaseHandler` objects provide a couple of methods that are directly
762 useful, and others that are meant to be used by derived classes. These are
763 intended for direct use:
764
765
766 .. method:: BaseHandler.add_parent(director)
767
768 Add a director as parent.
769
770
771 .. method:: BaseHandler.close()
772
773 Remove any parents.
774
775 The following attribute and methods should only be used by classes derived from
776 :class:`BaseHandler`.
777
778 .. note::
779
780 The convention has been adopted that subclasses defining
781 :meth:`!<protocol>_request` or :meth:`!<protocol>_response` methods are named
782 :class:`!\*Processor`; all others are named :class:`!\*Handler`.
783
784
785 .. attribute:: BaseHandler.parent
786
787 A valid :class:`OpenerDirector`, which can be used to open using a different
788 protocol, or handle errors.
789
790
791 .. method:: BaseHandler.default_open(req)
792
793 This method is *not* defined in :class:`BaseHandler`, but subclasses should
794 define it if they want to catch all URLs.
795
796 This method, if implemented, will be called by the parent
797 :class:`OpenerDirector`. It should return a file-like object as described in
798 the return value of the :meth:`~OpenerDirector.open` method of :class:`OpenerDirector`, or ``None``.
799 It should raise :exc:`~urllib.error.URLError`, unless a truly exceptional
800 thing happens (for example, :exc:`MemoryError` should not be mapped to
801 :exc:`~urllib.error.URLError`).
802
803 This method will be called before any protocol-specific open method.
804
805
806 .. _protocol_open:
807 .. method:: BaseHandler.<protocol>_open(req)
808 :noindex:
809
810 This method is *not* defined in :class:`BaseHandler`, but subclasses should
811 define it if they want to handle URLs with the given protocol.
812
813 This method, if defined, will be called by the parent :class:`OpenerDirector`.
814 Return values should be the same as for :meth:`~BaseHandler.default_open`.
815
816
817 .. method:: BaseHandler.unknown_open(req)
818
819 This method is *not* defined in :class:`BaseHandler`, but subclasses should
820 define it if they want to catch all URLs with no specific registered handler to
821 open it.
822
823 This method, if implemented, will be called by the :attr:`parent`
824 :class:`OpenerDirector`. Return values should be the same as for
825 :meth:`default_open`.
826
827
828 .. method:: BaseHandler.http_error_default(req, fp, code, msg, hdrs)
829
830 This method is *not* defined in :class:`BaseHandler`, but subclasses should
831 override it if they intend to provide a catch-all for otherwise unhandled HTTP
832 errors. It will be called automatically by the :class:`OpenerDirector` getting
833 the error, and should not normally be called in other circumstances.
834
835 :class:`OpenerDirector` will call this method with five positional arguments:
836
837 1. a :class:`Request` object,
838 #. a file-like object with the HTTP error body,
839 #. the three-digit code of the error, as a string,
840 #. the user-visible explanation of the code, as a string, and
841 #. the headers of the error, as a mapping object.
842
843 Return values and exceptions raised should be the same as those of
844 :func:`urlopen`.
845
846
847 .. _http_error_nnn:
848 .. method:: BaseHandler.http_error_<nnn>(req, fp, code, msg, hdrs)
849
850 *nnn* should be a three-digit HTTP error code. This method is also not defined
851 in :class:`BaseHandler`, but will be called, if it exists, on an instance of a
852 subclass, when an HTTP error with code *nnn* occurs.
853
854 Subclasses should override this method to handle specific HTTP errors.
855
856 Arguments, return values and exceptions raised should be the same as for
857 :meth:`~BaseHandler.http_error_default`.
858
859
860 .. _protocol_request:
861 .. method:: BaseHandler.<protocol>_request(req)
862 :noindex:
863
864 This method is *not* defined in :class:`BaseHandler`, but subclasses should
865 define it if they want to pre-process requests of the given protocol.
866
867 This method, if defined, will be called by the parent :class:`OpenerDirector`.
868 *req* will be a :class:`Request` object. The return value should be a
869 :class:`Request` object.
870
871
872 .. _protocol_response:
873 .. method:: BaseHandler.<protocol>_response(req, response)
874 :noindex:
875
876 This method is *not* defined in :class:`BaseHandler`, but subclasses should
877 define it if they want to post-process responses of the given protocol.
878
879 This method, if defined, will be called by the parent :class:`OpenerDirector`.
880 *req* will be a :class:`Request` object. *response* will be an object
881 implementing the same interface as the return value of :func:`urlopen`. The
882 return value should implement the same interface as the return value of
883 :func:`urlopen`.
884
885
886 .. _http-redirect-handler:
887
888 HTTPRedirectHandler Objects
889 ---------------------------
890
891 .. note::
892
893 Some HTTP redirections require action from this module's client code. If this
894 is the case, :exc:`~urllib.error.HTTPError` is raised. See :rfc:`2616` for
895 details of the precise meanings of the various redirection codes.
896
897 An :exc:`~urllib.error.HTTPError` exception raised as a security consideration if the
898 HTTPRedirectHandler is presented with a redirected URL which is not an HTTP,
899 HTTPS or FTP URL.
900
901
902 .. method:: HTTPRedirectHandler.redirect_request(req, fp, code, msg, hdrs, newurl)
903
904 Return a :class:`Request` or ``None`` in response to a redirect. This is called
905 by the default implementations of the :meth:`!http_error_30\*` methods when a
906 redirection is received from the server. If a redirection should take place,
907 return a new :class:`Request` to allow :meth:`!http_error_30\*` to perform the
908 redirect to *newurl*. Otherwise, raise :exc:`~urllib.error.HTTPError` if
909 no other handler should try to handle this URL, or return ``None`` if you
910 can't but another handler might.
911
912 .. note::
913
914 The default implementation of this method does not strictly follow :rfc:`2616`,
915 which says that 301 and 302 responses to ``POST`` requests must not be
916 automatically redirected without confirmation by the user. In reality, browsers
917 do allow automatic redirection of these responses, changing the POST to a
918 ``GET``, and the default implementation reproduces this behavior.
919
920
921 .. method:: HTTPRedirectHandler.http_error_301(req, fp, code, msg, hdrs)
922
923 Redirect to the ``Location:`` or ``URI:`` URL. This method is called by the
924 parent :class:`OpenerDirector` when getting an HTTP 'moved permanently' response.
925
926
927 .. method:: HTTPRedirectHandler.http_error_302(req, fp, code, msg, hdrs)
928
929 The same as :meth:`http_error_301`, but called for the 'found' response.
930
931
932 .. method:: HTTPRedirectHandler.http_error_303(req, fp, code, msg, hdrs)
933
934 The same as :meth:`http_error_301`, but called for the 'see other' response.
935
936
937 .. method:: HTTPRedirectHandler.http_error_307(req, fp, code, msg, hdrs)
938
939 The same as :meth:`http_error_301`, but called for the 'temporary redirect'
940 response. It does not allow changing the request method from ``POST``
941 to ``GET``.
942
943
944 .. method:: HTTPRedirectHandler.http_error_308(req, fp, code, msg, hdrs)
945
946 The same as :meth:`http_error_301`, but called for the 'permanent redirect'
947 response. It does not allow changing the request method from ``POST``
948 to ``GET``.
949
950 .. versionadded:: 3.11
951
952
953 .. _http-cookie-processor:
954
955 HTTPCookieProcessor Objects
956 ---------------------------
957
958 :class:`HTTPCookieProcessor` instances have one attribute:
959
960 .. attribute:: HTTPCookieProcessor.cookiejar
961
962 The :class:`http.cookiejar.CookieJar` in which cookies are stored.
963
964
965 .. _proxy-handler:
966
967 ProxyHandler Objects
968 --------------------
969
970
971 .. method:: ProxyHandler.<protocol>_open(request)
972 :noindex:
973
974 The :class:`ProxyHandler` will have a method :meth:`!<protocol>_open` for every
975 *protocol* which has a proxy in the *proxies* dictionary given in the
976 constructor. The method will modify requests to go through the proxy, by
977 calling ``request.set_proxy()``, and call the next handler in the chain to
978 actually execute the protocol.
979
980
981 .. _http-password-mgr:
982
983 HTTPPasswordMgr Objects
984 -----------------------
985
986 These methods are available on :class:`HTTPPasswordMgr` and
987 :class:`HTTPPasswordMgrWithDefaultRealm` objects.
988
989
990 .. method:: HTTPPasswordMgr.add_password(realm, uri, user, passwd)
991
992 *uri* can be either a single URI, or a sequence of URIs. *realm*, *user* and
993 *passwd* must be strings. This causes ``(user, passwd)`` to be used as
994 authentication tokens when authentication for *realm* and a super-URI of any of
995 the given URIs is given.
996
997
998 .. method:: HTTPPasswordMgr.find_user_password(realm, authuri)
999
1000 Get user/password for given realm and URI, if any. This method will return
1001 ``(None, None)`` if there is no matching user/password.
1002
1003 For :class:`HTTPPasswordMgrWithDefaultRealm` objects, the realm ``None`` will be
1004 searched if the given *realm* has no matching user/password.
1005
1006
1007 .. _http-password-mgr-with-prior-auth:
1008
1009 HTTPPasswordMgrWithPriorAuth Objects
1010 ------------------------------------
1011
1012 This password manager extends :class:`HTTPPasswordMgrWithDefaultRealm` to support
1013 tracking URIs for which authentication credentials should always be sent.
1014
1015
1016 .. method:: HTTPPasswordMgrWithPriorAuth.add_password(realm, uri, user, \
1017 passwd, is_authenticated=False)
1018
1019 *realm*, *uri*, *user*, *passwd* are as for
1020 :meth:`HTTPPasswordMgr.add_password`. *is_authenticated* sets the initial
1021 value of the ``is_authenticated`` flag for the given URI or list of URIs.
1022 If *is_authenticated* is specified as ``True``, *realm* is ignored.
1023
1024
1025 .. method:: HTTPPasswordMgrWithPriorAuth.find_user_password(realm, authuri)
1026
1027 Same as for :class:`HTTPPasswordMgrWithDefaultRealm` objects
1028
1029
1030 .. method:: HTTPPasswordMgrWithPriorAuth.update_authenticated(self, uri, \
1031 is_authenticated=False)
1032
1033 Update the ``is_authenticated`` flag for the given *uri* or list
1034 of URIs.
1035
1036
1037 .. method:: HTTPPasswordMgrWithPriorAuth.is_authenticated(self, authuri)
1038
1039 Returns the current state of the ``is_authenticated`` flag for
1040 the given URI.
1041
1042
1043 .. _abstract-basic-auth-handler:
1044
1045 AbstractBasicAuthHandler Objects
1046 --------------------------------
1047
1048
1049 .. method:: AbstractBasicAuthHandler.http_error_auth_reqed(authreq, host, req, headers)
1050
1051 Handle an authentication request by getting a user/password pair, and re-trying
1052 the request. *authreq* should be the name of the header where the information
1053 about the realm is included in the request, *host* specifies the URL and path to
1054 authenticate for, *req* should be the (failed) :class:`Request` object, and
1055 *headers* should be the error headers.
1056
1057 *host* is either an authority (e.g. ``"python.org"``) or a URL containing an
1058 authority component (e.g. ``"http://python.org/"``). In either case, the
1059 authority must not contain a userinfo component (so, ``"python.org"`` and
1060 ``"python.org:80"`` are fine, ``"joe:[email protected]"`` is not).
1061
1062
1063 .. _http-basic-auth-handler:
1064
1065 HTTPBasicAuthHandler Objects
1066 ----------------------------
1067
1068
1069 .. method:: HTTPBasicAuthHandler.http_error_401(req, fp, code, msg, hdrs)
1070
1071 Retry the request with authentication information, if available.
1072
1073
1074 .. _proxy-basic-auth-handler:
1075
1076 ProxyBasicAuthHandler Objects
1077 -----------------------------
1078
1079
1080 .. method:: ProxyBasicAuthHandler.http_error_407(req, fp, code, msg, hdrs)
1081
1082 Retry the request with authentication information, if available.
1083
1084
1085 .. _abstract-digest-auth-handler:
1086
1087 AbstractDigestAuthHandler Objects
1088 ---------------------------------
1089
1090
1091 .. method:: AbstractDigestAuthHandler.http_error_auth_reqed(authreq, host, req, headers)
1092
1093 *authreq* should be the name of the header where the information about the realm
1094 is included in the request, *host* should be the host to authenticate to, *req*
1095 should be the (failed) :class:`Request` object, and *headers* should be the
1096 error headers.
1097
1098
1099 .. _http-digest-auth-handler:
1100
1101 HTTPDigestAuthHandler Objects
1102 -----------------------------
1103
1104
1105 .. method:: HTTPDigestAuthHandler.http_error_401(req, fp, code, msg, hdrs)
1106
1107 Retry the request with authentication information, if available.
1108
1109
1110 .. _proxy-digest-auth-handler:
1111
1112 ProxyDigestAuthHandler Objects
1113 ------------------------------
1114
1115
1116 .. method:: ProxyDigestAuthHandler.http_error_407(req, fp, code, msg, hdrs)
1117
1118 Retry the request with authentication information, if available.
1119
1120
1121 .. _http-handler-objects:
1122
1123 HTTPHandler Objects
1124 -------------------
1125
1126
1127 .. method:: HTTPHandler.http_open(req)
1128
1129 Send an HTTP request, which can be either GET or POST, depending on
1130 ``req.data``.
1131
1132
1133 .. _https-handler-objects:
1134
1135 HTTPSHandler Objects
1136 --------------------
1137
1138
1139 .. method:: HTTPSHandler.https_open(req)
1140
1141 Send an HTTPS request, which can be either GET or POST, depending on
1142 ``req.data``.
1143
1144
1145 .. _file-handler-objects:
1146
1147 FileHandler Objects
1148 -------------------
1149
1150
1151 .. method:: FileHandler.file_open(req)
1152
1153 Open the file locally, if there is no host name, or the host name is
1154 ``'localhost'``.
1155
1156 .. versionchanged:: 3.2
1157 This method is applicable only for local hostnames. When a remote
1158 hostname is given, a :exc:`~urllib.error.URLError` is raised.
1159
1160
1161 .. _data-handler-objects:
1162
1163 DataHandler Objects
1164 -------------------
1165
1166 .. method:: DataHandler.data_open(req)
1167
1168 Read a data URL. This kind of URL contains the content encoded in the URL
1169 itself. The data URL syntax is specified in :rfc:`2397`. This implementation
1170 ignores white spaces in base64 encoded data URLs so the URL may be wrapped
1171 in whatever source file it comes from. But even though some browsers don't
1172 mind about a missing padding at the end of a base64 encoded data URL, this
1173 implementation will raise a :exc:`ValueError` in that case.
1174
1175
1176 .. _ftp-handler-objects:
1177
1178 FTPHandler Objects
1179 ------------------
1180
1181
1182 .. method:: FTPHandler.ftp_open(req)
1183
1184 Open the FTP file indicated by *req*. The login is always done with empty
1185 username and password.
1186
1187
1188 .. _cacheftp-handler-objects:
1189
1190 CacheFTPHandler Objects
1191 -----------------------
1192
1193 :class:`CacheFTPHandler` objects are :class:`FTPHandler` objects with the
1194 following additional methods:
1195
1196
1197 .. method:: CacheFTPHandler.setTimeout(t)
1198
1199 Set timeout of connections to *t* seconds.
1200
1201
1202 .. method:: CacheFTPHandler.setMaxConns(m)
1203
1204 Set maximum number of cached connections to *m*.
1205
1206
1207 .. _unknown-handler-objects:
1208
1209 UnknownHandler Objects
1210 ----------------------
1211
1212
1213 .. method:: UnknownHandler.unknown_open()
1214
1215 Raise a :exc:`~urllib.error.URLError` exception.
1216
1217
1218 .. _http-error-processor-objects:
1219
1220 HTTPErrorProcessor Objects
1221 --------------------------
1222
1223 .. method:: HTTPErrorProcessor.http_response(request, response)
1224
1225 Process HTTP error responses.
1226
1227 For 200 error codes, the response object is returned immediately.
1228
1229 For non-200 error codes, this simply passes the job on to the
1230 :meth:`!http_error_\<type\>` handler methods, via :meth:`OpenerDirector.error`.
1231 Eventually, :class:`HTTPDefaultErrorHandler` will raise an
1232 :exc:`~urllib.error.HTTPError` if no other handler handles the error.
1233
1234
1235 .. method:: HTTPErrorProcessor.https_response(request, response)
1236
1237 Process HTTPS error responses.
1238
1239 The behavior is same as :meth:`http_response`.
1240
1241
1242 .. _urllib-request-examples:
1243
1244 Examples
1245 --------
1246
1247 In addition to the examples below, more examples are given in
1248 :ref:`urllib-howto`.
1249
1250 This example gets the python.org main page and displays the first 300 bytes of
1251 it::
1252
1253 >>> import urllib.request
1254 >>> with urllib.request.urlopen('http://www.python.org/') as f:
1255 ... print(f.read(300))
1256 ...
1257 b'<!doctype html>\n<!--[if lt IE 7]> <html class="no-js ie6 lt-ie7 lt-ie8 lt-ie9"> <![endif]-->\n<!--[if IE 7]> <html class="no-js ie7 lt-ie8 lt-ie9"> <![endif]-->\n<!--[if IE 8]> <html class="no-js ie8 lt-ie9">
1258
1259 Note that urlopen returns a bytes object. This is because there is no way
1260 for urlopen to automatically determine the encoding of the byte stream
1261 it receives from the HTTP server. In general, a program will decode
1262 the returned bytes object to string once it determines or guesses
1263 the appropriate encoding.
1264
1265 The following HTML spec document, https://html.spec.whatwg.org/#charset, lists
1266 the various ways in which an HTML or an XML document could have specified its
1267 encoding information.
1268
1269 For additional information, see the W3C document: https://www.w3.org/International/questions/qa-html-encoding-declarations.
1270
1271 As the python.org website uses *utf-8* encoding as specified in its meta tag, we
1272 will use the same for decoding the bytes object::
1273
1274 >>> with urllib.request.urlopen('http://www.python.org/') as f:
1275 ... print(f.read(100).decode('utf-8'))
1276 ...
1277 <!doctype html>
1278 <!--[if lt IE 7]> <html class="no-js ie6 lt-ie7 lt-ie8 lt-ie9"> <![endif]-->
1279 <!-
1280
1281 It is also possible to achieve the same result without using the
1282 :term:`context manager` approach::
1283
1284 >>> import urllib.request
1285 >>> f = urllib.request.urlopen('http://www.python.org/')
1286 >>> try:
1287 ... print(f.read(100).decode('utf-8'))
1288 ... finally:
1289 ... f.close()
1290 ...
1291 <!doctype html>
1292 <!--[if lt IE 7]> <html class="no-js ie6 lt-ie7 lt-ie8 lt-ie9"> <![endif]-->
1293 <!--
1294
1295 In the following example, we are sending a data-stream to the stdin of a CGI
1296 and reading the data it returns to us. Note that this example will only work
1297 when the Python installation supports SSL. ::
1298
1299 >>> import urllib.request
1300 >>> req = urllib.request.Request(url='https://localhost/cgi-bin/test.cgi',
1301 ... data=b'This data is passed to stdin of the CGI')
1302 >>> with urllib.request.urlopen(req) as f:
1303 ... print(f.read().decode('utf-8'))
1304 ...
1305 Got Data: "This data is passed to stdin of the CGI"
1306
1307 The code for the sample CGI used in the above example is::
1308
1309 #!/usr/bin/env python
1310 import sys
1311 data = sys.stdin.read()
1312 print('Content-type: text/plain\n\nGot Data: "%s"' % data)
1313
1314 Here is an example of doing a ``PUT`` request using :class:`Request`::
1315
1316 import urllib.request
1317 DATA = b'some data'
1318 req = urllib.request.Request(url='http://localhost:8080', data=DATA, method='PUT')
1319 with urllib.request.urlopen(req) as f:
1320 pass
1321 print(f.status)
1322 print(f.reason)
1323
1324 Use of Basic HTTP Authentication::
1325
1326 import urllib.request
1327 # Create an OpenerDirector with support for Basic HTTP Authentication...
1328 auth_handler = urllib.request.HTTPBasicAuthHandler()
1329 auth_handler.add_password(realm='PDQ Application',
1330 uri='https://mahler:8092/site-updates.py',
1331 user='klem',
1332 passwd='kadidd!ehopper')
1333 opener = urllib.request.build_opener(auth_handler)
1334 # ...and install it globally so it can be used with urlopen.
1335 urllib.request.install_opener(opener)
1336 with urllib.request.urlopen('http://www.example.com/login.html') as f:
1337 print(f.read().decode('utf-8'))
1338
1339 :func:`build_opener` provides many handlers by default, including a
1340 :class:`ProxyHandler`. By default, :class:`ProxyHandler` uses the environment
1341 variables named ``<scheme>_proxy``, where ``<scheme>`` is the URL scheme
1342 involved. For example, the :envvar:`!http_proxy` environment variable is read to
1343 obtain the HTTP proxy's URL.
1344
1345 This example replaces the default :class:`ProxyHandler` with one that uses
1346 programmatically supplied proxy URLs, and adds proxy authorization support with
1347 :class:`ProxyBasicAuthHandler`. ::
1348
1349 proxy_handler = urllib.request.ProxyHandler({'http': 'http://www.example.com:3128/'})
1350 proxy_auth_handler = urllib.request.ProxyBasicAuthHandler()
1351 proxy_auth_handler.add_password('realm', 'host', 'username', 'password')
1352
1353 opener = urllib.request.build_opener(proxy_handler, proxy_auth_handler)
1354 # This time, rather than install the OpenerDirector, we use it directly:
1355 with opener.open('http://www.example.com/login.html') as f:
1356 print(f.read().decode('utf-8'))
1357
1358 Adding HTTP headers:
1359
1360 Use the *headers* argument to the :class:`Request` constructor, or::
1361
1362 import urllib.request
1363 req = urllib.request.Request('http://www.example.com/')
1364 req.add_header('Referer', 'http://www.python.org/')
1365 # Customize the default User-Agent header value:
1366 req.add_header('User-Agent', 'urllib-example/0.1 (Contact: . . .)')
1367 with urllib.request.urlopen(req) as f:
1368 print(f.read().decode('utf-8'))
1369
1370
1371 :class:`OpenerDirector` automatically adds a :mailheader:`User-Agent` header to
1372 every :class:`Request`. To change this::
1373
1374 import urllib.request
1375 opener = urllib.request.build_opener()
1376 opener.addheaders = [('User-agent', 'Mozilla/5.0')]
1377 with opener.open('http://www.example.com/') as f:
1378 print(f.read().decode('utf-8'))
1379
1380 Also, remember that a few standard headers (:mailheader:`Content-Length`,
1381 :mailheader:`Content-Type` and :mailheader:`Host`)
1382 are added when the :class:`Request` is passed to :func:`urlopen` (or
1383 :meth:`OpenerDirector.open`).
1384
1385 .. _urllib-examples:
1386
1387 Here is an example session that uses the ``GET`` method to retrieve a URL
1388 containing parameters::
1389
1390 >>> import urllib.request
1391 >>> import urllib.parse
1392 >>> params = urllib.parse.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
1393 >>> url = "http://www.musi-cal.com/cgi-bin/query?%s" % params
1394 >>> with urllib.request.urlopen(url) as f:
1395 ... print(f.read().decode('utf-8'))
1396 ...
1397
1398 The following example uses the ``POST`` method instead. Note that params output
1399 from urlencode is encoded to bytes before it is sent to urlopen as data::
1400
1401 >>> import urllib.request
1402 >>> import urllib.parse
1403 >>> data = urllib.parse.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
1404 >>> data = data.encode('ascii')
1405 >>> with urllib.request.urlopen("http://requestb.in/xrbl82xr", data) as f:
1406 ... print(f.read().decode('utf-8'))
1407 ...
1408
1409 The following example uses an explicitly specified HTTP proxy, overriding
1410 environment settings::
1411
1412 >>> import urllib.request
1413 >>> proxies = {'http': 'http://proxy.example.com:8080/'}
1414 >>> opener = urllib.request.build_opener(urllib.request.ProxyHandler(proxies))
1415 >>> with opener.open("http://www.python.org") as f:
1416 ... f.read().decode('utf-8')
1417 ...
1418
1419 The following example uses no proxies at all, overriding environment settings::
1420
1421 >>> import urllib.request
1422 >>> opener = urllib.request.build_opener(urllib.request.ProxyHandler({}}))
1423 >>> with opener.open("http://www.python.org/") as f:
1424 ... f.read().decode('utf-8')
1425 ...
1426
1427
1428 Legacy interface
1429 ----------------
1430
1431 The following functions and classes are ported from the Python 2 module
1432 ``urllib`` (as opposed to ``urllib2``). They might become deprecated at
1433 some point in the future.
1434
1435 .. function:: urlretrieve(url, filename=None, reporthook=None, data=None)
1436
1437 Copy a network object denoted by a URL to a local file. If the URL
1438 points to a local file, the object will not be copied unless filename is supplied.
1439 Return a tuple ``(filename, headers)`` where *filename* is the
1440 local file name under which the object can be found, and *headers* is whatever
1441 the :meth:`!info` method of the object returned by :func:`urlopen` returned (for
1442 a remote object). Exceptions are the same as for :func:`urlopen`.
1443
1444 The second argument, if present, specifies the file location to copy to (if
1445 absent, the location will be a tempfile with a generated name). The third
1446 argument, if present, is a callable that will be called once on
1447 establishment of the network connection and once after each block read
1448 thereafter. The callable will be passed three arguments; a count of blocks
1449 transferred so far, a block size in bytes, and the total size of the file. The
1450 third argument may be ``-1`` on older FTP servers which do not return a file
1451 size in response to a retrieval request.
1452
1453 The following example illustrates the most common usage scenario::
1454
1455 >>> import urllib.request
1456 >>> local_filename, headers = urllib.request.urlretrieve('http://python.org/')
1457 >>> html = open(local_filename)
1458 >>> html.close()
1459
1460 If the *url* uses the :file:`http:` scheme identifier, the optional *data*
1461 argument may be given to specify a ``POST`` request (normally the request
1462 type is ``GET``). The *data* argument must be a bytes object in standard
1463 :mimetype:`application/x-www-form-urlencoded` format; see the
1464 :func:`urllib.parse.urlencode` function.
1465
1466 :func:`urlretrieve` will raise :exc:`~urllib.error.ContentTooShortError` when it detects that
1467 the amount of data available was less than the expected amount (which is the
1468 size reported by a *Content-Length* header). This can occur, for example, when
1469 the download is interrupted.
1470
1471 The *Content-Length* is treated as a lower bound: if there's more data to read,
1472 urlretrieve reads more data, but if less data is available, it raises the
1473 exception.
1474
1475 You can still retrieve the downloaded data in this case, it is stored in the
1476 :attr:`!content` attribute of the exception instance.
1477
1478 If no *Content-Length* header was supplied, urlretrieve can not check the size
1479 of the data it has downloaded, and just returns it. In this case you just have
1480 to assume that the download was successful.
1481
1482 .. function:: urlcleanup()
1483
1484 Cleans up temporary files that may have been left behind by previous
1485 calls to :func:`urlretrieve`.
1486
1487
1488 :mod:`!urllib.request` Restrictions
1489 -----------------------------------
1490
1491 .. index::
1492 pair: HTTP; protocol
1493 pair: FTP; protocol
1494
1495 * Currently, only the following protocols are supported: HTTP (versions 0.9 and
1496 1.0), FTP, local files, and data URLs.
1497
1498 .. versionchanged:: 3.4 Added support for data URLs.
1499
1500 * The caching feature of :func:`urlretrieve` has been disabled until someone
1501 finds the time to hack proper processing of Expiration time headers.
1502
1503 * There should be a function to query whether a particular URL is in the cache.
1504
1505 * For backward compatibility, if a URL appears to point to a local file but the
1506 file can't be opened, the URL is re-interpreted using the FTP protocol. This
1507 can sometimes cause confusing error messages.
1508
1509 * The :func:`urlopen` and :func:`urlretrieve` functions can cause arbitrarily
1510 long delays while waiting for a network connection to be set up. This means
1511 that it is difficult to build an interactive web client using these functions
1512 without using threads.
1513
1514 .. index::
1515 single: HTML
1516 pair: HTTP; protocol
1517
1518 * The data returned by :func:`urlopen` or :func:`urlretrieve` is the raw data
1519 returned by the server. This may be binary data (such as an image), plain text
1520 or (for example) HTML. The HTTP protocol provides type information in the reply
1521 header, which can be inspected by looking at the :mailheader:`Content-Type`
1522 header. If the returned data is HTML, you can use the module
1523 :mod:`html.parser` to parse it.
1524
1525 .. index:: single: FTP
1526
1527 * The code handling the FTP protocol cannot differentiate between a file and a
1528 directory. This can lead to unexpected behavior when attempting to read a URL
1529 that points to a file that is not accessible. If the URL ends in a ``/``, it is
1530 assumed to refer to a directory and will be handled accordingly. But if an
1531 attempt to read a file leads to a 550 error (meaning the URL cannot be found or
1532 is not accessible, often for permission reasons), then the path is treated as a
1533 directory in order to handle the case when a directory is specified by a URL but
1534 the trailing ``/`` has been left off. This can cause misleading results when
1535 you try to fetch a file whose read permissions make it inaccessible; the FTP
1536 code will try to read it, fail with a 550 error, and then perform a directory
1537 listing for the unreadable file. If fine-grained control is needed, consider
1538 using the :mod:`ftplib` module.
1539
1540
1541
1542 :mod:`!urllib.response` --- Response classes used by urllib
1543 ===========================================================
1544
1545 .. module:: urllib.response
1546 :synopsis: Response classes used by urllib.
1547
1548 The :mod:`!urllib.response` module defines functions and classes which define a
1549 minimal file-like interface, including ``read()`` and ``readline()``.
1550 Functions defined by this module are used internally by the :mod:`!urllib.request` module.
1551 The typical response object is a :class:`urllib.response.addinfourl` instance:
1552
1553 .. class:: addinfourl
1554
1555 .. attribute:: url
1556
1557 URL of the resource retrieved, commonly used to determine if a redirect was followed.
1558
1559 .. attribute:: headers
1560
1561 Returns the headers of the response in the form of an :class:`~email.message.EmailMessage` instance.
1562
1563 .. attribute:: status
1564
1565 .. versionadded:: 3.9
1566
1567 Status code returned by server.
1568
1569 .. method:: geturl()
1570
1571 .. deprecated:: 3.9
1572 Deprecated in favor of :attr:`~addinfourl.url`.
1573
1574 .. method:: info()
1575
1576 .. deprecated:: 3.9
1577 Deprecated in favor of :attr:`~addinfourl.headers`.
1578
1579 .. attribute:: code
1580
1581 .. deprecated:: 3.9
1582 Deprecated in favor of :attr:`~addinfourl.status`.
1583
1584 .. method:: getcode()
1585
1586 .. deprecated:: 3.9
1587 Deprecated in favor of :attr:`~addinfourl.status`.