| 1 |
:mod:`!urllib.parse` --- Parse URLs into components |
| 2 |
=================================================== |
| 3 |
|
| 4 |
.. module:: urllib.parse |
| 5 |
:synopsis: Parse URLs into or assemble them from components. |
| 6 |
|
| 7 |
**Source code:** :source:`Lib/urllib/parse.py` |
| 8 |
|
| 9 |
.. index:: |
| 10 |
single: WWW |
| 11 |
single: World Wide Web |
| 12 |
single: URL |
| 13 |
pair: URL; parsing |
| 14 |
pair: relative; URL |
| 15 |
|
| 16 |
-------------- |
| 17 |
|
| 18 |
This module defines a standard interface to break Uniform Resource Locator (URL) |
| 19 |
strings up in components (addressing scheme, network location, path etc.), to |
| 20 |
combine the components back into a URL string, and to convert a "relative URL" |
| 21 |
to an absolute URL given a "base URL." |
| 22 |
|
| 23 |
The module has been designed to match the internet RFC on Relative Uniform |
| 24 |
Resource Locators. It supports the following URL schemes: ``file``, ``ftp``, |
| 25 |
``gopher``, ``hdl``, ``http``, ``https``, ``imap``, ``itms-services``, ``mailto``, ``mms``, |
| 26 |
``news``, ``nntp``, ``prospero``, ``rsync``, ``rtsp``, ``rtsps``, ``rtspu``, |
| 27 |
``sftp``, ``shttp``, ``sip``, ``sips``, ``snews``, ``svn``, ``svn+ssh``, |
| 28 |
``telnet``, ``wais``, ``ws``, ``wss``. |
| 29 |
|
| 30 |
.. impl-detail:: |
| 31 |
|
| 32 |
The inclusion of the ``itms-services`` URL scheme can prevent an app from |
| 33 |
passing Apple's App Store review process for the macOS and iOS App Stores. |
| 34 |
Handling for the ``itms-services`` scheme is always removed on iOS; on |
| 35 |
macOS, it *may* be removed if CPython has been built with the |
| 36 |
:option:`--with-app-store-compliance` option. |
| 37 |
|
| 38 |
The :mod:`!urllib.parse` module defines functions that fall into two broad |
| 39 |
categories: URL parsing and URL quoting. These are covered in detail in |
| 40 |
the following sections. |
| 41 |
|
| 42 |
This module's functions use the deprecated term ``netloc`` (or ``net_loc``), |
| 43 |
which was introduced in :rfc:`1808`. However, this term has been obsoleted by |
| 44 |
:rfc:`3986`, which introduced the term ``authority`` as its replacement. |
| 45 |
The use of ``netloc`` is continued for backward compatibility. |
| 46 |
|
| 47 |
URL Parsing |
| 48 |
----------- |
| 49 |
|
| 50 |
The URL parsing functions focus on splitting a URL string into its components, |
| 51 |
or on combining URL components into a URL string. |
| 52 |
|
| 53 |
.. function:: urlsplit(urlstring, scheme=None, allow_fragments=True) |
| 54 |
|
| 55 |
Parse a URL into five components, returning a 5-item :term:`named tuple` |
| 56 |
:class:`SplitResult` or :class:`SplitResultBytes`. |
| 57 |
This corresponds to the general structure of a URL: |
| 58 |
``scheme://netloc/path?query#fragment``. |
| 59 |
Each tuple item is a string, possibly empty. The components are not broken up |
| 60 |
into smaller parts (for example, the network location is a single string), and % |
| 61 |
escapes are not expanded. The delimiters as shown above are not part of the |
| 62 |
result, except for a leading slash in the *path* component, which is retained if |
| 63 |
present. For example: |
| 64 |
|
| 65 |
.. doctest:: |
| 66 |
:options: +NORMALIZE_WHITESPACE |
| 67 |
|
| 68 |
>>> from urllib.parse import urlsplit |
| 69 |
>>> urlsplit("scheme://netloc/path?query#fragment") |
| 70 |
SplitResult(scheme='scheme', netloc='netloc', path='/path', |
| 71 |
query='query', fragment='fragment') |
| 72 |
>>> o = urlsplit("http://docs.python.org:80/3/library/urllib.parse.html?" |
| 73 |
... "highlight=params#url-parsing") |
| 74 |
>>> o |
| 75 |
SplitResult(scheme='http', netloc='docs.python.org:80', |
| 76 |
path='/3/library/urllib.parse.html', |
| 77 |
query='highlight=params', fragment='url-parsing') |
| 78 |
>>> o.scheme |
| 79 |
'http' |
| 80 |
>>> o.netloc |
| 81 |
'docs.python.org:80' |
| 82 |
>>> o.hostname |
| 83 |
'docs.python.org' |
| 84 |
>>> o.port |
| 85 |
80 |
| 86 |
>>> o._replace(fragment="").geturl() |
| 87 |
'http://docs.python.org:80/3/library/urllib.parse.html?highlight=params' |
| 88 |
|
| 89 |
Following the syntax specifications in :rfc:`1808`, :func:`!urlsplit` recognizes |
| 90 |
a netloc only if it is properly introduced by '//'. Otherwise the |
| 91 |
input is presumed to be a relative URL and thus to start with |
| 92 |
a path component. |
| 93 |
|
| 94 |
.. doctest:: |
| 95 |
:options: +NORMALIZE_WHITESPACE |
| 96 |
|
| 97 |
>>> from urllib.parse import urlsplit |
| 98 |
>>> urlsplit('//www.cwi.nl:80/%7Eguido/Python.html') |
| 99 |
SplitResult(scheme='', netloc='www.cwi.nl:80', path='/%7Eguido/Python.html', |
| 100 |
query='', fragment='') |
| 101 |
>>> urlsplit('www.cwi.nl/%7Eguido/Python.html') |
| 102 |
SplitResult(scheme='', netloc='', path='www.cwi.nl/%7Eguido/Python.html', |
| 103 |
query='', fragment='') |
| 104 |
>>> urlsplit('help/Python.html') |
| 105 |
SplitResult(scheme='', netloc='', path='help/Python.html', |
| 106 |
query='', fragment='') |
| 107 |
|
| 108 |
The *scheme* argument gives the default addressing scheme, to be |
| 109 |
used only if the URL does not specify one. It should be the same type |
| 110 |
(text or bytes) as *urlstring*, except that the default value ``''`` is |
| 111 |
always allowed, and is automatically converted to ``b''`` if appropriate. |
| 112 |
|
| 113 |
If the *allow_fragments* argument is false, fragment identifiers are not |
| 114 |
recognized. Instead, they are parsed as part of the path, parameters |
| 115 |
or query component, and :attr:`fragment` is set to the empty string in |
| 116 |
the return value. |
| 117 |
|
| 118 |
The return value is a :term:`named tuple`, which means that its items can |
| 119 |
be accessed by index or as named attributes, which are: |
| 120 |
|
| 121 |
+------------------+-------+-------------------------+------------------------+ |
| 122 |
| Attribute | Index | Value | Value if not present | |
| 123 |
+==================+=======+=========================+========================+ |
| 124 |
| :attr:`scheme` | 0 | URL scheme specifier | *scheme* parameter | |
| 125 |
+------------------+-------+-------------------------+------------------------+ |
| 126 |
| :attr:`netloc` | 1 | Network location part | empty string | |
| 127 |
+------------------+-------+-------------------------+------------------------+ |
| 128 |
| :attr:`path` | 2 | Hierarchical path | empty string | |
| 129 |
+------------------+-------+-------------------------+------------------------+ |
| 130 |
| :attr:`query` | 3 | Query component | empty string | |
| 131 |
+------------------+-------+-------------------------+------------------------+ |
| 132 |
| :attr:`fragment` | 4 | Fragment identifier | empty string | |
| 133 |
+------------------+-------+-------------------------+------------------------+ |
| 134 |
| :attr:`username` | | User name | :const:`None` | |
| 135 |
+------------------+-------+-------------------------+------------------------+ |
| 136 |
| :attr:`password` | | Password | :const:`None` | |
| 137 |
+------------------+-------+-------------------------+------------------------+ |
| 138 |
| :attr:`hostname` | | Host name (lower case) | :const:`None` | |
| 139 |
+------------------+-------+-------------------------+------------------------+ |
| 140 |
| :attr:`port` | | Port number as integer, | :const:`None` | |
| 141 |
| | | if present | | |
| 142 |
+------------------+-------+-------------------------+------------------------+ |
| 143 |
|
| 144 |
Reading the :attr:`port` attribute will raise a :exc:`ValueError` if |
| 145 |
an invalid port is specified in the URL. See section |
| 146 |
:ref:`urlparse-result-object` for more information on the result object. |
| 147 |
|
| 148 |
Unmatched square brackets in the :attr:`netloc` attribute will raise a |
| 149 |
:exc:`ValueError`. |
| 150 |
|
| 151 |
Characters in the :attr:`netloc` attribute that decompose under NFKC |
| 152 |
normalization (as used by the IDNA encoding) into any of ``/``, ``?``, |
| 153 |
``#``, ``@``, or ``:`` will raise a :exc:`ValueError`. If the URL is |
| 154 |
decomposed before parsing, no error will be raised. |
| 155 |
|
| 156 |
Following some of the `WHATWG spec`_ that updates :rfc:`3986`, leading C0 |
| 157 |
control and space characters are stripped from the URL. ``\n``, |
| 158 |
``\r`` and tab ``\t`` characters are removed from the URL at any position. |
| 159 |
|
| 160 |
As is the case with all named tuples, the subclass has a few additional methods |
| 161 |
and attributes that are particularly useful. One such method is :meth:`_replace`. |
| 162 |
The :meth:`_replace` method will return a new :class:`SplitResult` object |
| 163 |
replacing specified fields with new values. |
| 164 |
|
| 165 |
.. doctest:: |
| 166 |
:options: +NORMALIZE_WHITESPACE |
| 167 |
|
| 168 |
>>> from urllib.parse import urlsplit |
| 169 |
>>> u = urlsplit('//www.cwi.nl:80/%7Eguido/Python.html') |
| 170 |
>>> u |
| 171 |
SplitResult(scheme='', netloc='www.cwi.nl:80', path='/%7Eguido/Python.html', |
| 172 |
query='', fragment='') |
| 173 |
>>> u._replace(scheme='http') |
| 174 |
SplitResult(scheme='http', netloc='www.cwi.nl:80', path='/%7Eguido/Python.html', |
| 175 |
query='', fragment='') |
| 176 |
|
| 177 |
.. warning:: |
| 178 |
|
| 179 |
:func:`urlsplit` does not perform validation. See :ref:`URL parsing |
| 180 |
security <url-parsing-security>` for details. |
| 181 |
|
| 182 |
.. versionchanged:: 3.2 |
| 183 |
Added IPv6 URL parsing capabilities. |
| 184 |
|
| 185 |
.. versionchanged:: 3.3 |
| 186 |
The fragment is now parsed for all URL schemes (unless *allow_fragments* is |
| 187 |
false), in accordance with :rfc:`3986`. Previously, an allowlist of |
| 188 |
schemes that support fragments existed. |
| 189 |
|
| 190 |
.. versionchanged:: 3.6 |
| 191 |
Out-of-range port numbers now raise :exc:`ValueError`, instead of |
| 192 |
returning :const:`None`. |
| 193 |
|
| 194 |
.. versionchanged:: 3.8 |
| 195 |
Characters that affect netloc parsing under NFKC normalization will |
| 196 |
now raise :exc:`ValueError`. |
| 197 |
|
| 198 |
.. versionchanged:: 3.10 |
| 199 |
ASCII newline and tab characters are stripped from the URL. |
| 200 |
|
| 201 |
.. versionchanged:: 3.12 |
| 202 |
Leading WHATWG C0 control and space characters are stripped from the URL. |
| 203 |
|
| 204 |
.. _WHATWG spec: https://url.spec.whatwg.org/#concept-basic-url-parser |
| 205 |
|
| 206 |
|
| 207 |
.. function:: parse_qs(qs, keep_blank_values=False, strict_parsing=False, encoding='utf-8', errors='replace', max_num_fields=None, separator='&') |
| 208 |
|
| 209 |
Parse a query string given as a string argument (data of type |
| 210 |
:mimetype:`application/x-www-form-urlencoded`). Data are returned as a |
| 211 |
dictionary. The dictionary keys are the unique query variable names and the |
| 212 |
values are lists of values for each name. |
| 213 |
|
| 214 |
The optional argument *keep_blank_values* is a flag indicating whether blank |
| 215 |
values in percent-encoded queries should be treated as blank strings. A true value |
| 216 |
indicates that blanks should be retained as blank strings. The default false |
| 217 |
value indicates that blank values are to be ignored and treated as if they were |
| 218 |
not included. |
| 219 |
|
| 220 |
The optional argument *strict_parsing* is a flag indicating what to do with |
| 221 |
parsing errors. If false (the default), errors are silently ignored. If true, |
| 222 |
errors raise a :exc:`ValueError` exception. |
| 223 |
|
| 224 |
The optional *encoding* and *errors* parameters specify how to decode |
| 225 |
percent-encoded sequences into Unicode characters, as accepted by the |
| 226 |
:meth:`bytes.decode` method. |
| 227 |
|
| 228 |
The optional argument *max_num_fields* is the maximum number of fields to |
| 229 |
read. If set, then throws a :exc:`ValueError` if there are more than |
| 230 |
*max_num_fields* fields read. |
| 231 |
|
| 232 |
The optional argument *separator* is the symbol to use for separating the |
| 233 |
query arguments. It defaults to ``&``. |
| 234 |
|
| 235 |
Use the :func:`urllib.parse.urlencode` function (with the ``doseq`` |
| 236 |
parameter set to ``True``) to convert such dictionaries into query |
| 237 |
strings. |
| 238 |
|
| 239 |
|
| 240 |
.. versionchanged:: 3.2 |
| 241 |
Add *encoding* and *errors* parameters. |
| 242 |
|
| 243 |
.. versionchanged:: 3.8 |
| 244 |
Added *max_num_fields* parameter. |
| 245 |
|
| 246 |
.. versionchanged:: 3.10 |
| 247 |
Added *separator* parameter with the default value of ``&``. Python |
| 248 |
versions earlier than Python 3.10 allowed using both ``;`` and ``&`` as |
| 249 |
query parameter separator. This has been changed to allow only a single |
| 250 |
separator key, with ``&`` as the default separator. |
| 251 |
|
| 252 |
.. deprecated:: 3.14 |
| 253 |
Accepting objects with false values (like ``0`` and ``[]``) except empty |
| 254 |
strings and byte-like objects and ``None`` is now deprecated. |
| 255 |
|
| 256 |
|
| 257 |
.. function:: parse_qsl(qs, keep_blank_values=False, strict_parsing=False, encoding='utf-8', errors='replace', max_num_fields=None, separator='&') |
| 258 |
|
| 259 |
Parse a query string given as a string argument (data of type |
| 260 |
:mimetype:`application/x-www-form-urlencoded`). Data are returned as a list of |
| 261 |
name, value pairs. |
| 262 |
|
| 263 |
The optional argument *keep_blank_values* is a flag indicating whether blank |
| 264 |
values in percent-encoded queries should be treated as blank strings. A true value |
| 265 |
indicates that blanks should be retained as blank strings. The default false |
| 266 |
value indicates that blank values are to be ignored and treated as if they were |
| 267 |
not included. |
| 268 |
|
| 269 |
The optional argument *strict_parsing* is a flag indicating what to do with |
| 270 |
parsing errors. If false (the default), errors are silently ignored. If true, |
| 271 |
errors raise a :exc:`ValueError` exception. |
| 272 |
|
| 273 |
The optional *encoding* and *errors* parameters specify how to decode |
| 274 |
percent-encoded sequences into Unicode characters, as accepted by the |
| 275 |
:meth:`bytes.decode` method. |
| 276 |
|
| 277 |
The optional argument *max_num_fields* is the maximum number of fields to |
| 278 |
read. If set, then throws a :exc:`ValueError` if there are more than |
| 279 |
*max_num_fields* fields read. |
| 280 |
|
| 281 |
The optional argument *separator* is the symbol to use for separating the |
| 282 |
query arguments. It defaults to ``&``. |
| 283 |
|
| 284 |
Use the :func:`urllib.parse.urlencode` function to convert such lists of pairs into |
| 285 |
query strings. |
| 286 |
|
| 287 |
.. versionchanged:: 3.2 |
| 288 |
Add *encoding* and *errors* parameters. |
| 289 |
|
| 290 |
.. versionchanged:: 3.8 |
| 291 |
Added *max_num_fields* parameter. |
| 292 |
|
| 293 |
.. versionchanged:: 3.10 |
| 294 |
Added *separator* parameter with the default value of ``&``. Python |
| 295 |
versions earlier than Python 3.10 allowed using both ``;`` and ``&`` as |
| 296 |
query parameter separator. This has been changed to allow only a single |
| 297 |
separator key, with ``&`` as the default separator. |
| 298 |
|
| 299 |
|
| 300 |
.. function:: urlunsplit(parts) |
| 301 |
|
| 302 |
Construct a URL from a tuple as returned by ``urlsplit()``. The *parts* |
| 303 |
argument can be any five-item iterable. This may result in a slightly |
| 304 |
different, but equivalent URL, if the URL that was parsed originally had |
| 305 |
unnecessary delimiters (for example, a ``?`` with an empty query; the RFC |
| 306 |
states that these are equivalent). |
| 307 |
|
| 308 |
|
| 309 |
.. function:: urlparse(urlstring, scheme=None, allow_fragments=True) |
| 310 |
|
| 311 |
This is similar to :func:`urlsplit`, but additionally splits the *path* |
| 312 |
component on *path* and *params*. |
| 313 |
This function returns a 6-item :term:`named tuple` :class:`ParseResult` |
| 314 |
or :class:`ParseResultBytes`. |
| 315 |
Its items are the same as for the :func:`!urlsplit` result, except that |
| 316 |
*params* is inserted at index 3, between *path* and *query*. |
| 317 |
|
| 318 |
This function is based on obsoleted :rfc:`1738` and :rfc:`1808`, which |
| 319 |
listed *params* as the main URL component. |
| 320 |
The more recent URL syntax allows parameters to be applied to each segment |
| 321 |
of the *path* portion of the URL (see :rfc:`3986`). |
| 322 |
:func:`urlsplit` should generally be used instead of :func:`urlparse`. |
| 323 |
A separate function is needed to separate the path segments and parameters. |
| 324 |
|
| 325 |
.. function:: urlunparse(parts) |
| 326 |
|
| 327 |
Combine the elements of a tuple as returned by :func:`urlparse` into a |
| 328 |
complete URL as a string. The *parts* argument can be any six-item |
| 329 |
iterable. This may result in a slightly different, but equivalent URL, if the |
| 330 |
URL that was parsed originally had unnecessary delimiters (for example, a ? |
| 331 |
with an empty query; the RFC states that these are equivalent). |
| 332 |
|
| 333 |
|
| 334 |
.. function:: urljoin(base, url, allow_fragments=True) |
| 335 |
|
| 336 |
Construct a full ("absolute") URL by combining a "base URL" (*base*) with |
| 337 |
another URL (*url*). Informally, this uses components of the base URL, in |
| 338 |
particular the addressing scheme, the network location and (part of) the |
| 339 |
path, to provide missing components in the relative URL. For example: |
| 340 |
|
| 341 |
>>> from urllib.parse import urljoin |
| 342 |
>>> urljoin('http://www.cwi.nl/%7Eguido/Python.html', 'FAQ.html') |
| 343 |
'http://www.cwi.nl/%7Eguido/FAQ.html' |
| 344 |
|
| 345 |
The *allow_fragments* argument has the same meaning and default as for |
| 346 |
:func:`urlsplit`. |
| 347 |
|
| 348 |
.. note:: |
| 349 |
|
| 350 |
If *url* is an absolute URL (that is, it starts with ``//`` or ``scheme://``), |
| 351 |
the *url*'s hostname and/or scheme will be present in the result. For example: |
| 352 |
|
| 353 |
.. doctest:: |
| 354 |
|
| 355 |
>>> urljoin('http://www.cwi.nl/%7Eguido/Python.html', |
| 356 |
... '//www.python.org/%7Eguido') |
| 357 |
'http://www.python.org/%7Eguido' |
| 358 |
|
| 359 |
If you do not want that behavior, preprocess the *url* with :func:`urlsplit` and |
| 360 |
:func:`urlunsplit`, removing possible *scheme* and *netloc* parts. |
| 361 |
|
| 362 |
.. warning:: |
| 363 |
|
| 364 |
Because an absolute URL may be passed as the ``url`` parameter, it is |
| 365 |
generally **not secure** to use ``urljoin`` with an attacker-controlled |
| 366 |
``url``. For example in, |
| 367 |
``urljoin("https://website.com/users/", username)``, if ``username`` can |
| 368 |
contain an absolute URL, the result of ``urljoin`` will be the absolute |
| 369 |
URL. |
| 370 |
|
| 371 |
|
| 372 |
.. versionchanged:: 3.5 |
| 373 |
|
| 374 |
Behavior updated to match the semantics defined in :rfc:`3986`. |
| 375 |
|
| 376 |
|
| 377 |
.. function:: urldefrag(url) |
| 378 |
|
| 379 |
If *url* contains a fragment identifier, return a modified version of *url* |
| 380 |
with no fragment identifier, and the fragment identifier as a separate |
| 381 |
string. If there is no fragment identifier in *url*, return *url* unmodified |
| 382 |
and an empty string. |
| 383 |
|
| 384 |
The return value is a :term:`named tuple`, its items can be accessed by index |
| 385 |
or as named attributes: |
| 386 |
|
| 387 |
+------------------+-------+-------------------------+----------------------+ |
| 388 |
| Attribute | Index | Value | Value if not present | |
| 389 |
+==================+=======+=========================+======================+ |
| 390 |
| :attr:`url` | 0 | URL with no fragment | empty string | |
| 391 |
+------------------+-------+-------------------------+----------------------+ |
| 392 |
| :attr:`fragment` | 1 | Fragment identifier | empty string | |
| 393 |
+------------------+-------+-------------------------+----------------------+ |
| 394 |
|
| 395 |
See section :ref:`urlparse-result-object` for more information on the result |
| 396 |
object. |
| 397 |
|
| 398 |
.. versionchanged:: 3.2 |
| 399 |
Result is a structured object rather than a simple 2-tuple. |
| 400 |
|
| 401 |
.. function:: unwrap(url) |
| 402 |
|
| 403 |
Extract the url from a wrapped URL (that is, a string formatted as |
| 404 |
``<URL:scheme://host/path>``, ``<scheme://host/path>``, ``URL:scheme://host/path`` |
| 405 |
or ``scheme://host/path``). If *url* is not a wrapped URL, it is returned |
| 406 |
without changes. |
| 407 |
|
| 408 |
.. _url-parsing-security: |
| 409 |
|
| 410 |
URL parsing security |
| 411 |
-------------------- |
| 412 |
|
| 413 |
The :func:`urlsplit` and :func:`urlparse` APIs do not perform **validation** of |
| 414 |
inputs. They may not raise errors on inputs that other applications consider |
| 415 |
invalid. They may also succeed on some inputs that might not be considered |
| 416 |
URLs elsewhere. Their purpose is for practical functionality rather than |
| 417 |
purity. |
| 418 |
|
| 419 |
Instead of raising an exception on unusual input, they may instead return some |
| 420 |
component parts as empty strings. Or components may contain more than perhaps |
| 421 |
they should. |
| 422 |
|
| 423 |
We recommend that users of these APIs where the values may be used anywhere |
| 424 |
with security implications code defensively. Do some verification within your |
| 425 |
code before trusting a returned component part. Does that ``scheme`` make |
| 426 |
sense? Is that a sensible ``path``? Is there anything strange about that |
| 427 |
``hostname``? etc. |
| 428 |
|
| 429 |
What constitutes a URL is not universally well defined. Different applications |
| 430 |
have different needs and desired constraints. For instance the living `WHATWG |
| 431 |
spec`_ describes what user facing web clients such as a web browser require. |
| 432 |
While :rfc:`3986` is more general. These functions incorporate some aspects of |
| 433 |
both, but cannot be claimed compliant with either. The APIs and existing user |
| 434 |
code with expectations on specific behaviors predate both standards leading us |
| 435 |
to be very cautious about making API behavior changes. |
| 436 |
|
| 437 |
.. _parsing-ascii-encoded-bytes: |
| 438 |
|
| 439 |
Parsing ASCII Encoded Bytes |
| 440 |
--------------------------- |
| 441 |
|
| 442 |
The URL parsing functions were originally designed to operate on character |
| 443 |
strings only. In practice, it is useful to be able to manipulate properly |
| 444 |
quoted and encoded URLs as sequences of ASCII bytes. Accordingly, the |
| 445 |
URL parsing functions in this module all operate on :class:`bytes` and |
| 446 |
:class:`bytearray` objects in addition to :class:`str` objects. |
| 447 |
|
| 448 |
If :class:`str` data is passed in, the result will also contain only |
| 449 |
:class:`str` data. If :class:`bytes` or :class:`bytearray` data is |
| 450 |
passed in, the result will contain only :class:`bytes` data. |
| 451 |
|
| 452 |
Attempting to mix :class:`str` data with :class:`bytes` or |
| 453 |
:class:`bytearray` in a single function call will result in a |
| 454 |
:exc:`TypeError` being raised, while attempting to pass in non-ASCII |
| 455 |
byte values will trigger :exc:`UnicodeDecodeError`. |
| 456 |
|
| 457 |
To support easier conversion of result objects between :class:`str` and |
| 458 |
:class:`bytes`, all return values from URL parsing functions provide |
| 459 |
either an :meth:`encode` method (when the result contains :class:`str` |
| 460 |
data) or a :meth:`decode` method (when the result contains :class:`bytes` |
| 461 |
data). The signatures of these methods match those of the corresponding |
| 462 |
:class:`str` and :class:`bytes` methods (except that the default encoding |
| 463 |
is ``'ascii'`` rather than ``'utf-8'``). Each produces a value of a |
| 464 |
corresponding type that contains either :class:`bytes` data (for |
| 465 |
:meth:`encode` methods) or :class:`str` data (for |
| 466 |
:meth:`decode` methods). |
| 467 |
|
| 468 |
Applications that need to operate on potentially improperly quoted URLs |
| 469 |
that may contain non-ASCII data will need to do their own decoding from |
| 470 |
bytes to characters before invoking the URL parsing methods. |
| 471 |
|
| 472 |
The behaviour described in this section applies only to the URL parsing |
| 473 |
functions. The URL quoting functions use their own rules when producing |
| 474 |
or consuming byte sequences as detailed in the documentation of the |
| 475 |
individual URL quoting functions. |
| 476 |
|
| 477 |
.. versionchanged:: 3.2 |
| 478 |
URL parsing functions now accept ASCII encoded byte sequences |
| 479 |
|
| 480 |
|
| 481 |
.. _urlparse-result-object: |
| 482 |
|
| 483 |
Structured Parse Results |
| 484 |
------------------------ |
| 485 |
|
| 486 |
The result objects from the :func:`urlsplit`, :func:`urlparse` and |
| 487 |
:func:`urldefrag` functions are subclasses of the :class:`tuple` type. |
| 488 |
These subclasses add the attributes listed in the documentation for |
| 489 |
those functions, the encoding and decoding support described in the |
| 490 |
previous section, as well as an additional method: |
| 491 |
|
| 492 |
.. method:: urllib.parse.SplitResult.geturl() |
| 493 |
|
| 494 |
Return the re-combined version of the original URL as a string. This may |
| 495 |
differ from the original URL in that the scheme may be normalized to lower |
| 496 |
case and empty components may be dropped. Specifically, empty parameters, |
| 497 |
queries, and fragment identifiers will be removed. |
| 498 |
|
| 499 |
For :func:`urldefrag` results, only empty fragment identifiers will be removed. |
| 500 |
For :func:`urlsplit` and :func:`urlparse` results, all noted changes will be |
| 501 |
made to the URL returned by this method. |
| 502 |
|
| 503 |
The result of this method remains unchanged if passed back through the original |
| 504 |
parsing function: |
| 505 |
|
| 506 |
>>> from urllib.parse import urlsplit |
| 507 |
>>> url = 'HTTP://www.Python.org/doc/#' |
| 508 |
>>> r1 = urlsplit(url) |
| 509 |
>>> r1.geturl() |
| 510 |
'http://www.Python.org/doc/' |
| 511 |
>>> r2 = urlsplit(r1.geturl()) |
| 512 |
>>> r2.geturl() |
| 513 |
'http://www.Python.org/doc/' |
| 514 |
|
| 515 |
|
| 516 |
The following classes provide the implementations of the structured parse |
| 517 |
results when operating on :class:`str` objects: |
| 518 |
|
| 519 |
.. class:: DefragResult(url, fragment) |
| 520 |
|
| 521 |
Concrete class for :func:`urldefrag` results containing :class:`str` |
| 522 |
data. The :meth:`encode` method returns a :class:`DefragResultBytes` |
| 523 |
instance. |
| 524 |
|
| 525 |
.. versionadded:: 3.2 |
| 526 |
|
| 527 |
.. class:: ParseResult(scheme, netloc, path, params, query, fragment) |
| 528 |
|
| 529 |
Concrete class for :func:`urlparse` results containing :class:`str` |
| 530 |
data. The :meth:`encode` method returns a :class:`ParseResultBytes` |
| 531 |
instance. |
| 532 |
|
| 533 |
.. class:: SplitResult(scheme, netloc, path, query, fragment) |
| 534 |
|
| 535 |
Concrete class for :func:`urlsplit` results containing :class:`str` |
| 536 |
data. The :meth:`encode` method returns a :class:`SplitResultBytes` |
| 537 |
instance. |
| 538 |
|
| 539 |
|
| 540 |
The following classes provide the implementations of the parse results when |
| 541 |
operating on :class:`bytes` or :class:`bytearray` objects: |
| 542 |
|
| 543 |
.. class:: DefragResultBytes(url, fragment) |
| 544 |
|
| 545 |
Concrete class for :func:`urldefrag` results containing :class:`bytes` |
| 546 |
data. The :meth:`decode` method returns a :class:`DefragResult` |
| 547 |
instance. |
| 548 |
|
| 549 |
.. versionadded:: 3.2 |
| 550 |
|
| 551 |
.. class:: ParseResultBytes(scheme, netloc, path, params, query, fragment) |
| 552 |
|
| 553 |
Concrete class for :func:`urlparse` results containing :class:`bytes` |
| 554 |
data. The :meth:`decode` method returns a :class:`ParseResult` |
| 555 |
instance. |
| 556 |
|
| 557 |
.. versionadded:: 3.2 |
| 558 |
|
| 559 |
.. class:: SplitResultBytes(scheme, netloc, path, query, fragment) |
| 560 |
|
| 561 |
Concrete class for :func:`urlsplit` results containing :class:`bytes` |
| 562 |
data. The :meth:`decode` method returns a :class:`SplitResult` |
| 563 |
instance. |
| 564 |
|
| 565 |
.. versionadded:: 3.2 |
| 566 |
|
| 567 |
|
| 568 |
URL Quoting |
| 569 |
----------- |
| 570 |
|
| 571 |
The URL quoting functions focus on taking program data and making it safe |
| 572 |
for use as URL components by quoting special characters and appropriately |
| 573 |
encoding non-ASCII text. They also support reversing these operations to |
| 574 |
recreate the original data from the contents of a URL component if that |
| 575 |
task isn't already covered by the URL parsing functions above. |
| 576 |
|
| 577 |
.. function:: quote(string, safe='/', encoding=None, errors=None) |
| 578 |
|
| 579 |
Replace special characters in *string* using the :samp:`%{xx}` escape. Letters, |
| 580 |
digits, and the characters ``'_.-~'`` are never quoted. By default, this |
| 581 |
function is intended for quoting the path section of a URL. The optional |
| 582 |
*safe* parameter specifies additional ASCII characters that should not be |
| 583 |
quoted --- its default value is ``'/'``. |
| 584 |
|
| 585 |
*string* may be either a :class:`str` or a :class:`bytes` object. |
| 586 |
|
| 587 |
.. versionchanged:: 3.7 |
| 588 |
Moved from :rfc:`2396` to :rfc:`3986` for quoting URL strings. "~" is now |
| 589 |
included in the set of unreserved characters. |
| 590 |
|
| 591 |
The optional *encoding* and *errors* parameters specify how to deal with |
| 592 |
non-ASCII characters, as accepted by the :meth:`str.encode` method. |
| 593 |
*encoding* defaults to ``'utf-8'``. |
| 594 |
*errors* defaults to ``'strict'``, meaning unsupported characters raise a |
| 595 |
:class:`UnicodeEncodeError`. |
| 596 |
*encoding* and *errors* must not be supplied if *string* is a |
| 597 |
:class:`bytes`, or a :class:`TypeError` is raised. |
| 598 |
|
| 599 |
Note that ``quote(string, safe, encoding, errors)`` is equivalent to |
| 600 |
``quote_from_bytes(string.encode(encoding, errors), safe)``. |
| 601 |
|
| 602 |
Example: ``quote('/El Niño/')`` yields ``'/El%20Ni%C3%B1o/'``. |
| 603 |
|
| 604 |
|
| 605 |
.. function:: quote_plus(string, safe='', encoding=None, errors=None) |
| 606 |
|
| 607 |
Like :func:`quote`, but also replace spaces with plus signs, as required for |
| 608 |
quoting HTML form values when building up a query string to go into a URL. |
| 609 |
Plus signs in the original string are escaped unless they are included in |
| 610 |
*safe*. It also does not have *safe* default to ``'/'``. |
| 611 |
|
| 612 |
Example: ``quote_plus('/El Niño/')`` yields ``'%2FEl+Ni%C3%B1o%2F'``. |
| 613 |
|
| 614 |
|
| 615 |
.. function:: quote_from_bytes(bytes, safe='/') |
| 616 |
|
| 617 |
Like :func:`quote`, but accepts a :class:`bytes` object rather than a |
| 618 |
:class:`str`, and does not perform string-to-bytes encoding. |
| 619 |
|
| 620 |
Example: ``quote_from_bytes(b'a&\xef')`` yields |
| 621 |
``'a%26%EF'``. |
| 622 |
|
| 623 |
|
| 624 |
.. function:: unquote(string, encoding='utf-8', errors='replace') |
| 625 |
|
| 626 |
Replace :samp:`%{xx}` escapes with their single-character equivalent. |
| 627 |
The optional *encoding* and *errors* parameters specify how to decode |
| 628 |
percent-encoded sequences into Unicode characters, as accepted by the |
| 629 |
:meth:`bytes.decode` method. |
| 630 |
|
| 631 |
*string* may be either a :class:`str` or a :class:`bytes` object. |
| 632 |
|
| 633 |
*encoding* defaults to ``'utf-8'``. |
| 634 |
*errors* defaults to ``'replace'``, meaning invalid sequences are replaced |
| 635 |
by a placeholder character. |
| 636 |
|
| 637 |
Example: ``unquote('/El%20Ni%C3%B1o/')`` yields ``'/El Niño/'``. |
| 638 |
|
| 639 |
.. versionchanged:: 3.9 |
| 640 |
*string* parameter supports bytes and str objects (previously only str). |
| 641 |
|
| 642 |
|
| 643 |
|
| 644 |
|
| 645 |
.. function:: unquote_plus(string, encoding='utf-8', errors='replace') |
| 646 |
|
| 647 |
Like :func:`unquote`, but also replace plus signs with spaces, as required |
| 648 |
for unquoting HTML form values. |
| 649 |
|
| 650 |
*string* must be a :class:`str`. |
| 651 |
|
| 652 |
Example: ``unquote_plus('/El+Ni%C3%B1o/')`` yields ``'/El Niño/'``. |
| 653 |
|
| 654 |
|
| 655 |
.. function:: unquote_to_bytes(string) |
| 656 |
|
| 657 |
Replace :samp:`%{xx}` escapes with their single-octet equivalent, and return a |
| 658 |
:class:`bytes` object. |
| 659 |
|
| 660 |
*string* may be either a :class:`str` or a :class:`bytes` object. |
| 661 |
|
| 662 |
If it is a :class:`str`, unescaped non-ASCII characters in *string* |
| 663 |
are encoded into UTF-8 bytes. |
| 664 |
|
| 665 |
Example: ``unquote_to_bytes('a%26%EF')`` yields ``b'a&\xef'``. |
| 666 |
|
| 667 |
|
| 668 |
.. function:: urlencode(query, doseq=False, safe='', encoding=None, \ |
| 669 |
errors=None, quote_via=quote_plus) |
| 670 |
|
| 671 |
Convert a mapping object or a sequence of two-element tuples, which may |
| 672 |
contain :class:`str` or :class:`bytes` objects, to a percent-encoded ASCII |
| 673 |
text string. If the resultant string is to be used as a *data* for POST |
| 674 |
operation with the :func:`~urllib.request.urlopen` function, then |
| 675 |
it should be encoded to bytes, otherwise it would result in a |
| 676 |
:exc:`TypeError`. |
| 677 |
|
| 678 |
The resulting string is a series of ``key=value`` pairs separated by ``'&'`` |
| 679 |
characters, where both *key* and *value* are quoted using the *quote_via* |
| 680 |
function. By default, :func:`quote_plus` is used to quote the values, which |
| 681 |
means spaces are quoted as a ``'+'`` character and '/' characters are |
| 682 |
encoded as ``%2F``, which follows the standard for GET requests |
| 683 |
(``application/x-www-form-urlencoded``). An alternate function that can be |
| 684 |
passed as *quote_via* is :func:`quote`, which will encode spaces as ``%20`` |
| 685 |
and not encode '/' characters. For maximum control of what is quoted, use |
| 686 |
``quote`` and specify a value for *safe*. |
| 687 |
|
| 688 |
When a sequence of two-element tuples is used as the *query* |
| 689 |
argument, the first element of each tuple is a key and the second is a |
| 690 |
value. The value element in itself can be a sequence and in that case, if |
| 691 |
the optional parameter *doseq* evaluates to ``True``, individual |
| 692 |
``key=value`` pairs separated by ``'&'`` are generated for each element of |
| 693 |
the value sequence for the key. The order of parameters in the encoded |
| 694 |
string will match the order of parameter tuples in the sequence. |
| 695 |
|
| 696 |
The *safe*, *encoding*, and *errors* parameters are passed down to |
| 697 |
*quote_via* (the *encoding* and *errors* parameters are only passed |
| 698 |
when a query element is a :class:`str`). |
| 699 |
|
| 700 |
To reverse this encoding process, :func:`parse_qs` and :func:`parse_qsl` are |
| 701 |
provided in this module to parse query strings into Python data structures. |
| 702 |
|
| 703 |
Refer to :ref:`urllib examples <urllib-examples>` to find out how the |
| 704 |
:func:`urllib.parse.urlencode` method can be used for generating the query |
| 705 |
string of a URL or data for a POST request. |
| 706 |
|
| 707 |
.. versionchanged:: 3.2 |
| 708 |
*query* supports bytes and string objects. |
| 709 |
|
| 710 |
.. versionchanged:: 3.5 |
| 711 |
Added the *quote_via* parameter. |
| 712 |
|
| 713 |
.. deprecated:: 3.14 |
| 714 |
Accepting objects with false values (like ``0`` and ``[]``) except empty |
| 715 |
strings and byte-like objects and ``None`` is now deprecated. |
| 716 |
|
| 717 |
|
| 718 |
.. seealso:: |
| 719 |
|
| 720 |
`WHATWG`_ - URL Living standard |
| 721 |
Working Group for the URL Standard that defines URLs, domains, IP addresses, the |
| 722 |
application/x-www-form-urlencoded format, and their API. |
| 723 |
|
| 724 |
:rfc:`3986` - Uniform Resource Identifiers |
| 725 |
This is the current standard (STD66). Any changes to urllib.parse module |
| 726 |
should conform to this. Certain deviations could be observed, which are |
| 727 |
mostly for backward compatibility purposes and for certain de-facto |
| 728 |
parsing requirements as commonly observed in major browsers. |
| 729 |
|
| 730 |
:rfc:`2732` - Format for Literal IPv6 Addresses in URL's. |
| 731 |
This specifies the parsing requirements of IPv6 URLs. |
| 732 |
|
| 733 |
:rfc:`2396` - Uniform Resource Identifiers (URI): Generic Syntax |
| 734 |
Document describing the generic syntactic requirements for both Uniform Resource |
| 735 |
Names (URNs) and Uniform Resource Locators (URLs). |
| 736 |
|
| 737 |
:rfc:`2368` - The mailto URL scheme. |
| 738 |
Parsing requirements for mailto URL schemes. |
| 739 |
|
| 740 |
:rfc:`1808` - Relative Uniform Resource Locators |
| 741 |
This Request For Comments includes the rules for joining an absolute and a |
| 742 |
relative URL, including a fair number of "Abnormal Examples" which govern the |
| 743 |
treatment of border cases. |
| 744 |
|
| 745 |
:rfc:`1738` - Uniform Resource Locators (URL) |
| 746 |
This specifies the formal syntax and semantics of absolute URLs. |
| 747 |
|
| 748 |
.. _WHATWG: https://url.spec.whatwg.org/ |