Debian Codegraph @

Doc/reference/import.rst

raw
1
2 .. _importsystem:
3
4 *****************
5 The import system
6 *****************
7
8 .. index:: single: import machinery
9
10 Python code in one :term:`module` gains access to the code in another module
11 by the process of :term:`importing` it. The :keyword:`import` statement is
12 the most common way of invoking the import machinery, but it is not the only
13 way. Functions such as :func:`importlib.import_module` and built-in
14 :func:`__import__` can also be used to invoke the import machinery.
15
16 The :keyword:`import` statement combines two operations; it searches for the
17 named module, then it binds the results of that search to a name in the local
18 scope. The search operation of the :keyword:`!import` statement is defined as
19 a call to the :func:`__import__` function, with the appropriate arguments.
20 The return value of :func:`__import__` is used to perform the name
21 binding operation of the :keyword:`!import` statement. See the
22 :keyword:`!import` statement for the exact details of that name binding
23 operation.
24
25 A direct call to :func:`__import__` performs only the module search and, if
26 found, the module creation operation. While certain side-effects may occur,
27 such as the importing of parent packages, and the updating of various caches
28 (including :data:`sys.modules`), only the :keyword:`import` statement performs
29 a name binding operation.
30
31 When an :keyword:`import` statement is executed, the standard builtin
32 :func:`__import__` function is called. Other mechanisms for invoking the
33 import system (such as :func:`importlib.import_module`) may choose to bypass
34 :func:`__import__` and use their own solutions to implement import semantics.
35
36 When a module is first imported, Python searches for the module and if found,
37 it creates a module object [#fnmo]_, initializing it. If the named module
38 cannot be found, a :exc:`ModuleNotFoundError` is raised. Python implements various
39 strategies to search for the named module when the import machinery is
40 invoked. These strategies can be modified and extended by using various hooks
41 described in the sections below.
42
43 .. versionchanged:: 3.3
44 The import system has been updated to fully implement the second phase
45 of :pep:`302`. There is no longer any implicit import machinery - the full
46 import system is exposed through :data:`sys.meta_path`. In addition,
47 native namespace package support has been implemented (see :pep:`420`).
48
49
50 :mod:`importlib`
51 ================
52
53 The :mod:`importlib` module provides a rich API for interacting with the
54 import system. For example :func:`importlib.import_module` provides a
55 recommended, simpler API than built-in :func:`__import__` for invoking the
56 import machinery. Refer to the :mod:`importlib` library documentation for
57 additional detail.
58
59
60
61 Packages
62 ========
63
64 .. index::
65 single: package
66
67 Python has only one type of module object, and all modules are of this type,
68 regardless of whether the module is implemented in Python, C, or something
69 else. To help organize modules and provide a naming hierarchy, Python has a
70 concept of :term:`packages <package>`.
71
72 You can think of packages as the directories on a file system and modules as
73 files within directories, but don't take this analogy too literally since
74 packages and modules need not originate from the file system. For the
75 purposes of this documentation, we'll use this convenient analogy of
76 directories and files. Like file system directories, packages are organized
77 hierarchically, and packages may themselves contain subpackages, as well as
78 regular modules.
79
80 It's important to keep in mind that all packages are modules, but not all
81 modules are packages. Or put another way, packages are just a special kind of
82 module. Specifically, any module that contains a ``__path__`` attribute is
83 considered a package.
84
85 All modules have a name. Subpackage names are separated from their parent
86 package name by a dot, akin to Python's standard attribute access syntax. Thus
87 you might have a package called :mod:`email`, which in turn has a subpackage
88 called :mod:`email.mime` and a module within that subpackage called
89 :mod:`email.mime.text`.
90
91
92 Regular packages
93 ----------------
94
95 .. index::
96 pair: package; regular
97
98 Python defines two types of packages, :term:`regular packages <regular
99 package>` and :term:`namespace packages <namespace package>`. Regular
100 packages are traditional packages as they existed in Python 3.2 and earlier.
101 A regular package is typically implemented as a directory containing an
102 ``__init__.py`` file. When a regular package is imported, this
103 ``__init__.py`` file is implicitly executed, and the objects it defines are
104 bound to names in the package's namespace. The ``__init__.py`` file can
105 contain the same Python code that any other module can contain, and Python
106 will add some additional attributes to the module when it is imported.
107
108 For example, the following file system layout defines a top level ``parent``
109 package with three subpackages::
110
111 parent/
112 __init__.py
113 one/
114 __init__.py
115 two/
116 __init__.py
117 three/
118 __init__.py
119
120 Importing ``parent.one`` will implicitly execute ``parent/__init__.py`` and
121 ``parent/one/__init__.py``. Subsequent imports of ``parent.two`` or
122 ``parent.three`` will execute ``parent/two/__init__.py`` and
123 ``parent/three/__init__.py`` respectively.
124
125 A subdirectory inside a regular package that does not contain an
126 ``__init__.py`` file is treated as an implicit
127 :ref:`namespace package <reference-namespace-package>` (a "namespace
128 subpackage") rooted in that parent. See :pep:`420` for the underlying
129 specification.
130
131
132 .. _reference-namespace-package:
133
134 Namespace packages
135 ------------------
136
137 .. index::
138 pair: package; namespace
139 pair: package; portion
140
141 A namespace package is a composite of various :term:`portions <portion>`,
142 where each portion contributes a subpackage to the parent package. Portions
143 may reside in different locations on the file system. Portions may also be
144 found in zip files, on the network, or anywhere else that Python searches
145 during import. Namespace packages may or may not correspond directly to
146 objects on the file system; they may be virtual modules that have no concrete
147 representation.
148
149 Namespace packages do not use an ordinary list for their ``__path__``
150 attribute. They instead use a custom iterable type which will automatically
151 perform a new search for package portions on the next import attempt within
152 that package if the path of their parent package (or :data:`sys.path` for a
153 top level package) changes.
154
155 With namespace packages, there is no ``parent/__init__.py`` file. In fact,
156 there may be multiple ``parent`` directories found during import search, where
157 each one is provided by a different portion. Thus ``parent/one`` may not be
158 physically located next to ``parent/two``. In this case, Python will create a
159 namespace package for the top-level ``parent`` package whenever it or one of
160 its subpackages is imported.
161
162 Namespace packages may also be nested inside a regular package. When the
163 import system searches a regular package's ``__path__`` and encounters a
164 subdirectory that does not contain an ``__init__.py`` file, that
165 subdirectory becomes a :term:`portion` contributing to a namespace
166 subpackage of the enclosing regular package.
167
168 See also :pep:`420` for the namespace package specification.
169
170
171 Searching
172 =========
173
174 To begin the search, Python needs the :term:`fully qualified <qualified name>`
175 name of the module (or package, but for the purposes of this discussion, the
176 difference is immaterial) being imported. This name may come from various
177 arguments to the :keyword:`import` statement, or from the parameters to the
178 :func:`importlib.import_module` or :func:`__import__` functions.
179
180 This name will be used in various phases of the import search, and it may be
181 the dotted path to a submodule, e.g. ``foo.bar.baz``. In this case, Python
182 first tries to import ``foo``, then ``foo.bar``, and finally ``foo.bar.baz``.
183 If any of the intermediate imports fail, a :exc:`ModuleNotFoundError` is raised.
184
185
186 The module cache
187 ----------------
188
189 .. index::
190 single: sys.modules
191
192 The first place checked during import search is :data:`sys.modules`. This
193 mapping serves as a cache of all modules that have been previously imported,
194 including the intermediate paths. So if ``foo.bar.baz`` was previously
195 imported, :data:`sys.modules` will contain entries for ``foo``, ``foo.bar``,
196 and ``foo.bar.baz``. Each key will have as its value the corresponding module
197 object.
198
199 During import, the module name is looked up in :data:`sys.modules` and if
200 present, the associated value is the module satisfying the import, and the
201 process completes. However, if the value is ``None``, then a
202 :exc:`ModuleNotFoundError` is raised. If the module name is missing, Python will
203 continue searching for the module.
204
205 :data:`sys.modules` is writable. Deleting a key may not destroy the
206 associated module (as other modules may hold references to it),
207 but it will invalidate the cache entry for the named module, causing
208 Python to search anew for the named module upon its next
209 import. The key can also be assigned to ``None``, forcing the next import
210 of the module to result in a :exc:`ModuleNotFoundError`.
211
212 Beware though, as if you keep a reference to the module object,
213 invalidate its cache entry in :data:`sys.modules`, and then re-import the
214 named module, the two module objects will *not* be the same. By contrast,
215 :func:`importlib.reload` will reuse the *same* module object, and simply
216 reinitialise the module contents by rerunning the module's code.
217
218
219 .. _finders-and-loaders:
220
221 Finders and loaders
222 -------------------
223
224 .. index::
225 single: finder
226 single: loader
227 single: module spec
228
229 If the named module is not found in :data:`sys.modules`, then Python's import
230 protocol is invoked to find and load the module. This protocol consists of
231 two conceptual objects, :term:`finders <finder>` and :term:`loaders <loader>`.
232 A finder's job is to determine whether it can find the named module using
233 whatever strategy it knows about. Objects that implement both of these
234 interfaces are referred to as :term:`importers <importer>` - they return
235 themselves when they find that they can load the requested module.
236
237 Python includes a number of default finders and importers. The first one
238 knows how to locate built-in modules, and the second knows how to locate
239 frozen modules. A third default finder searches an :term:`import path`
240 for modules. The :term:`import path` is a list of locations that may
241 name file system paths or zip files. It can also be extended to search
242 for any locatable resource, such as those identified by URLs.
243
244 The import machinery is extensible, so new finders can be added to extend the
245 range and scope of module searching.
246
247 Finders do not actually load modules. If they can find the named module, they
248 return a :dfn:`module spec`, an encapsulation of the module's import-related
249 information, which the import machinery then uses when loading the module.
250
251 The following sections describe the protocol for finders and loaders in more
252 detail, including how you can create and register new ones to extend the
253 import machinery.
254
255 .. versionchanged:: 3.4
256 In previous versions of Python, finders returned :term:`loaders <loader>`
257 directly, whereas now they return module specs which *contain* loaders.
258 Loaders are still used during import but have fewer responsibilities.
259
260 Import hooks
261 ------------
262
263 .. index::
264 single: import hooks
265 single: meta hooks
266 single: path hooks
267 pair: hooks; import
268 pair: hooks; meta
269 pair: hooks; path
270
271 The import machinery is designed to be extensible; the primary mechanism for
272 this are the *import hooks*. There are two types of import hooks: *meta
273 hooks* and *import path hooks*.
274
275 Meta hooks are called at the start of import processing, before any other
276 import processing has occurred, other than :data:`sys.modules` cache look up.
277 This allows meta hooks to override :data:`sys.path` processing, frozen
278 modules, or even built-in modules. Meta hooks are registered by adding new
279 finder objects to :data:`sys.meta_path`, as described below.
280
281 Import path hooks are called as part of :data:`sys.path` (or
282 ``package.__path__``) processing, at the point where their associated path
283 item is encountered. Import path hooks are registered by adding new callables
284 to :data:`sys.path_hooks` as described below.
285
286
287 The meta path
288 -------------
289
290 .. index::
291 single: sys.meta_path
292 pair: finder; find_spec
293
294 When the named module is not found in :data:`sys.modules`, Python next
295 searches :data:`sys.meta_path`, which contains a list of meta path finder
296 objects. These finders are queried in order to see if they know how to handle
297 the named module. Meta path finders must implement a method called
298 :meth:`~importlib.abc.MetaPathFinder.find_spec` which takes three arguments:
299 a name, an import path, and (optionally) a target module. The meta path
300 finder can use any strategy it wants to determine whether it can handle
301 the named module or not.
302
303 If the meta path finder knows how to handle the named module, it returns a
304 spec object. If it cannot handle the named module, it returns ``None``. If
305 :data:`sys.meta_path` processing reaches the end of its list without returning
306 a spec, then a :exc:`ModuleNotFoundError` is raised. Any other exceptions
307 raised are simply propagated up, aborting the import process.
308
309 The :meth:`~importlib.abc.MetaPathFinder.find_spec` method of meta path
310 finders is called with two or three arguments. The first is the fully
311 qualified name of the module being imported, for example ``foo.bar.baz``.
312 The second argument is the path entries to use for the module search. For
313 top-level modules, the second argument is ``None``, but for submodules or
314 subpackages, the second argument is the value of the parent package's
315 ``__path__`` attribute. If the appropriate ``__path__`` attribute cannot
316 be accessed, a :exc:`ModuleNotFoundError` is raised. The third argument
317 is an existing module object that will be the target of loading later.
318 The import system passes in a target module only during reload.
319
320 The meta path may be traversed multiple times for a single import request.
321 For example, assuming none of the modules involved has already been cached,
322 importing ``foo.bar.baz`` will first perform a top level import, calling
323 ``mpf.find_spec("foo", None, None)`` on each meta path finder (``mpf``). After
324 ``foo`` has been imported, ``foo.bar`` will be imported by traversing the
325 meta path a second time, calling
326 ``mpf.find_spec("foo.bar", foo.__path__, None)``. Once ``foo.bar`` has been
327 imported, the final traversal will call
328 ``mpf.find_spec("foo.bar.baz", foo.bar.__path__, None)``.
329
330 Some meta path finders only support top level imports. These importers will
331 always return ``None`` when anything other than ``None`` is passed as the
332 second argument.
333
334 Python's default :data:`sys.meta_path` has three meta path finders, one that
335 knows how to import built-in modules, one that knows how to import frozen
336 modules, and one that knows how to import modules from an :term:`import path`
337 (i.e. the :term:`path based finder`).
338
339 .. versionchanged:: 3.4
340 The :meth:`~importlib.abc.MetaPathFinder.find_spec` method of meta path
341 finders replaced :meth:`!find_module`, which
342 is now deprecated. While it will continue to work without change, the
343 import machinery will try it only if the finder does not implement
344 :meth:`~importlib.abc.MetaPathFinder.find_spec`.
345
346 .. versionchanged:: 3.10
347 Use of :meth:`!find_module` by the import system
348 now raises :exc:`ImportWarning`.
349
350 .. versionchanged:: 3.12
351 :meth:`!find_module` has been removed.
352 Use :meth:`~importlib.abc.MetaPathFinder.find_spec` instead.
353
354
355 Loading
356 =======
357
358 If and when a module spec is found, the import machinery will use it (and
359 the loader it contains) when loading the module. Here is an approximation
360 of what happens during the loading portion of import::
361
362 module = None
363 if spec.loader is not None and hasattr(spec.loader, 'create_module'):
364 # It is assumed 'exec_module' will also be defined on the loader.
365 module = spec.loader.create_module(spec)
366 if module is None:
367 module = ModuleType(spec.name)
368 # The import-related module attributes get set here:
369 _init_module_attrs(spec, module)
370
371 if spec.loader is None:
372 # unsupported
373 raise ImportError
374 if spec.origin is None and spec.submodule_search_locations is not None:
375 # namespace package
376 sys.modules[spec.name] = module
377 elif not hasattr(spec.loader, 'exec_module'):
378 module = spec.loader.load_module(spec.name)
379 else:
380 sys.modules[spec.name] = module
381 try:
382 spec.loader.exec_module(module)
383 except BaseException:
384 try:
385 del sys.modules[spec.name]
386 except KeyError:
387 pass
388 raise
389 return sys.modules[spec.name]
390
391 Note the following details:
392
393 * If there is an existing module object with the given name in
394 :data:`sys.modules`, import will have already returned it.
395
396 * The module will exist in :data:`sys.modules` before the loader
397 executes the module code. This is crucial because the module code may
398 (directly or indirectly) import itself; adding it to :data:`sys.modules`
399 beforehand prevents unbounded recursion in the worst case and multiple
400 loading in the best.
401
402 * If loading fails, the failing module -- and only the failing module --
403 gets removed from :data:`sys.modules`. Any module already in the
404 :data:`sys.modules` cache, and any module that was successfully loaded
405 as a side-effect, must remain in the cache. This contrasts with
406 reloading where even the failing module is left in :data:`sys.modules`.
407
408 * After the module is created but before execution, the import machinery
409 sets the import-related module attributes ("_init_module_attrs" in
410 the pseudo-code example above), as summarized in a
411 :ref:`later section <import-mod-attrs>`.
412
413 * Module execution is the key moment of loading in which the module's
414 namespace gets populated. Execution is entirely delegated to the
415 loader, which gets to decide what gets populated and how.
416
417 * The module created during loading and passed to exec_module() may
418 not be the one returned at the end of import [#fnlo]_.
419
420 .. versionchanged:: 3.4
421 The import system has taken over the boilerplate responsibilities of
422 loaders. These were previously performed by the
423 :meth:`importlib.abc.Loader.load_module` method.
424
425 Loaders
426 -------
427
428 Module loaders provide the critical function of loading: module execution.
429 The import machinery calls the :meth:`importlib.abc.Loader.exec_module`
430 method with a single argument, the module object to execute. Any value
431 returned from :meth:`~importlib.abc.Loader.exec_module` is ignored.
432
433 Loaders must satisfy the following requirements:
434
435 * If the module is a Python module (as opposed to a built-in module or a
436 dynamically loaded extension), the loader should execute the module's code
437 in the module's global name space (``module.__dict__``).
438
439 * If the loader cannot execute the module, it should raise an
440 :exc:`ImportError`, although any other exception raised during
441 :meth:`~importlib.abc.Loader.exec_module` will be propagated.
442
443 In many cases, the finder and loader can be the same object; in such cases the
444 :meth:`~importlib.abc.MetaPathFinder.find_spec` method would just return a
445 spec with the loader set to ``self``.
446
447 Module loaders may opt in to creating the module object during loading
448 by implementing a :meth:`~importlib.abc.Loader.create_module` method.
449 It takes one argument, the module spec, and returns the new module object
450 to use during loading. ``create_module()`` does not need to set any attributes
451 on the module object. If the method returns ``None``, the
452 import machinery will create the new module itself.
453
454 .. versionadded:: 3.4
455 The :meth:`~importlib.abc.Loader.create_module` method of loaders.
456
457 .. versionchanged:: 3.4
458 The :meth:`~importlib.abc.Loader.load_module` method was replaced by
459 :meth:`~importlib.abc.Loader.exec_module` and the import
460 machinery assumed all the boilerplate responsibilities of loading.
461
462 For compatibility with existing loaders, the import machinery will use
463 the ``load_module()`` method of loaders if it exists and the loader does
464 not also implement ``exec_module()``. However, ``load_module()`` has been
465 deprecated and loaders should implement ``exec_module()`` instead.
466
467 The ``load_module()`` method must implement all the boilerplate loading
468 functionality described above in addition to executing the module. All
469 the same constraints apply, with some additional clarification:
470
471 * If there is an existing module object with the given name in
472 :data:`sys.modules`, the loader must use that existing module.
473 (Otherwise, :func:`importlib.reload` will not work correctly.) If the
474 named module does not exist in :data:`sys.modules`, the loader
475 must create a new module object and add it to :data:`sys.modules`.
476
477 * The module *must* exist in :data:`sys.modules` before the loader
478 executes the module code, to prevent unbounded recursion or multiple
479 loading.
480
481 * If loading fails, the loader must remove any modules it has inserted
482 into :data:`sys.modules`, but it must remove **only** the failing
483 module(s), and only if the loader itself has loaded the module(s)
484 explicitly.
485
486 .. versionchanged:: 3.5
487 A :exc:`DeprecationWarning` is raised when ``exec_module()`` is defined but
488 ``create_module()`` is not.
489
490 .. versionchanged:: 3.6
491 An :exc:`ImportError` is raised when ``exec_module()`` is defined but
492 ``create_module()`` is not.
493
494 .. versionchanged:: 3.10
495 Use of ``load_module()`` will raise :exc:`ImportWarning`.
496
497 Submodules
498 ----------
499
500 When a submodule is loaded using any mechanism (e.g. ``importlib`` APIs, the
501 ``import`` or ``import-from`` statements, or built-in ``__import__()``) a
502 binding is placed in the parent module's namespace to the submodule object.
503 For example, if package ``spam`` has a submodule ``foo``, after importing
504 ``spam.foo``, ``spam`` will have an attribute ``foo`` which is bound to the
505 submodule. Let's say you have the following directory structure::
506
507 spam/
508 __init__.py
509 foo.py
510
511 and ``spam/__init__.py`` has the following line in it::
512
513 from .foo import Foo
514
515 then executing the following puts name bindings for ``foo`` and ``Foo`` in the
516 ``spam`` module::
517
518 >>> import spam
519 >>> spam.foo
520 <module 'spam.foo' from '/tmp/imports/spam/foo.py'>
521 >>> spam.Foo
522 <class 'spam.foo.Foo'>
523
524 Given Python's familiar name binding rules this might seem surprising, but
525 it's actually a fundamental feature of the import system. The invariant
526 holding is that if you have ``sys.modules['spam']`` and
527 ``sys.modules['spam.foo']`` (as you would after the above import), the latter
528 must appear as the ``foo`` attribute of the former.
529
530 .. _module-specs:
531
532 Module specs
533 ------------
534
535 The import machinery uses a variety of information about each module
536 during import, especially before loading. Most of the information is
537 common to all modules. The purpose of a module's spec is to encapsulate
538 this import-related information on a per-module basis.
539
540 Using a spec during import allows state to be transferred between import
541 system components, e.g. between the finder that creates the module spec
542 and the loader that executes it. Most importantly, it allows the
543 import machinery to perform the boilerplate operations of loading,
544 whereas without a module spec the loader had that responsibility.
545
546 The module's spec is exposed as :attr:`module.__spec__`. Setting
547 :attr:`!__spec__` appropriately applies equally to
548 :ref:`modules initialized during interpreter startup <programs>`.
549 The one exception is ``__main__``, where :attr:`!__spec__` is
550 :ref:`set to None in some cases <main_spec>`.
551
552 See :class:`~importlib.machinery.ModuleSpec` for details on the contents of
553 the module spec.
554
555 .. versionadded:: 3.4
556
557 .. _package-path-rules:
558
559 __path__ attributes on modules
560 ------------------------------
561
562 The :attr:`~module.__path__` attribute should be a (possibly empty)
563 :term:`sequence` of strings enumerating the locations where the package's
564 submodules will be found. By definition, if a module has a :attr:`!__path__`
565 attribute, it is a :term:`package`.
566
567 A package's :attr:`~module.__path__` attribute is used during imports of its
568 subpackages.
569 Within the import machinery, it functions much the same as :data:`sys.path`,
570 i.e. providing a list of locations to search for modules during import.
571 However, :attr:`!__path__` is typically much more constrained than
572 :data:`!sys.path`.
573
574 The same rules used for :data:`sys.path` also apply to a package's
575 :attr:`!__path__`. :data:`sys.path_hooks` (described below) are
576 consulted when traversing a package's :attr:`!__path__`.
577
578 A package's ``__init__.py`` file may set or alter the package's
579 :attr:`~module.__path__`
580 attribute, and this was typically the way namespace packages were implemented
581 prior to :pep:`420`. With the adoption of :pep:`420`, namespace packages no
582 longer need to supply ``__init__.py`` files containing only :attr:`!__path__`
583 manipulation code; the import machinery automatically sets :attr:`!__path__`
584 correctly for the namespace package.
585
586 Module reprs
587 ------------
588
589 By default, all modules have a usable repr, however depending on the
590 attributes set above, and in the module's spec, you can more explicitly
591 control the repr of module objects.
592
593 If the module has a spec (``__spec__``), the import machinery will try
594 to generate a repr from it. If that fails or there is no spec, the import
595 system will craft a default repr using whatever information is available
596 on the module. It will try to use the ``module.__name__``,
597 ``module.__file__``, and ``module.__loader__`` as input into the repr,
598 with defaults for whatever information is missing.
599
600 Here are the exact rules used:
601
602 * If the module has a ``__spec__`` attribute, the information in the spec
603 is used to generate the repr. The "name", "loader", "origin", and
604 "has_location" attributes are consulted.
605
606 * If the module has a ``__file__`` attribute, this is used as part of the
607 module's repr.
608
609 * If the module has no ``__file__`` but does have a ``__loader__`` that is not
610 ``None``, then the loader's repr is used as part of the module's repr.
611
612 * Otherwise, just use the module's ``__name__`` in the repr.
613
614 .. versionchanged:: 3.12
615 Use of :meth:`!module_repr`, having been deprecated since Python 3.4, was
616 removed in Python 3.12 and is no longer called during the resolution of a
617 module's repr.
618
619 .. _pyc-invalidation:
620
621 Cached bytecode invalidation
622 ----------------------------
623
624 Before Python loads cached bytecode from a ``.pyc`` file, it checks whether the
625 cache is up-to-date with the source ``.py`` file. By default, Python does this
626 by storing the source's last-modified timestamp and size in the cache file when
627 writing it. At runtime, the import system then validates the cache file by
628 checking the stored metadata in the cache file against the source's
629 metadata.
630
631 Python also supports "hash-based" cache files, which store a hash of the source
632 file's contents rather than its metadata. There are two variants of hash-based
633 ``.pyc`` files: checked and unchecked. For checked hash-based ``.pyc`` files,
634 Python validates the cache file by hashing the source file and comparing the
635 resulting hash with the hash in the cache file. If a checked hash-based cache
636 file is found to be invalid, Python regenerates it and writes a new checked
637 hash-based cache file. For unchecked hash-based ``.pyc`` files, Python simply
638 assumes the cache file is valid if it exists. Hash-based ``.pyc`` files
639 validation behavior may be overridden with the :option:`--check-hash-based-pycs`
640 flag.
641
642 .. versionchanged:: 3.7
643 Added hash-based ``.pyc`` files. Previously, Python only supported
644 timestamp-based invalidation of bytecode caches.
645
646
647 The Path Based Finder
648 =====================
649
650 .. index::
651 single: path based finder
652
653 As mentioned previously, Python comes with several default meta path finders.
654 One of these, called the :term:`path based finder`
655 (:class:`~importlib.machinery.PathFinder`), searches an :term:`import path`,
656 which contains a list of :term:`path entries <path entry>`. Each path
657 entry names a location to search for modules.
658
659 The path based finder itself doesn't know how to import anything. Instead, it
660 traverses the individual path entries, associating each of them with a
661 path entry finder that knows how to handle that particular kind of path.
662
663 The default set of path entry finders implement all the semantics for finding
664 modules on the file system, handling special file types such as Python source
665 code (``.py`` files), Python byte code (``.pyc`` files) and
666 shared libraries (e.g. ``.so`` files). When supported by the :mod:`zipimport`
667 module in the standard library, the default path entry finders also handle
668 loading all of these file types (other than shared libraries) from zipfiles.
669
670 Path entries need not be limited to file system locations. They can refer to
671 URLs, database queries, or any other location that can be specified as a
672 string.
673
674 The path based finder provides additional hooks and protocols so that you
675 can extend and customize the types of searchable path entries. For example,
676 if you wanted to support path entries as network URLs, you could write a hook
677 that implements HTTP semantics to find modules on the web. This hook (a
678 callable) would return a :term:`path entry finder` supporting the protocol
679 described below, which was then used to get a loader for the module from the
680 web.
681
682 A word of warning: this section and the previous both use the term *finder*,
683 distinguishing between them by using the terms :term:`meta path finder` and
684 :term:`path entry finder`. These two types of finders are very similar,
685 support similar protocols, and function in similar ways during the import
686 process, but it's important to keep in mind that they are subtly different.
687 In particular, meta path finders operate at the beginning of the import
688 process, as keyed off the :data:`sys.meta_path` traversal.
689
690 By contrast, path entry finders are in a sense an implementation detail
691 of the path based finder, and in fact, if the path based finder were to be
692 removed from :data:`sys.meta_path`, none of the path entry finder semantics
693 would be invoked.
694
695
696 Path entry finders
697 ------------------
698
699 .. index::
700 single: sys.path
701 single: sys.path_hooks
702 single: sys.path_importer_cache
703 single: PYTHONPATH
704
705 The :term:`path based finder` is responsible for finding and loading
706 Python modules and packages whose location is specified with a string
707 :term:`path entry`. Most path entries name locations in the file system,
708 but they need not be limited to this.
709
710 As a meta path finder, the :term:`path based finder` implements the
711 :meth:`~importlib.abc.MetaPathFinder.find_spec` protocol previously
712 described, however it exposes additional hooks that can be used to
713 customize how modules are found and loaded from the :term:`import path`.
714
715 Three variables are used by the :term:`path based finder`, :data:`sys.path`,
716 :data:`sys.path_hooks` and :data:`sys.path_importer_cache`. The ``__path__``
717 attributes on package objects are also used. These provide additional ways
718 that the import machinery can be customized.
719
720 :data:`sys.path` contains a list of strings providing search locations for
721 modules and packages. It is initialized from the :envvar:`PYTHONPATH`
722 environment variable and various other installation- and
723 implementation-specific defaults. Entries in :data:`sys.path` can name
724 directories on the file system, zip files, and potentially other "locations"
725 (see the :mod:`site` module) that should be searched for modules, such as
726 URLs, or database queries. Only strings should be present on
727 :data:`sys.path`; all other data types are ignored.
728
729 The :term:`path based finder` is a :term:`meta path finder`, so the import
730 machinery begins the :term:`import path` search by calling the path
731 based finder's :meth:`~importlib.machinery.PathFinder.find_spec` method as
732 described previously. When the ``path`` argument to
733 :meth:`~importlib.machinery.PathFinder.find_spec` is given, it will be a
734 list of string paths to traverse - typically a package's ``__path__``
735 attribute for an import within that package. If the ``path`` argument is
736 ``None``, this indicates a top level import and :data:`sys.path` is used.
737
738 The path based finder iterates over every entry in the search path, and
739 for each of these, looks for an appropriate :term:`path entry finder`
740 (:class:`~importlib.abc.PathEntryFinder`) for the
741 path entry. Because this can be an expensive operation (e.g. there may be
742 ``stat()`` call overheads for this search), the path based finder maintains
743 a cache mapping path entries to path entry finders. This cache is maintained
744 in :data:`sys.path_importer_cache` (despite the name, this cache actually
745 stores finder objects rather than being limited to :term:`importer` objects).
746 In this way, the expensive search for a particular :term:`path entry`
747 location's :term:`path entry finder` need only be done once. User code is
748 free to remove cache entries from :data:`sys.path_importer_cache` forcing
749 the path based finder to perform the path entry search again.
750
751 If the path entry is not present in the cache, the path based finder iterates
752 over every callable in :data:`sys.path_hooks`. Each of the :term:`path entry
753 hooks <path entry hook>` in this list is called with a single argument, the
754 path entry to be searched. This callable may either return a :term:`path
755 entry finder` that can handle the path entry, or it may raise
756 :exc:`ImportError`. An :exc:`ImportError` is used by the path based finder to
757 signal that the hook cannot find a :term:`path entry finder`
758 for that :term:`path entry`. The
759 exception is ignored and :term:`import path` iteration continues. The hook
760 should expect either a string or bytes object; the encoding of bytes objects
761 is up to the hook (e.g. it may be a file system encoding, UTF-8, or something
762 else), and if the hook cannot decode the argument, it should raise
763 :exc:`ImportError`.
764
765 If :data:`sys.path_hooks` iteration ends with no :term:`path entry finder`
766 being returned, then the path based finder's
767 :meth:`~importlib.machinery.PathFinder.find_spec` method will store ``None``
768 in :data:`sys.path_importer_cache` (to indicate that there is no finder for
769 this path entry) and return ``None``, indicating that this
770 :term:`meta path finder` could not find the module.
771
772 If a :term:`path entry finder` *is* returned by one of the :term:`path entry
773 hook` callables on :data:`sys.path_hooks`, then the following protocol is used
774 to ask the finder for a module spec, which is then used when loading the
775 module.
776
777 The current working directory -- denoted by an empty string -- is handled
778 slightly differently from other entries on :data:`sys.path`. First, if the
779 current working directory cannot be determined or is found not to exist, no
780 value is stored in :data:`sys.path_importer_cache`. Second, the value for the
781 current working directory is looked up fresh for each module lookup. Third,
782 the path used for :data:`sys.path_importer_cache` and returned by
783 :meth:`importlib.machinery.PathFinder.find_spec` will be the actual current
784 working directory and not the empty string.
785
786 Path entry finder protocol
787 --------------------------
788
789 In order to support imports of modules and initialized packages and also to
790 contribute portions to namespace packages, path entry finders must implement
791 the :meth:`~importlib.abc.PathEntryFinder.find_spec` method.
792
793 :meth:`~importlib.abc.PathEntryFinder.find_spec` takes two arguments: the
794 fully qualified name of the module being imported, and the (optional) target
795 module. ``find_spec()`` returns a fully populated spec for the module.
796 This spec will always have "loader" set (with one exception).
797
798 To indicate to the import machinery that the spec represents a namespace
799 :term:`portion`, the path entry finder sets ``submodule_search_locations`` to
800 a list containing the portion.
801
802 .. versionchanged:: 3.4
803 :meth:`~importlib.abc.PathEntryFinder.find_spec` replaced
804 :meth:`!find_loader` and
805 :meth:`!find_module`, both of which
806 are now deprecated, but will be used if ``find_spec()`` is not defined.
807
808 Older path entry finders may implement one of these two deprecated methods
809 instead of ``find_spec()``. The methods are still respected for the
810 sake of backward compatibility. However, if ``find_spec()`` is
811 implemented on the path entry finder, the legacy methods are ignored.
812
813 :meth:`!find_loader` takes one argument, the
814 fully qualified name of the module being imported. ``find_loader()``
815 returns a 2-tuple where the first item is the loader and the second item
816 is a namespace :term:`portion`.
817
818 For backwards compatibility with other implementations of the import
819 protocol, many path entry finders also support the same,
820 traditional ``find_module()`` method that meta path finders support.
821 However path entry finder ``find_module()`` methods are never called
822 with a ``path`` argument (they are expected to record the appropriate
823 path information from the initial call to the path hook).
824
825 The ``find_module()`` method on path entry finders is deprecated,
826 as it does not allow the path entry finder to contribute portions to
827 namespace packages. If both ``find_loader()`` and ``find_module()``
828 exist on a path entry finder, the import system will always call
829 ``find_loader()`` in preference to ``find_module()``.
830
831 .. versionchanged:: 3.10
832 Calls to :meth:`!find_module` and
833 :meth:`!find_loader` by the import
834 system will raise :exc:`ImportWarning`.
835
836 .. versionchanged:: 3.12
837 ``find_module()`` and ``find_loader()`` have been removed.
838
839
840 Replacing the standard import system
841 ====================================
842
843 The most reliable mechanism for replacing the entire import system is to
844 delete the default contents of :data:`sys.meta_path`, replacing them
845 entirely with a custom meta path hook.
846
847 If it is acceptable to only alter the behaviour of import statements
848 without affecting other APIs that access the import system, then replacing
849 the builtin :func:`__import__` function may be sufficient.
850
851 To selectively prevent the import of some modules from a hook early on the
852 meta path (rather than disabling the standard import system entirely),
853 it is sufficient to raise :exc:`ModuleNotFoundError` directly from
854 :meth:`~importlib.abc.MetaPathFinder.find_spec` instead of returning
855 ``None``. The latter indicates that the meta path search should continue,
856 while raising an exception terminates it immediately.
857
858 .. _relativeimports:
859
860 Package Relative Imports
861 ========================
862
863 Relative imports use leading dots. A single leading dot indicates a relative
864 import, starting with the current package. Two or more leading dots indicate a
865 relative import to the parent(s) of the current package, one level per dot
866 after the first. For example, given the following package layout::
867
868 package/
869 __init__.py
870 subpackage1/
871 __init__.py
872 moduleX.py
873 moduleY.py
874 subpackage2/
875 __init__.py
876 moduleZ.py
877 moduleA.py
878
879 In either ``subpackage1/moduleX.py`` or ``subpackage1/__init__.py``,
880 the following are valid relative imports::
881
882 from .moduleY import spam
883 from .moduleY import spam as ham
884 from . import moduleY
885 from ..subpackage1 import moduleY
886 from ..subpackage2.moduleZ import eggs
887 from ..moduleA import foo
888
889 Absolute imports may use either the ``import <>`` or ``from <> import <>``
890 syntax, but relative imports may only use the second form; the reason
891 for this is that::
892
893 import XXX.YYY.ZZZ
894
895 should expose ``XXX.YYY.ZZZ`` as a usable expression, but .moduleY is
896 not a valid expression.
897
898
899 .. _import-dunder-main:
900
901 Special considerations for __main__
902 ===================================
903
904 The :mod:`__main__` module is a special case relative to Python's import
905 system. As noted :ref:`elsewhere <programs>`, the ``__main__`` module
906 is directly initialized at interpreter startup, much like :mod:`sys` and
907 :mod:`builtins`. However, unlike those two, it doesn't strictly
908 qualify as a built-in module. This is because the manner in which
909 ``__main__`` is initialized depends on the flags and other options with
910 which the interpreter is invoked.
911
912 .. _main_spec:
913
914 __main__.__spec__
915 -----------------
916
917 Depending on how :mod:`__main__` is initialized, ``__main__.__spec__``
918 gets set appropriately or to ``None``.
919
920 When Python is started with the :option:`-m` option, ``__spec__`` is set
921 to the module spec of the corresponding module or package. ``__spec__`` is
922 also populated when the ``__main__`` module is loaded as part of executing a
923 directory, zipfile or other :data:`sys.path` entry.
924
925 In :ref:`the remaining cases <using-on-interface-options>`
926 ``__main__.__spec__`` is set to ``None``, as the code used to populate the
927 :mod:`__main__` does not correspond directly with an importable module:
928
929 - interactive prompt
930 - :option:`-c` option
931 - running from stdin
932 - running directly from a source or bytecode file
933
934 Note that ``__main__.__spec__`` is always ``None`` in the last case,
935 *even if* the file could technically be imported directly as a module
936 instead. Use the :option:`-m` switch if valid module metadata is desired
937 in :mod:`__main__`.
938
939 Note also that even when ``__main__`` corresponds with an importable module
940 and ``__main__.__spec__`` is set accordingly, they're still considered
941 *distinct* modules. This is due to the fact that blocks guarded by
942 ``if __name__ == "__main__":`` checks only execute when the module is used
943 to populate the ``__main__`` namespace, and not during normal import.
944
945
946 References
947 ==========
948
949 The import machinery has evolved considerably since Python's early days. The
950 original `specification for packages
951 <https://www.python.org/doc/essays/packages/>`_ is still available to read,
952 although some details have changed since the writing of that document.
953
954 The original specification for :data:`sys.meta_path` was :pep:`302`, with
955 subsequent extension in :pep:`420`.
956
957 :pep:`420` introduced :term:`namespace packages <namespace package>` for
958 Python 3.3. :pep:`420` also introduced the :meth:`!find_loader` protocol as an
959 alternative to :meth:`!find_module`.
960
961 :pep:`366` describes the addition of the ``__package__`` attribute for
962 explicit relative imports in main modules.
963
964 :pep:`328` introduced absolute and explicit relative imports and initially
965 proposed ``__name__`` for semantics :pep:`366` would eventually specify for
966 ``__package__``.
967
968 :pep:`338` defines executing modules as scripts.
969
970 :pep:`451` adds the encapsulation of per-module import state in spec
971 objects. It also off-loads most of the boilerplate responsibilities of
972 loaders back onto the import machinery. These changes allow the
973 deprecation of several APIs in the import system and also addition of new
974 methods to finders and loaders.
975
976 .. rubric:: Footnotes
977
978 .. [#fnmo] See :class:`types.ModuleType`.
979
980 .. [#fnlo] The importlib implementation avoids using the return value
981 directly. Instead, it gets the module object by looking the module name up
982 in :data:`sys.modules`. The indirect effect of this is that an imported
983 module may replace itself in :data:`sys.modules`. This is
984 implementation-specific behavior that is not guaranteed to work in other
985 Python implementations.