Debian Codegraph @

Doc/reference/import.rst

raw
1
2 .. _importsystem:
3
4 *****************
5 The import system
6 *****************
7
8 .. index:: single: import machinery
9
10 Python code in one :term:`module` gains access to the code in another module
11 by the process of :term:`importing` it. The :keyword:`import` statement is
12 the most common way of invoking the import machinery, but it is not the only
13 way. Functions such as :func:`importlib.import_module` and built-in
14 :func:`__import__` can also be used to invoke the import machinery.
15
16 The :keyword:`import` statement combines two operations; it searches for the
17 named module, then it binds the results of that search to a name in the local
18 scope. The search operation of the :keyword:`!import` statement is defined as
19 a call to the :func:`__import__` function, with the appropriate arguments.
20 The return value of :func:`__import__` is used to perform the name
21 binding operation of the :keyword:`!import` statement. See the
22 :keyword:`!import` statement for the exact details of that name binding
23 operation.
24
25 A direct call to :func:`__import__` performs only the module search and, if
26 found, the module creation operation. While certain side-effects may occur,
27 such as the importing of parent packages, and the updating of various caches
28 (including :data:`sys.modules`), only the :keyword:`import` statement performs
29 a name binding operation.
30
31 When an :keyword:`import` statement is executed, the standard builtin
32 :func:`__import__` function is called. Other mechanisms for invoking the
33 import system (such as :func:`importlib.import_module`) may choose to bypass
34 :func:`__import__` and use their own solutions to implement import semantics.
35
36 When a module is first imported, Python searches for the module and if found,
37 it creates a module object [#fnmo]_, initializing it. If the named module
38 cannot be found, a :exc:`ModuleNotFoundError` is raised. Python implements various
39 strategies to search for the named module when the import machinery is
40 invoked. These strategies can be modified and extended by using various hooks
41 described in the sections below.
42
43 .. versionchanged:: 3.3
44 The import system has been updated to fully implement the second phase
45 of :pep:`302`. There is no longer any implicit import machinery - the full
46 import system is exposed through :data:`sys.meta_path`. In addition,
47 native namespace package support has been implemented (see :pep:`420`).
48
49
50 :mod:`importlib`
51 ================
52
53 The :mod:`importlib` module provides a rich API for interacting with the
54 import system. For example :func:`importlib.import_module` provides a
55 recommended, simpler API than built-in :func:`__import__` for invoking the
56 import machinery. Refer to the :mod:`importlib` library documentation for
57 additional detail.
58
59
60
61 Packages
62 ========
63
64 .. index::
65 single: package
66
67 Python has only one type of module object, and all modules are of this type,
68 regardless of whether the module is implemented in Python, C, or something
69 else. To help organize modules and provide a naming hierarchy, Python has a
70 concept of :term:`packages <package>`.
71
72 You can think of packages as the directories on a file system and modules as
73 files within directories, but don't take this analogy too literally since
74 packages and modules need not originate from the file system. For the
75 purposes of this documentation, we'll use this convenient analogy of
76 directories and files. Like file system directories, packages are organized
77 hierarchically, and packages may themselves contain subpackages, as well as
78 regular modules.
79
80 It's important to keep in mind that all packages are modules, but not all
81 modules are packages. Or put another way, packages are just a special kind of
82 module. Specifically, any module that contains a ``__path__`` attribute is
83 considered a package.
84
85 All modules have a name. Subpackage names are separated from their parent
86 package name by a dot, akin to Python's standard attribute access syntax. Thus
87 you might have a package called :mod:`email`, which in turn has a subpackage
88 called :mod:`email.mime` and a module within that subpackage called
89 :mod:`email.mime.text`.
90
91
92 Regular packages
93 ----------------
94
95 .. index::
96 pair: package; regular
97
98 Python defines two types of packages, :term:`regular packages <regular
99 package>` and :term:`namespace packages <namespace package>`. Regular
100 packages are traditional packages as they existed in Python 3.2 and earlier.
101 A regular package is typically implemented as a directory containing an
102 ``__init__.py`` file. When a regular package is imported, this
103 ``__init__.py`` file is implicitly executed, and the objects it defines are
104 bound to names in the package's namespace. The ``__init__.py`` file can
105 contain the same Python code that any other module can contain, and Python
106 will add some additional attributes to the module when it is imported.
107
108 For example, the following file system layout defines a top level ``parent``
109 package with three subpackages::
110
111 parent/
112 __init__.py
113 one/
114 __init__.py
115 two/
116 __init__.py
117 three/
118 __init__.py
119
120 Importing ``parent.one`` will implicitly execute ``parent/__init__.py`` and
121 ``parent/one/__init__.py``. Subsequent imports of ``parent.two`` or
122 ``parent.three`` will execute ``parent/two/__init__.py`` and
123 ``parent/three/__init__.py`` respectively.
124
125
126 .. _reference-namespace-package:
127
128 Namespace packages
129 ------------------
130
131 .. index::
132 pair: package; namespace
133 pair: package; portion
134
135 A namespace package is a composite of various :term:`portions <portion>`,
136 where each portion contributes a subpackage to the parent package. Portions
137 may reside in different locations on the file system. Portions may also be
138 found in zip files, on the network, or anywhere else that Python searches
139 during import. Namespace packages may or may not correspond directly to
140 objects on the file system; they may be virtual modules that have no concrete
141 representation.
142
143 Namespace packages do not use an ordinary list for their ``__path__``
144 attribute. They instead use a custom iterable type which will automatically
145 perform a new search for package portions on the next import attempt within
146 that package if the path of their parent package (or :data:`sys.path` for a
147 top level package) changes.
148
149 With namespace packages, there is no ``parent/__init__.py`` file. In fact,
150 there may be multiple ``parent`` directories found during import search, where
151 each one is provided by a different portion. Thus ``parent/one`` may not be
152 physically located next to ``parent/two``. In this case, Python will create a
153 namespace package for the top-level ``parent`` package whenever it or one of
154 its subpackages is imported.
155
156 See also :pep:`420` for the namespace package specification.
157
158
159 Searching
160 =========
161
162 To begin the search, Python needs the :term:`fully qualified <qualified name>`
163 name of the module (or package, but for the purposes of this discussion, the
164 difference is immaterial) being imported. This name may come from various
165 arguments to the :keyword:`import` statement, or from the parameters to the
166 :func:`importlib.import_module` or :func:`__import__` functions.
167
168 This name will be used in various phases of the import search, and it may be
169 the dotted path to a submodule, e.g. ``foo.bar.baz``. In this case, Python
170 first tries to import ``foo``, then ``foo.bar``, and finally ``foo.bar.baz``.
171 If any of the intermediate imports fail, a :exc:`ModuleNotFoundError` is raised.
172
173
174 The module cache
175 ----------------
176
177 .. index::
178 single: sys.modules
179
180 The first place checked during import search is :data:`sys.modules`. This
181 mapping serves as a cache of all modules that have been previously imported,
182 including the intermediate paths. So if ``foo.bar.baz`` was previously
183 imported, :data:`sys.modules` will contain entries for ``foo``, ``foo.bar``,
184 and ``foo.bar.baz``. Each key will have as its value the corresponding module
185 object.
186
187 During import, the module name is looked up in :data:`sys.modules` and if
188 present, the associated value is the module satisfying the import, and the
189 process completes. However, if the value is ``None``, then a
190 :exc:`ModuleNotFoundError` is raised. If the module name is missing, Python will
191 continue searching for the module.
192
193 :data:`sys.modules` is writable. Deleting a key may not destroy the
194 associated module (as other modules may hold references to it),
195 but it will invalidate the cache entry for the named module, causing
196 Python to search anew for the named module upon its next
197 import. The key can also be assigned to ``None``, forcing the next import
198 of the module to result in a :exc:`ModuleNotFoundError`.
199
200 Beware though, as if you keep a reference to the module object,
201 invalidate its cache entry in :data:`sys.modules`, and then re-import the
202 named module, the two module objects will *not* be the same. By contrast,
203 :func:`importlib.reload` will reuse the *same* module object, and simply
204 reinitialise the module contents by rerunning the module's code.
205
206
207 .. _finders-and-loaders:
208
209 Finders and loaders
210 -------------------
211
212 .. index::
213 single: finder
214 single: loader
215 single: module spec
216
217 If the named module is not found in :data:`sys.modules`, then Python's import
218 protocol is invoked to find and load the module. This protocol consists of
219 two conceptual objects, :term:`finders <finder>` and :term:`loaders <loader>`.
220 A finder's job is to determine whether it can find the named module using
221 whatever strategy it knows about. Objects that implement both of these
222 interfaces are referred to as :term:`importers <importer>` - they return
223 themselves when they find that they can load the requested module.
224
225 Python includes a number of default finders and importers. The first one
226 knows how to locate built-in modules, and the second knows how to locate
227 frozen modules. A third default finder searches an :term:`import path`
228 for modules. The :term:`import path` is a list of locations that may
229 name file system paths or zip files. It can also be extended to search
230 for any locatable resource, such as those identified by URLs.
231
232 The import machinery is extensible, so new finders can be added to extend the
233 range and scope of module searching.
234
235 Finders do not actually load modules. If they can find the named module, they
236 return a :dfn:`module spec`, an encapsulation of the module's import-related
237 information, which the import machinery then uses when loading the module.
238
239 The following sections describe the protocol for finders and loaders in more
240 detail, including how you can create and register new ones to extend the
241 import machinery.
242
243 .. versionchanged:: 3.4
244 In previous versions of Python, finders returned :term:`loaders <loader>`
245 directly, whereas now they return module specs which *contain* loaders.
246 Loaders are still used during import but have fewer responsibilities.
247
248 Import hooks
249 ------------
250
251 .. index::
252 single: import hooks
253 single: meta hooks
254 single: path hooks
255 pair: hooks; import
256 pair: hooks; meta
257 pair: hooks; path
258
259 The import machinery is designed to be extensible; the primary mechanism for
260 this are the *import hooks*. There are two types of import hooks: *meta
261 hooks* and *import path hooks*.
262
263 Meta hooks are called at the start of import processing, before any other
264 import processing has occurred, other than :data:`sys.modules` cache look up.
265 This allows meta hooks to override :data:`sys.path` processing, frozen
266 modules, or even built-in modules. Meta hooks are registered by adding new
267 finder objects to :data:`sys.meta_path`, as described below.
268
269 Import path hooks are called as part of :data:`sys.path` (or
270 ``package.__path__``) processing, at the point where their associated path
271 item is encountered. Import path hooks are registered by adding new callables
272 to :data:`sys.path_hooks` as described below.
273
274
275 The meta path
276 -------------
277
278 .. index::
279 single: sys.meta_path
280 pair: finder; find_spec
281
282 When the named module is not found in :data:`sys.modules`, Python next
283 searches :data:`sys.meta_path`, which contains a list of meta path finder
284 objects. These finders are queried in order to see if they know how to handle
285 the named module. Meta path finders must implement a method called
286 :meth:`~importlib.abc.MetaPathFinder.find_spec` which takes three arguments:
287 a name, an import path, and (optionally) a target module. The meta path
288 finder can use any strategy it wants to determine whether it can handle
289 the named module or not.
290
291 If the meta path finder knows how to handle the named module, it returns a
292 spec object. If it cannot handle the named module, it returns ``None``. If
293 :data:`sys.meta_path` processing reaches the end of its list without returning
294 a spec, then a :exc:`ModuleNotFoundError` is raised. Any other exceptions
295 raised are simply propagated up, aborting the import process.
296
297 The :meth:`~importlib.abc.MetaPathFinder.find_spec` method of meta path
298 finders is called with two or three arguments. The first is the fully
299 qualified name of the module being imported, for example ``foo.bar.baz``.
300 The second argument is the path entries to use for the module search. For
301 top-level modules, the second argument is ``None``, but for submodules or
302 subpackages, the second argument is the value of the parent package's
303 ``__path__`` attribute. If the appropriate ``__path__`` attribute cannot
304 be accessed, a :exc:`ModuleNotFoundError` is raised. The third argument
305 is an existing module object that will be the target of loading later.
306 The import system passes in a target module only during reload.
307
308 The meta path may be traversed multiple times for a single import request.
309 For example, assuming none of the modules involved has already been cached,
310 importing ``foo.bar.baz`` will first perform a top level import, calling
311 ``mpf.find_spec("foo", None, None)`` on each meta path finder (``mpf``). After
312 ``foo`` has been imported, ``foo.bar`` will be imported by traversing the
313 meta path a second time, calling
314 ``mpf.find_spec("foo.bar", foo.__path__, None)``. Once ``foo.bar`` has been
315 imported, the final traversal will call
316 ``mpf.find_spec("foo.bar.baz", foo.bar.__path__, None)``.
317
318 Some meta path finders only support top level imports. These importers will
319 always return ``None`` when anything other than ``None`` is passed as the
320 second argument.
321
322 Python's default :data:`sys.meta_path` has three meta path finders, one that
323 knows how to import built-in modules, one that knows how to import frozen
324 modules, and one that knows how to import modules from an :term:`import path`
325 (i.e. the :term:`path based finder`).
326
327 .. versionchanged:: 3.4
328 The :meth:`~importlib.abc.MetaPathFinder.find_spec` method of meta path
329 finders replaced :meth:`!find_module`, which
330 is now deprecated. While it will continue to work without change, the
331 import machinery will try it only if the finder does not implement
332 :meth:`~importlib.abc.MetaPathFinder.find_spec`.
333
334 .. versionchanged:: 3.10
335 Use of :meth:`!find_module` by the import system
336 now raises :exc:`ImportWarning`.
337
338 .. versionchanged:: 3.12
339 :meth:`!find_module` has been removed.
340 Use :meth:`~importlib.abc.MetaPathFinder.find_spec` instead.
341
342
343 Loading
344 =======
345
346 If and when a module spec is found, the import machinery will use it (and
347 the loader it contains) when loading the module. Here is an approximation
348 of what happens during the loading portion of import::
349
350 module = None
351 if spec.loader is not None and hasattr(spec.loader, 'create_module'):
352 # It is assumed 'exec_module' will also be defined on the loader.
353 module = spec.loader.create_module(spec)
354 if module is None:
355 module = ModuleType(spec.name)
356 # The import-related module attributes get set here:
357 _init_module_attrs(spec, module)
358
359 if spec.loader is None:
360 # unsupported
361 raise ImportError
362 if spec.origin is None and spec.submodule_search_locations is not None:
363 # namespace package
364 sys.modules[spec.name] = module
365 elif not hasattr(spec.loader, 'exec_module'):
366 module = spec.loader.load_module(spec.name)
367 else:
368 sys.modules[spec.name] = module
369 try:
370 spec.loader.exec_module(module)
371 except BaseException:
372 try:
373 del sys.modules[spec.name]
374 except KeyError:
375 pass
376 raise
377 return sys.modules[spec.name]
378
379 Note the following details:
380
381 * If there is an existing module object with the given name in
382 :data:`sys.modules`, import will have already returned it.
383
384 * The module will exist in :data:`sys.modules` before the loader
385 executes the module code. This is crucial because the module code may
386 (directly or indirectly) import itself; adding it to :data:`sys.modules`
387 beforehand prevents unbounded recursion in the worst case and multiple
388 loading in the best.
389
390 * If loading fails, the failing module -- and only the failing module --
391 gets removed from :data:`sys.modules`. Any module already in the
392 :data:`sys.modules` cache, and any module that was successfully loaded
393 as a side-effect, must remain in the cache. This contrasts with
394 reloading where even the failing module is left in :data:`sys.modules`.
395
396 * After the module is created but before execution, the import machinery
397 sets the import-related module attributes ("_init_module_attrs" in
398 the pseudo-code example above), as summarized in a
399 :ref:`later section <import-mod-attrs>`.
400
401 * Module execution is the key moment of loading in which the module's
402 namespace gets populated. Execution is entirely delegated to the
403 loader, which gets to decide what gets populated and how.
404
405 * The module created during loading and passed to exec_module() may
406 not be the one returned at the end of import [#fnlo]_.
407
408 .. versionchanged:: 3.4
409 The import system has taken over the boilerplate responsibilities of
410 loaders. These were previously performed by the
411 :meth:`importlib.abc.Loader.load_module` method.
412
413 Loaders
414 -------
415
416 Module loaders provide the critical function of loading: module execution.
417 The import machinery calls the :meth:`importlib.abc.Loader.exec_module`
418 method with a single argument, the module object to execute. Any value
419 returned from :meth:`~importlib.abc.Loader.exec_module` is ignored.
420
421 Loaders must satisfy the following requirements:
422
423 * If the module is a Python module (as opposed to a built-in module or a
424 dynamically loaded extension), the loader should execute the module's code
425 in the module's global name space (``module.__dict__``).
426
427 * If the loader cannot execute the module, it should raise an
428 :exc:`ImportError`, although any other exception raised during
429 :meth:`~importlib.abc.Loader.exec_module` will be propagated.
430
431 In many cases, the finder and loader can be the same object; in such cases the
432 :meth:`~importlib.abc.MetaPathFinder.find_spec` method would just return a
433 spec with the loader set to ``self``.
434
435 Module loaders may opt in to creating the module object during loading
436 by implementing a :meth:`~importlib.abc.Loader.create_module` method.
437 It takes one argument, the module spec, and returns the new module object
438 to use during loading. ``create_module()`` does not need to set any attributes
439 on the module object. If the method returns ``None``, the
440 import machinery will create the new module itself.
441
442 .. versionadded:: 3.4
443 The :meth:`~importlib.abc.Loader.create_module` method of loaders.
444
445 .. versionchanged:: 3.4
446 The :meth:`~importlib.abc.Loader.load_module` method was replaced by
447 :meth:`~importlib.abc.Loader.exec_module` and the import
448 machinery assumed all the boilerplate responsibilities of loading.
449
450 For compatibility with existing loaders, the import machinery will use
451 the ``load_module()`` method of loaders if it exists and the loader does
452 not also implement ``exec_module()``. However, ``load_module()`` has been
453 deprecated and loaders should implement ``exec_module()`` instead.
454
455 The ``load_module()`` method must implement all the boilerplate loading
456 functionality described above in addition to executing the module. All
457 the same constraints apply, with some additional clarification:
458
459 * If there is an existing module object with the given name in
460 :data:`sys.modules`, the loader must use that existing module.
461 (Otherwise, :func:`importlib.reload` will not work correctly.) If the
462 named module does not exist in :data:`sys.modules`, the loader
463 must create a new module object and add it to :data:`sys.modules`.
464
465 * The module *must* exist in :data:`sys.modules` before the loader
466 executes the module code, to prevent unbounded recursion or multiple
467 loading.
468
469 * If loading fails, the loader must remove any modules it has inserted
470 into :data:`sys.modules`, but it must remove **only** the failing
471 module(s), and only if the loader itself has loaded the module(s)
472 explicitly.
473
474 .. versionchanged:: 3.5
475 A :exc:`DeprecationWarning` is raised when ``exec_module()`` is defined but
476 ``create_module()`` is not.
477
478 .. versionchanged:: 3.6
479 An :exc:`ImportError` is raised when ``exec_module()`` is defined but
480 ``create_module()`` is not.
481
482 .. versionchanged:: 3.10
483 Use of ``load_module()`` will raise :exc:`ImportWarning`.
484
485 Submodules
486 ----------
487
488 When a submodule is loaded using any mechanism (e.g. ``importlib`` APIs, the
489 ``import`` or ``import-from`` statements, or built-in ``__import__()``) a
490 binding is placed in the parent module's namespace to the submodule object.
491 For example, if package ``spam`` has a submodule ``foo``, after importing
492 ``spam.foo``, ``spam`` will have an attribute ``foo`` which is bound to the
493 submodule. Let's say you have the following directory structure::
494
495 spam/
496 __init__.py
497 foo.py
498
499 and ``spam/__init__.py`` has the following line in it::
500
501 from .foo import Foo
502
503 then executing the following puts name bindings for ``foo`` and ``Foo`` in the
504 ``spam`` module::
505
506 >>> import spam
507 >>> spam.foo
508 <module 'spam.foo' from '/tmp/imports/spam/foo.py'>
509 >>> spam.Foo
510 <class 'spam.foo.Foo'>
511
512 Given Python's familiar name binding rules this might seem surprising, but
513 it's actually a fundamental feature of the import system. The invariant
514 holding is that if you have ``sys.modules['spam']`` and
515 ``sys.modules['spam.foo']`` (as you would after the above import), the latter
516 must appear as the ``foo`` attribute of the former.
517
518 .. _module-specs:
519
520 Module specs
521 ------------
522
523 The import machinery uses a variety of information about each module
524 during import, especially before loading. Most of the information is
525 common to all modules. The purpose of a module's spec is to encapsulate
526 this import-related information on a per-module basis.
527
528 Using a spec during import allows state to be transferred between import
529 system components, e.g. between the finder that creates the module spec
530 and the loader that executes it. Most importantly, it allows the
531 import machinery to perform the boilerplate operations of loading,
532 whereas without a module spec the loader had that responsibility.
533
534 The module's spec is exposed as :attr:`module.__spec__`. Setting
535 :attr:`!__spec__` appropriately applies equally to
536 :ref:`modules initialized during interpreter startup <programs>`.
537 The one exception is ``__main__``, where :attr:`!__spec__` is
538 :ref:`set to None in some cases <main_spec>`.
539
540 See :class:`~importlib.machinery.ModuleSpec` for details on the contents of
541 the module spec.
542
543 .. versionadded:: 3.4
544
545 .. _package-path-rules:
546
547 __path__ attributes on modules
548 ------------------------------
549
550 The :attr:`~module.__path__` attribute should be a (possibly empty)
551 :term:`sequence` of strings enumerating the locations where the package's
552 submodules will be found. By definition, if a module has a :attr:`!__path__`
553 attribute, it is a :term:`package`.
554
555 A package's :attr:`~module.__path__` attribute is used during imports of its
556 subpackages.
557 Within the import machinery, it functions much the same as :data:`sys.path`,
558 i.e. providing a list of locations to search for modules during import.
559 However, :attr:`!__path__` is typically much more constrained than
560 :data:`!sys.path`.
561
562 The same rules used for :data:`sys.path` also apply to a package's
563 :attr:`!__path__`. :data:`sys.path_hooks` (described below) are
564 consulted when traversing a package's :attr:`!__path__`.
565
566 A package's ``__init__.py`` file may set or alter the package's
567 :attr:`~module.__path__`
568 attribute, and this was typically the way namespace packages were implemented
569 prior to :pep:`420`. With the adoption of :pep:`420`, namespace packages no
570 longer need to supply ``__init__.py`` files containing only :attr:`!__path__`
571 manipulation code; the import machinery automatically sets :attr:`!__path__`
572 correctly for the namespace package.
573
574 Module reprs
575 ------------
576
577 By default, all modules have a usable repr, however depending on the
578 attributes set above, and in the module's spec, you can more explicitly
579 control the repr of module objects.
580
581 If the module has a spec (``__spec__``), the import machinery will try
582 to generate a repr from it. If that fails or there is no spec, the import
583 system will craft a default repr using whatever information is available
584 on the module. It will try to use the ``module.__name__``,
585 ``module.__file__``, and ``module.__loader__`` as input into the repr,
586 with defaults for whatever information is missing.
587
588 Here are the exact rules used:
589
590 * If the module has a ``__spec__`` attribute, the information in the spec
591 is used to generate the repr. The "name", "loader", "origin", and
592 "has_location" attributes are consulted.
593
594 * If the module has a ``__file__`` attribute, this is used as part of the
595 module's repr.
596
597 * If the module has no ``__file__`` but does have a ``__loader__`` that is not
598 ``None``, then the loader's repr is used as part of the module's repr.
599
600 * Otherwise, just use the module's ``__name__`` in the repr.
601
602 .. versionchanged:: 3.12
603 Use of :meth:`!module_repr`, having been deprecated since Python 3.4, was
604 removed in Python 3.12 and is no longer called during the resolution of a
605 module's repr.
606
607 .. _pyc-invalidation:
608
609 Cached bytecode invalidation
610 ----------------------------
611
612 Before Python loads cached bytecode from a ``.pyc`` file, it checks whether the
613 cache is up-to-date with the source ``.py`` file. By default, Python does this
614 by storing the source's last-modified timestamp and size in the cache file when
615 writing it. At runtime, the import system then validates the cache file by
616 checking the stored metadata in the cache file against the source's
617 metadata.
618
619 Python also supports "hash-based" cache files, which store a hash of the source
620 file's contents rather than its metadata. There are two variants of hash-based
621 ``.pyc`` files: checked and unchecked. For checked hash-based ``.pyc`` files,
622 Python validates the cache file by hashing the source file and comparing the
623 resulting hash with the hash in the cache file. If a checked hash-based cache
624 file is found to be invalid, Python regenerates it and writes a new checked
625 hash-based cache file. For unchecked hash-based ``.pyc`` files, Python simply
626 assumes the cache file is valid if it exists. Hash-based ``.pyc`` files
627 validation behavior may be overridden with the :option:`--check-hash-based-pycs`
628 flag.
629
630 .. versionchanged:: 3.7
631 Added hash-based ``.pyc`` files. Previously, Python only supported
632 timestamp-based invalidation of bytecode caches.
633
634
635 The Path Based Finder
636 =====================
637
638 .. index::
639 single: path based finder
640
641 As mentioned previously, Python comes with several default meta path finders.
642 One of these, called the :term:`path based finder`
643 (:class:`~importlib.machinery.PathFinder`), searches an :term:`import path`,
644 which contains a list of :term:`path entries <path entry>`. Each path
645 entry names a location to search for modules.
646
647 The path based finder itself doesn't know how to import anything. Instead, it
648 traverses the individual path entries, associating each of them with a
649 path entry finder that knows how to handle that particular kind of path.
650
651 The default set of path entry finders implement all the semantics for finding
652 modules on the file system, handling special file types such as Python source
653 code (``.py`` files), Python byte code (``.pyc`` files) and
654 shared libraries (e.g. ``.so`` files). When supported by the :mod:`zipimport`
655 module in the standard library, the default path entry finders also handle
656 loading all of these file types (other than shared libraries) from zipfiles.
657
658 Path entries need not be limited to file system locations. They can refer to
659 URLs, database queries, or any other location that can be specified as a
660 string.
661
662 The path based finder provides additional hooks and protocols so that you
663 can extend and customize the types of searchable path entries. For example,
664 if you wanted to support path entries as network URLs, you could write a hook
665 that implements HTTP semantics to find modules on the web. This hook (a
666 callable) would return a :term:`path entry finder` supporting the protocol
667 described below, which was then used to get a loader for the module from the
668 web.
669
670 A word of warning: this section and the previous both use the term *finder*,
671 distinguishing between them by using the terms :term:`meta path finder` and
672 :term:`path entry finder`. These two types of finders are very similar,
673 support similar protocols, and function in similar ways during the import
674 process, but it's important to keep in mind that they are subtly different.
675 In particular, meta path finders operate at the beginning of the import
676 process, as keyed off the :data:`sys.meta_path` traversal.
677
678 By contrast, path entry finders are in a sense an implementation detail
679 of the path based finder, and in fact, if the path based finder were to be
680 removed from :data:`sys.meta_path`, none of the path entry finder semantics
681 would be invoked.
682
683
684 Path entry finders
685 ------------------
686
687 .. index::
688 single: sys.path
689 single: sys.path_hooks
690 single: sys.path_importer_cache
691 single: PYTHONPATH
692
693 The :term:`path based finder` is responsible for finding and loading
694 Python modules and packages whose location is specified with a string
695 :term:`path entry`. Most path entries name locations in the file system,
696 but they need not be limited to this.
697
698 As a meta path finder, the :term:`path based finder` implements the
699 :meth:`~importlib.abc.MetaPathFinder.find_spec` protocol previously
700 described, however it exposes additional hooks that can be used to
701 customize how modules are found and loaded from the :term:`import path`.
702
703 Three variables are used by the :term:`path based finder`, :data:`sys.path`,
704 :data:`sys.path_hooks` and :data:`sys.path_importer_cache`. The ``__path__``
705 attributes on package objects are also used. These provide additional ways
706 that the import machinery can be customized.
707
708 :data:`sys.path` contains a list of strings providing search locations for
709 modules and packages. It is initialized from the :envvar:`PYTHONPATH`
710 environment variable and various other installation- and
711 implementation-specific defaults. Entries in :data:`sys.path` can name
712 directories on the file system, zip files, and potentially other "locations"
713 (see the :mod:`site` module) that should be searched for modules, such as
714 URLs, or database queries. Only strings should be present on
715 :data:`sys.path`; all other data types are ignored.
716
717 The :term:`path based finder` is a :term:`meta path finder`, so the import
718 machinery begins the :term:`import path` search by calling the path
719 based finder's :meth:`~importlib.machinery.PathFinder.find_spec` method as
720 described previously. When the ``path`` argument to
721 :meth:`~importlib.machinery.PathFinder.find_spec` is given, it will be a
722 list of string paths to traverse - typically a package's ``__path__``
723 attribute for an import within that package. If the ``path`` argument is
724 ``None``, this indicates a top level import and :data:`sys.path` is used.
725
726 The path based finder iterates over every entry in the search path, and
727 for each of these, looks for an appropriate :term:`path entry finder`
728 (:class:`~importlib.abc.PathEntryFinder`) for the
729 path entry. Because this can be an expensive operation (e.g. there may be
730 ``stat()`` call overheads for this search), the path based finder maintains
731 a cache mapping path entries to path entry finders. This cache is maintained
732 in :data:`sys.path_importer_cache` (despite the name, this cache actually
733 stores finder objects rather than being limited to :term:`importer` objects).
734 In this way, the expensive search for a particular :term:`path entry`
735 location's :term:`path entry finder` need only be done once. User code is
736 free to remove cache entries from :data:`sys.path_importer_cache` forcing
737 the path based finder to perform the path entry search again.
738
739 If the path entry is not present in the cache, the path based finder iterates
740 over every callable in :data:`sys.path_hooks`. Each of the :term:`path entry
741 hooks <path entry hook>` in this list is called with a single argument, the
742 path entry to be searched. This callable may either return a :term:`path
743 entry finder` that can handle the path entry, or it may raise
744 :exc:`ImportError`. An :exc:`ImportError` is used by the path based finder to
745 signal that the hook cannot find a :term:`path entry finder`
746 for that :term:`path entry`. The
747 exception is ignored and :term:`import path` iteration continues. The hook
748 should expect either a string or bytes object; the encoding of bytes objects
749 is up to the hook (e.g. it may be a file system encoding, UTF-8, or something
750 else), and if the hook cannot decode the argument, it should raise
751 :exc:`ImportError`.
752
753 If :data:`sys.path_hooks` iteration ends with no :term:`path entry finder`
754 being returned, then the path based finder's
755 :meth:`~importlib.machinery.PathFinder.find_spec` method will store ``None``
756 in :data:`sys.path_importer_cache` (to indicate that there is no finder for
757 this path entry) and return ``None``, indicating that this
758 :term:`meta path finder` could not find the module.
759
760 If a :term:`path entry finder` *is* returned by one of the :term:`path entry
761 hook` callables on :data:`sys.path_hooks`, then the following protocol is used
762 to ask the finder for a module spec, which is then used when loading the
763 module.
764
765 The current working directory -- denoted by an empty string -- is handled
766 slightly differently from other entries on :data:`sys.path`. First, if the
767 current working directory cannot be determined or is found not to exist, no
768 value is stored in :data:`sys.path_importer_cache`. Second, the value for the
769 current working directory is looked up fresh for each module lookup. Third,
770 the path used for :data:`sys.path_importer_cache` and returned by
771 :meth:`importlib.machinery.PathFinder.find_spec` will be the actual current
772 working directory and not the empty string.
773
774 Path entry finder protocol
775 --------------------------
776
777 In order to support imports of modules and initialized packages and also to
778 contribute portions to namespace packages, path entry finders must implement
779 the :meth:`~importlib.abc.PathEntryFinder.find_spec` method.
780
781 :meth:`~importlib.abc.PathEntryFinder.find_spec` takes two arguments: the
782 fully qualified name of the module being imported, and the (optional) target
783 module. ``find_spec()`` returns a fully populated spec for the module.
784 This spec will always have "loader" set (with one exception).
785
786 To indicate to the import machinery that the spec represents a namespace
787 :term:`portion`, the path entry finder sets ``submodule_search_locations`` to
788 a list containing the portion.
789
790 .. versionchanged:: 3.4
791 :meth:`~importlib.abc.PathEntryFinder.find_spec` replaced
792 :meth:`!find_loader` and
793 :meth:`!find_module`, both of which
794 are now deprecated, but will be used if ``find_spec()`` is not defined.
795
796 Older path entry finders may implement one of these two deprecated methods
797 instead of ``find_spec()``. The methods are still respected for the
798 sake of backward compatibility. However, if ``find_spec()`` is
799 implemented on the path entry finder, the legacy methods are ignored.
800
801 :meth:`!find_loader` takes one argument, the
802 fully qualified name of the module being imported. ``find_loader()``
803 returns a 2-tuple where the first item is the loader and the second item
804 is a namespace :term:`portion`.
805
806 For backwards compatibility with other implementations of the import
807 protocol, many path entry finders also support the same,
808 traditional ``find_module()`` method that meta path finders support.
809 However path entry finder ``find_module()`` methods are never called
810 with a ``path`` argument (they are expected to record the appropriate
811 path information from the initial call to the path hook).
812
813 The ``find_module()`` method on path entry finders is deprecated,
814 as it does not allow the path entry finder to contribute portions to
815 namespace packages. If both ``find_loader()`` and ``find_module()``
816 exist on a path entry finder, the import system will always call
817 ``find_loader()`` in preference to ``find_module()``.
818
819 .. versionchanged:: 3.10
820 Calls to :meth:`!find_module` and
821 :meth:`!find_loader` by the import
822 system will raise :exc:`ImportWarning`.
823
824 .. versionchanged:: 3.12
825 ``find_module()`` and ``find_loader()`` have been removed.
826
827
828 Replacing the standard import system
829 ====================================
830
831 The most reliable mechanism for replacing the entire import system is to
832 delete the default contents of :data:`sys.meta_path`, replacing them
833 entirely with a custom meta path hook.
834
835 If it is acceptable to only alter the behaviour of import statements
836 without affecting other APIs that access the import system, then replacing
837 the builtin :func:`__import__` function may be sufficient.
838
839 To selectively prevent the import of some modules from a hook early on the
840 meta path (rather than disabling the standard import system entirely),
841 it is sufficient to raise :exc:`ModuleNotFoundError` directly from
842 :meth:`~importlib.abc.MetaPathFinder.find_spec` instead of returning
843 ``None``. The latter indicates that the meta path search should continue,
844 while raising an exception terminates it immediately.
845
846 .. _relativeimports:
847
848 Package Relative Imports
849 ========================
850
851 Relative imports use leading dots. A single leading dot indicates a relative
852 import, starting with the current package. Two or more leading dots indicate a
853 relative import to the parent(s) of the current package, one level per dot
854 after the first. For example, given the following package layout::
855
856 package/
857 __init__.py
858 subpackage1/
859 __init__.py
860 moduleX.py
861 moduleY.py
862 subpackage2/
863 __init__.py
864 moduleZ.py
865 moduleA.py
866
867 In either ``subpackage1/moduleX.py`` or ``subpackage1/__init__.py``,
868 the following are valid relative imports::
869
870 from .moduleY import spam
871 from .moduleY import spam as ham
872 from . import moduleY
873 from ..subpackage1 import moduleY
874 from ..subpackage2.moduleZ import eggs
875 from ..moduleA import foo
876
877 Absolute imports may use either the ``import <>`` or ``from <> import <>``
878 syntax, but relative imports may only use the second form; the reason
879 for this is that::
880
881 import XXX.YYY.ZZZ
882
883 should expose ``XXX.YYY.ZZZ`` as a usable expression, but .moduleY is
884 not a valid expression.
885
886
887 .. _import-dunder-main:
888
889 Special considerations for __main__
890 ===================================
891
892 The :mod:`__main__` module is a special case relative to Python's import
893 system. As noted :ref:`elsewhere <programs>`, the ``__main__`` module
894 is directly initialized at interpreter startup, much like :mod:`sys` and
895 :mod:`builtins`. However, unlike those two, it doesn't strictly
896 qualify as a built-in module. This is because the manner in which
897 ``__main__`` is initialized depends on the flags and other options with
898 which the interpreter is invoked.
899
900 .. _main_spec:
901
902 __main__.__spec__
903 -----------------
904
905 Depending on how :mod:`__main__` is initialized, ``__main__.__spec__``
906 gets set appropriately or to ``None``.
907
908 When Python is started with the :option:`-m` option, ``__spec__`` is set
909 to the module spec of the corresponding module or package. ``__spec__`` is
910 also populated when the ``__main__`` module is loaded as part of executing a
911 directory, zipfile or other :data:`sys.path` entry.
912
913 In :ref:`the remaining cases <using-on-interface-options>`
914 ``__main__.__spec__`` is set to ``None``, as the code used to populate the
915 :mod:`__main__` does not correspond directly with an importable module:
916
917 - interactive prompt
918 - :option:`-c` option
919 - running from stdin
920 - running directly from a source or bytecode file
921
922 Note that ``__main__.__spec__`` is always ``None`` in the last case,
923 *even if* the file could technically be imported directly as a module
924 instead. Use the :option:`-m` switch if valid module metadata is desired
925 in :mod:`__main__`.
926
927 Note also that even when ``__main__`` corresponds with an importable module
928 and ``__main__.__spec__`` is set accordingly, they're still considered
929 *distinct* modules. This is due to the fact that blocks guarded by
930 ``if __name__ == "__main__":`` checks only execute when the module is used
931 to populate the ``__main__`` namespace, and not during normal import.
932
933
934 References
935 ==========
936
937 The import machinery has evolved considerably since Python's early days. The
938 original `specification for packages
939 <https://www.python.org/doc/essays/packages/>`_ is still available to read,
940 although some details have changed since the writing of that document.
941
942 The original specification for :data:`sys.meta_path` was :pep:`302`, with
943 subsequent extension in :pep:`420`.
944
945 :pep:`420` introduced :term:`namespace packages <namespace package>` for
946 Python 3.3. :pep:`420` also introduced the :meth:`!find_loader` protocol as an
947 alternative to :meth:`!find_module`.
948
949 :pep:`366` describes the addition of the ``__package__`` attribute for
950 explicit relative imports in main modules.
951
952 :pep:`328` introduced absolute and explicit relative imports and initially
953 proposed ``__name__`` for semantics :pep:`366` would eventually specify for
954 ``__package__``.
955
956 :pep:`338` defines executing modules as scripts.
957
958 :pep:`451` adds the encapsulation of per-module import state in spec
959 objects. It also off-loads most of the boilerplate responsibilities of
960 loaders back onto the import machinery. These changes allow the
961 deprecation of several APIs in the import system and also addition of new
962 methods to finders and loaders.
963
964 .. rubric:: Footnotes
965
966 .. [#fnmo] See :class:`types.ModuleType`.
967
968 .. [#fnlo] The importlib implementation avoids using the return value
969 directly. Instead, it gets the module object by looking the module name up
970 in :data:`sys.modules`. The indirect effect of this is that an imported
971 module may replace itself in :data:`sys.modules`. This is
972 implementation-specific behavior that is not guaranteed to work in other
973 Python implementations.