Debian Codegraph @

Doc/tutorial/modules.rst

raw
1 .. _tut-modules:
2
3 *******
4 Modules
5 *******
6
7 If you quit from the Python interpreter and enter it again, the definitions you
8 have made (functions and variables) are lost. Therefore, if you want to write a
9 somewhat longer program, you are better off using a text editor to prepare the
10 input for the interpreter and running it with that file as input instead. This
11 is known as creating a *script*. As your program gets longer, you may want to
12 split it into several files for easier maintenance. You may also want to use a
13 handy function that you've written in several programs without copying its
14 definition into each program.
15
16 To support this, Python has a way to put definitions in a file and use them in a
17 script or in an interactive instance of the interpreter. Such a file is called a
18 *module*; definitions from a module can be *imported* into other modules or into
19 the *main* module (the collection of variables that you have access to in a
20 script executed at the top level and in calculator mode).
21
22 A module is a file containing Python definitions and statements. The file name
23 is the module name with the suffix :file:`.py` appended. Within a module, the
24 module's name (as a string) is available as the value of the global variable
25 ``__name__``. For instance, use your favorite text editor to create a file
26 called :file:`fibo.py` in the current directory with the following contents::
27
28 # Fibonacci numbers module
29
30 def fib(n):
31 """Write Fibonacci series up to n."""
32 a, b = 0, 1
33 while a < n:
34 print(a, end=' ')
35 a, b = b, a+b
36 print()
37
38 def fib2(n):
39 """Return Fibonacci series up to n."""
40 result = []
41 a, b = 0, 1
42 while a < n:
43 result.append(a)
44 a, b = b, a+b
45 return result
46
47 Now enter the Python interpreter and import this module with the following
48 command::
49
50 >>> import fibo
51
52 This does not add the names of the functions defined in ``fibo`` directly to
53 the current :term:`namespace` (see :ref:`tut-scopes` for more details);
54 it only adds the module name ``fibo`` there. Using
55 the module name you can access the functions::
56
57 >>> fibo.fib(1000)
58 0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987
59 >>> fibo.fib2(100)
60 [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89]
61 >>> fibo.__name__
62 'fibo'
63
64 If you intend to use a function often you can assign it to a local name::
65
66 >>> fib = fibo.fib
67 >>> fib(500)
68 0 1 1 2 3 5 8 13 21 34 55 89 144 233 377
69
70
71 .. _tut-moremodules:
72
73 More on Modules
74 ===============
75
76 A module can contain executable statements as well as function definitions.
77 These statements are intended to initialize the module. They are executed only
78 the *first* time the module name is encountered in an import statement. [#]_
79 (They are also run if the file is executed as a script.)
80
81 Each module has its own private namespace, which is used as the global namespace
82 by all functions defined in the module. Thus, the author of a module can
83 use global variables in the module without worrying about accidental clashes
84 with a user's global variables. On the other hand, if you know what you are
85 doing you can touch a module's global variables with the same notation used to
86 refer to its functions, ``modname.itemname``.
87
88 Modules can import other modules. It is customary but not required to place all
89 :keyword:`import` statements at the beginning of a module (or script, for that
90 matter). The imported module names, if placed at the top level of a module
91 (outside any functions or classes), are added to the module's global namespace.
92
93 There is a variant of the :keyword:`import` statement that imports names from a
94 module directly into the importing module's namespace. For example::
95
96 >>> from fibo import fib, fib2
97 >>> fib(500)
98 0 1 1 2 3 5 8 13 21 34 55 89 144 233 377
99
100 This does not introduce the module name from which the imports are taken in the
101 local namespace (so in the example, ``fibo`` is not defined).
102
103 There is even a variant to import all names that a module defines::
104
105 >>> from fibo import *
106 >>> fib(500)
107 0 1 1 2 3 5 8 13 21 34 55 89 144 233 377
108
109 This imports all names except those beginning with an underscore (``_``).
110 In most cases Python programmers do not use this facility since it introduces
111 an unknown set of names into the interpreter, possibly hiding some things
112 you have already defined.
113
114 Note that in general the practice of importing ``*`` from a module or package is
115 frowned upon, since it often causes poorly readable code. However, it is okay to
116 use it to save typing in interactive sessions.
117
118 If the module name is followed by :keyword:`!as`, then the name
119 following :keyword:`!as` is bound directly to the imported module.
120
121 ::
122
123 >>> import fibo as fib
124 >>> fib.fib(500)
125 0 1 1 2 3 5 8 13 21 34 55 89 144 233 377
126
127 This is effectively importing the module in the same way that ``import fibo``
128 will do, with the only difference of it being available as ``fib``.
129
130 It can also be used when utilising :keyword:`from` with similar effects::
131
132 >>> from fibo import fib as fibonacci
133 >>> fibonacci(500)
134 0 1 1 2 3 5 8 13 21 34 55 89 144 233 377
135
136
137 .. note::
138
139 For efficiency reasons, each module is only imported once per interpreter
140 session. Therefore, if you change your modules, you must restart the
141 interpreter -- or, if it's just one module you want to test interactively,
142 use :func:`importlib.reload`, e.g. ``import importlib;
143 importlib.reload(modulename)``.
144
145
146 .. _tut-modulesasscripts:
147
148 Executing modules as scripts
149 ----------------------------
150
151 When you run a Python module with ::
152
153 python fibo.py <arguments>
154
155 the code in the module will be executed, just as if you imported it, but with
156 the ``__name__`` set to ``"__main__"``. That means that by adding this code at
157 the end of your module::
158
159 if __name__ == "__main__":
160 import sys
161 fib(int(sys.argv[1]))
162
163 you can make the file usable as a script as well as an importable module,
164 because the code that parses the command line only runs if the module is
165 executed as the "main" file:
166
167 .. code-block:: shell-session
168
169 $ python fibo.py 50
170 0 1 1 2 3 5 8 13 21 34
171
172 If the module is imported, the code is not run::
173
174 >>> import fibo
175 >>>
176
177 This is often used either to provide a convenient user interface to a module, or
178 for testing purposes (running the module as a script executes a test suite).
179
180
181 .. _tut-searchpath:
182
183 The Module Search Path
184 ----------------------
185
186 .. index:: triple: module; search; path
187
188 When a module named :mod:`!spam` is imported, the interpreter first searches for
189 a built-in module with that name. These module names are listed in
190 :data:`sys.builtin_module_names`. If not found, it then searches for a file
191 named :file:`spam.py` in a list of directories given by the variable
192 :data:`sys.path`. :data:`sys.path` is initialized from these locations:
193
194 * The directory containing the input script (or the current directory when no
195 file is specified).
196 * :envvar:`PYTHONPATH` (a list of directory names, with the same syntax as the
197 shell variable :envvar:`PATH`).
198 * The installation-dependent default (by convention including a
199 ``site-packages`` directory, handled by the :mod:`site` module).
200
201 More details are at :ref:`sys-path-init`.
202
203 .. note::
204 On file systems which support symlinks, the directory containing the input
205 script is calculated after the symlink is followed. In other words the
206 directory containing the symlink is **not** added to the module search path.
207
208 After initialization, Python programs can modify :data:`sys.path`. The
209 directory containing the script being run is placed at the beginning of the
210 search path, ahead of the standard library path. This means that scripts in that
211 directory will be loaded instead of modules of the same name in the library
212 directory. This is an error unless the replacement is intended. See section
213 :ref:`tut-standardmodules` for more information.
214
215 .. %
216 Do we need stuff on zip files etc. ? DUBOIS
217
218 .. _tut-pycache:
219
220 "Compiled" Python files
221 -----------------------
222
223 To speed up loading modules, Python caches the compiled version of each module
224 in the ``__pycache__`` directory under the name :file:`module.{version}.pyc`,
225 where the version encodes the format of the compiled file; it generally contains
226 the Python version number. For example, in CPython release 3.3 the compiled
227 version of spam.py would be cached as ``__pycache__/spam.cpython-33.pyc``. This
228 naming convention allows compiled modules from different releases and different
229 versions of Python to coexist.
230
231 Python checks the modification date of the source against the compiled version
232 to see if it's out of date and needs to be recompiled. This is a completely
233 automatic process. Also, the compiled modules are platform-independent, so the
234 same library can be shared among systems with different architectures.
235
236 Python does not check the cache in two circumstances. First, it always
237 recompiles and does not store the result for the module that's loaded directly
238 from the command line. Second, it does not check the cache if there is no
239 source module. To support a non-source (compiled only) distribution, the
240 compiled module must be in the source directory, and there must not be a source
241 module.
242
243 Some tips for experts:
244
245 * You can use the :option:`-O` or :option:`-OO` switches on the Python command
246 to reduce the size of a compiled module. The ``-O`` switch removes assert
247 statements, the ``-OO`` switch removes both assert statements and __doc__
248 strings. Since some programs may rely on having these available, you should
249 only use this option if you know what you're doing. "Optimized" modules have
250 an ``opt-`` tag and are usually smaller. Future releases may
251 change the effects of optimization.
252
253 * A program doesn't run any faster when it is read from a ``.pyc``
254 file than when it is read from a ``.py`` file; the only thing that's faster
255 about ``.pyc`` files is the speed with which they are loaded.
256
257 * The module :mod:`compileall` can create .pyc files for all modules in a
258 directory.
259
260 * There is more detail on this process, including a flow chart of the
261 decisions, in :pep:`3147`.
262
263
264 .. _tut-standardmodules:
265
266 Standard Modules
267 ================
268
269 .. index:: pair: module; sys
270
271 Python comes with a library of standard modules, described in a separate
272 document, the Python Library Reference ("Library Reference" hereafter). Some
273 modules are built into the interpreter; these provide access to operations that
274 are not part of the core of the language but are nevertheless built in, either
275 for efficiency or to provide access to operating system primitives such as
276 system calls. The set of such modules is a configuration option which also
277 depends on the underlying platform. For example, the :mod:`winreg` module is only
278 provided on Windows systems. One particular module deserves some attention:
279 :mod:`sys`, which is built into every Python interpreter. The variables
280 ``sys.ps1`` and ``sys.ps2`` define the strings used as primary and secondary
281 prompts::
282
283 >>> import sys
284 >>> sys.ps1
285 '>>> '
286 >>> sys.ps2
287 '... '
288 >>> sys.ps1 = 'C> '
289 C> print('Yuck!')
290 Yuck!
291 C>
292
293
294 These two variables are only defined if the interpreter is in interactive mode.
295
296 The variable ``sys.path`` is a list of strings that determines the interpreter's
297 search path for modules. It is initialized to a default path taken from the
298 environment variable :envvar:`PYTHONPATH`, or from a built-in default if
299 :envvar:`PYTHONPATH` is not set. You can modify it using standard list
300 operations::
301
302 >>> import sys
303 >>> sys.path.append('/ufs/guido/lib/python')
304
305
306 .. _tut-dir:
307
308 The :func:`dir` Function
309 ========================
310
311 The built-in function :func:`dir` is used to find out which names a module
312 defines. It returns a sorted list of strings::
313
314 >>> import fibo, sys
315 >>> dir(fibo)
316 ['__name__', 'fib', 'fib2']
317 >>> dir(sys) # doctest: +NORMALIZE_WHITESPACE
318 ['__breakpointhook__', '__displayhook__', '__doc__', '__excepthook__',
319 '__interactivehook__', '__loader__', '__name__', '__package__', '__spec__',
320 '__stderr__', '__stdin__', '__stdout__', '__unraisablehook__',
321 '_clear_type_cache', '_current_frames', '_debugmallocstats', '_framework',
322 '_getframe', '_git', '_home', '_xoptions', 'abiflags', 'addaudithook',
323 'api_version', 'argv', 'audit', 'base_exec_prefix', 'base_prefix',
324 'breakpointhook', 'builtin_module_names', 'byteorder', 'call_tracing',
325 'callstats', 'copyright', 'displayhook', 'dont_write_bytecode', 'exc_info',
326 'excepthook', 'exec_prefix', 'executable', 'exit', 'flags', 'float_info',
327 'float_repr_style', 'get_asyncgen_hooks', 'get_coroutine_origin_tracking_depth',
328 'getallocatedblocks', 'getdefaultencoding', 'getdlopenflags',
329 'getfilesystemencodeerrors', 'getfilesystemencoding', 'getprofile',
330 'getrecursionlimit', 'getrefcount', 'getsizeof', 'getswitchinterval',
331 'gettrace', 'hash_info', 'hexversion', 'implementation', 'int_info',
332 'intern', 'is_finalizing', 'last_traceback', 'last_type', 'last_value',
333 'maxsize', 'maxunicode', 'meta_path', 'modules', 'path', 'path_hooks',
334 'path_importer_cache', 'platform', 'prefix', 'ps1', 'ps2', 'pycache_prefix',
335 'set_asyncgen_hooks', 'set_coroutine_origin_tracking_depth', 'setdlopenflags',
336 'setprofile', 'setrecursionlimit', 'setswitchinterval', 'settrace', 'stderr',
337 'stdin', 'stdout', 'thread_info', 'unraisablehook', 'version', 'version_info',
338 'warnoptions']
339
340 Without arguments, :func:`dir` lists the names you have defined currently::
341
342 >>> a = [1, 2, 3, 4, 5]
343 >>> import fibo
344 >>> fib = fibo.fib
345 >>> dir()
346 ['__builtins__', '__name__', 'a', 'fib', 'fibo', 'sys']
347
348 Note that it lists all types of names: variables, modules, functions, etc.
349
350 .. index:: pair: module; builtins
351
352 :func:`dir` does not list the names of built-in functions and variables. If you
353 want a list of those, they are defined in the standard module
354 :mod:`builtins`::
355
356 >>> import builtins
357 >>> dir(builtins) # doctest: +NORMALIZE_WHITESPACE
358 ['ArithmeticError', 'AssertionError', 'AttributeError', 'BaseException',
359 'BlockingIOError', 'BrokenPipeError', 'BufferError', 'BytesWarning',
360 'ChildProcessError', 'ConnectionAbortedError', 'ConnectionError',
361 'ConnectionRefusedError', 'ConnectionResetError', 'DeprecationWarning',
362 'EOFError', 'Ellipsis', 'EnvironmentError', 'Exception', 'False',
363 'FileExistsError', 'FileNotFoundError', 'FloatingPointError',
364 'FutureWarning', 'GeneratorExit', 'IOError', 'ImportError',
365 'ImportWarning', 'IndentationError', 'IndexError', 'InterruptedError',
366 'IsADirectoryError', 'KeyError', 'KeyboardInterrupt', 'LookupError',
367 'MemoryError', 'NameError', 'None', 'NotADirectoryError', 'NotImplemented',
368 'NotImplementedError', 'OSError', 'OverflowError',
369 'PendingDeprecationWarning', 'PermissionError', 'ProcessLookupError',
370 'ReferenceError', 'ResourceWarning', 'RuntimeError', 'RuntimeWarning',
371 'StopIteration', 'SyntaxError', 'SyntaxWarning', 'SystemError',
372 'SystemExit', 'TabError', 'TimeoutError', 'True', 'TypeError',
373 'UnboundLocalError', 'UnicodeDecodeError', 'UnicodeEncodeError',
374 'UnicodeError', 'UnicodeTranslateError', 'UnicodeWarning', 'UserWarning',
375 'ValueError', 'Warning', 'ZeroDivisionError', '_', '__build_class__',
376 '__debug__', '__doc__', '__import__', '__name__', '__package__', 'abs',
377 'all', 'any', 'ascii', 'bin', 'bool', 'bytearray', 'bytes', 'callable',
378 'chr', 'classmethod', 'compile', 'complex', 'copyright', 'credits',
379 'delattr', 'dict', 'dir', 'divmod', 'enumerate', 'eval', 'exec', 'exit',
380 'filter', 'float', 'format', 'frozenset', 'getattr', 'globals', 'hasattr',
381 'hash', 'help', 'hex', 'id', 'input', 'int', 'isinstance', 'issubclass',
382 'iter', 'len', 'license', 'list', 'locals', 'map', 'max', 'memoryview',
383 'min', 'next', 'object', 'oct', 'open', 'ord', 'pow', 'print', 'property',
384 'quit', 'range', 'repr', 'reversed', 'round', 'set', 'setattr', 'slice',
385 'sorted', 'staticmethod', 'str', 'sum', 'super', 'tuple', 'type', 'vars',
386 'zip']
387
388 .. _tut-packages:
389
390 Packages
391 ========
392
393 Packages are a way of structuring Python's module namespace by using "dotted
394 module names". For example, the module name :mod:`!A.B` designates a submodule
395 named ``B`` in a package named ``A``. Just like the use of modules saves the
396 authors of different modules from having to worry about each other's global
397 variable names, the use of dotted module names saves the authors of multi-module
398 packages like NumPy or Pillow from having to worry about
399 each other's module names.
400
401 Suppose you want to design a collection of modules (a "package") for the uniform
402 handling of sound files and sound data. There are many different sound file
403 formats (usually recognized by their extension, for example: :file:`.wav`,
404 :file:`.aiff`, :file:`.au`), so you may need to create and maintain a growing
405 collection of modules for the conversion between the various file formats.
406 There are also many different operations you might want to perform on sound data
407 (such as mixing, adding echo, applying an equalizer function, creating an
408 artificial stereo effect), so in addition you will be writing a never-ending
409 stream of modules to perform these operations. Here's a possible structure for
410 your package (expressed in terms of a hierarchical filesystem):
411
412 .. code-block:: text
413
414 sound/ Top-level package
415 __init__.py Initialize the sound package
416 formats/ Subpackage for file format conversions
417 __init__.py
418 wavread.py
419 wavwrite.py
420 aiffread.py
421 aiffwrite.py
422 auread.py
423 auwrite.py
424 ...
425 effects/ Subpackage for sound effects
426 __init__.py
427 echo.py
428 surround.py
429 reverse.py
430 ...
431 filters/ Subpackage for filters
432 __init__.py
433 equalizer.py
434 vocoder.py
435 karaoke.py
436 ...
437
438 When importing the package, Python searches through the directories on
439 ``sys.path`` looking for the package subdirectory.
440
441 The :file:`__init__.py` files are required to make Python treat directories
442 containing the file as packages (unless using a :term:`namespace package`, a
443 relatively advanced feature). This prevents directories with a common name,
444 such as ``string``, from unintentionally hiding valid modules that occur later
445 on the module search path. In the simplest case, :file:`__init__.py` can just be
446 an empty file, but it can also execute initialization code for the package or
447 set the ``__all__`` variable, described later.
448
449 Users of the package can import individual modules from the package, for
450 example::
451
452 import sound.effects.echo
453
454 This loads the submodule :mod:`!sound.effects.echo`. It must be referenced with
455 its full name. ::
456
457 sound.effects.echo.echofilter(input, output, delay=0.7, atten=4)
458
459 An alternative way of importing the submodule is::
460
461 from sound.effects import echo
462
463 This also loads the submodule :mod:`!echo`, and makes it available without its
464 package prefix, so it can be used as follows::
465
466 echo.echofilter(input, output, delay=0.7, atten=4)
467
468 Yet another variation is to import the desired function or variable directly::
469
470 from sound.effects.echo import echofilter
471
472 Again, this loads the submodule :mod:`!echo`, but this makes its function
473 :func:`!echofilter` directly available::
474
475 echofilter(input, output, delay=0.7, atten=4)
476
477 Note that when using ``from package import item``, the item can be either a
478 submodule (or subpackage) of the package, or some other name defined in the
479 package, like a function, class or variable. The ``import`` statement first
480 tests whether the item is defined in the package; if not, it assumes it is a
481 module and attempts to load it. If it fails to find it, an :exc:`ImportError`
482 exception is raised.
483
484 Contrarily, when using syntax like ``import item.subitem.subsubitem``, each item
485 except for the last must be a package; the last item can be a module or a
486 package but can't be a class or function or variable defined in the previous
487 item.
488
489
490 .. _tut-pkg-import-star:
491
492 Importing \* From a Package
493 ---------------------------
494
495 .. index:: single: __all__
496
497 Now what happens when the user writes ``from sound.effects import *``? Ideally,
498 one would hope that this somehow goes out to the filesystem, finds which
499 submodules are present in the package, and imports them all. This could take a
500 long time and importing sub-modules might have unwanted side-effects that should
501 only happen when the sub-module is explicitly imported.
502
503 The only solution is for the package author to provide an explicit index of the
504 package. The :keyword:`import` statement uses the following convention: if a package's
505 :file:`__init__.py` code defines a list named ``__all__``, it is taken to be the
506 list of module names that should be imported when ``from package import *`` is
507 encountered. It is up to the package author to keep this list up-to-date when a
508 new version of the package is released. Package authors may also decide not to
509 support it, if they don't see a use for importing \* from their package. For
510 example, the file :file:`sound/effects/__init__.py` could contain the following
511 code::
512
513 __all__ = ["echo", "surround", "reverse"]
514
515 This would mean that ``from sound.effects import *`` would import the three
516 named submodules of the :mod:`!sound.effects` package.
517
518 Be aware that submodules might become shadowed by locally defined names. For
519 example, if you added a ``reverse`` function to the
520 :file:`sound/effects/__init__.py` file, the ``from sound.effects import *``
521 would only import the two submodules ``echo`` and ``surround``, but *not* the
522 ``reverse`` submodule, because it is shadowed by the locally defined
523 ``reverse`` function::
524
525 __all__ = [
526 "echo", # refers to the 'echo.py' file
527 "surround", # refers to the 'surround.py' file
528 "reverse", # !!! refers to the 'reverse' function now !!!
529 ]
530
531 def reverse(msg: str): # <-- this name shadows the 'reverse.py' submodule
532 return msg[::-1] # in the case of a 'from sound.effects import *'
533
534 If ``__all__`` is not defined, the statement ``from sound.effects import *``
535 does *not* import all submodules from the package :mod:`!sound.effects` into the
536 current namespace; it only ensures that the package :mod:`!sound.effects` has
537 been imported (possibly running any initialization code in :file:`__init__.py`)
538 and then imports whatever names are defined in the package. This includes any
539 names defined (and submodules explicitly loaded) by :file:`__init__.py`. It
540 also includes any submodules of the package that were explicitly loaded by
541 previous :keyword:`import` statements. Consider this code::
542
543 import sound.effects.echo
544 import sound.effects.surround
545 from sound.effects import *
546
547 In this example, the :mod:`!echo` and :mod:`!surround` modules are imported in the
548 current namespace because they are defined in the :mod:`!sound.effects` package
549 when the ``from...import`` statement is executed. (This also works when
550 ``__all__`` is defined.)
551
552 Although certain modules are designed to export only names that follow certain
553 patterns when you use ``import *``, it is still considered bad practice in
554 production code.
555
556 Remember, there is nothing wrong with using ``from package import
557 specific_submodule``! In fact, this is the recommended notation unless the
558 importing module needs to use submodules with the same name from different
559 packages.
560
561
562 .. _intra-package-references:
563
564 Intra-package References
565 ------------------------
566
567 When packages are structured into subpackages (as with the :mod:`!sound` package
568 in the example), you can use absolute imports to refer to submodules of siblings
569 packages. For example, if the module :mod:`!sound.filters.vocoder` needs to use
570 the :mod:`!echo` module in the :mod:`!sound.effects` package, it can use ``from
571 sound.effects import echo``.
572
573 You can also write relative imports, with the ``from module import name`` form
574 of import statement. These imports use leading dots to indicate the current and
575 parent packages involved in the relative import. From the :mod:`!surround`
576 module for example, you might use::
577
578 from . import echo
579 from .. import formats
580 from ..filters import equalizer
581
582 Note that relative imports are based on the name of the current module's package.
583 Since the main module does not have a package, modules intended for use
584 as the main module of a Python application must always use absolute imports.
585
586
587 Packages in Multiple Directories
588 --------------------------------
589
590 Packages support one more special attribute, :attr:`~module.__path__`. This is
591 initialized to be a :term:`sequence` of strings containing the name of the
592 directory holding the
593 package's :file:`__init__.py` before the code in that file is executed. This
594 variable can be modified; doing so affects future searches for modules and
595 subpackages contained in the package.
596
597 While this feature is not often needed, it can be used to extend the set of
598 modules found in a package.
599
600
601 .. rubric:: Footnotes
602
603 .. [#] In fact function definitions are also 'statements' that are 'executed'; the
604 execution of a module-level function definition adds the function name to
605 the module's global namespace.