Debian Codegraph @

Doc/reference/executionmodel.rst

raw
1
2 .. _execmodel:
3
4 ***************
5 Execution model
6 ***************
7
8 .. index::
9 single: execution model
10 pair: code; block
11
12 .. _prog_structure:
13
14 Structure of a program
15 ======================
16
17 .. index:: block
18
19 A Python program is constructed from code blocks.
20 A :dfn:`block` is a piece of Python program text that is executed as a unit.
21 The following are blocks: a module, a function body, and a class definition.
22 Each command typed interactively is a block. A script file (a file given as
23 standard input to the interpreter or specified as a command line argument to the
24 interpreter) is a code block. A script command (a command specified on the
25 interpreter command line with the :option:`-c` option) is a code block.
26 A module run as a top level script (as module ``__main__``) from the command
27 line using a :option:`-m` argument is also a code block. The string
28 argument passed to the built-in functions :func:`eval` and :func:`exec` is a
29 code block.
30
31 .. index:: pair: execution; frame
32
33 A code block is executed in an :dfn:`execution frame`. A frame contains some
34 administrative information (used for debugging) and determines where and how
35 execution continues after the code block's execution has completed.
36
37 .. _naming:
38
39 Naming and binding
40 ==================
41
42 .. index::
43 single: namespace
44 single: scope
45
46 .. _bind_names:
47
48 Binding of names
49 ----------------
50
51 .. index::
52 single: name
53 pair: binding; name
54
55 :dfn:`Names` refer to objects. Names are introduced by name binding operations.
56
57 .. index:: single: from; import statement
58
59 The following constructs bind names:
60
61 * formal parameters to functions,
62 * class definitions,
63 * function definitions,
64 * assignment expressions,
65 * :ref:`targets <assignment>` that are identifiers if occurring in
66 an assignment:
67
68 + :keyword:`for` loop header,
69 + after :keyword:`!as` in a :keyword:`with` statement, :keyword:`except`
70 clause, :keyword:`except* <except_star>` clause, or in the as-pattern in structural pattern matching,
71 + in a capture pattern in structural pattern matching
72
73 * :keyword:`import` statements.
74 * :keyword:`type` statements.
75 * :ref:`type parameter lists <type-params>`.
76
77 The :keyword:`!import` statement of the form ``from ... import *`` binds all
78 names defined in the imported module, except those beginning with an underscore.
79 This form may only be used at the module level.
80
81 A target occurring in a :keyword:`del` statement is also considered bound for
82 this purpose (though the actual semantics are to unbind the name).
83
84 Each assignment or import statement occurs within a block defined by a class or
85 function definition or at the module level (the top-level code block).
86
87 .. index:: pair: free; variable
88
89 If a name is bound in a block, it is a local variable of that block, unless
90 declared as :keyword:`nonlocal` or :keyword:`global`. If a name is bound at
91 the module level, it is a global variable. (The variables of the module code
92 block are local and global.) If a variable is used in a code block but not
93 defined there, it is a :term:`free variable`.
94
95 Each occurrence of a name in the program text refers to the :dfn:`binding` of
96 that name established by the following name resolution rules.
97
98 .. _resolve_names:
99
100 Resolution of names
101 -------------------
102
103 .. index:: scope
104
105 A :dfn:`scope` defines the visibility of a name within a block. If a local
106 variable is defined in a block, its scope includes that block. If the
107 definition occurs in a function block, the scope extends to any blocks contained
108 within the defining one, unless a contained block introduces a different binding
109 for the name.
110
111 .. index:: single: environment
112
113 When a name is used in a code block, it is resolved using the nearest enclosing
114 scope. The set of all such scopes visible to a code block is called the block's
115 :dfn:`environment`.
116
117 .. index::
118 single: NameError (built-in exception)
119 single: UnboundLocalError
120
121 When a name is not found at all, a :exc:`NameError` exception is raised.
122 If the current scope is a function scope, and the name refers to a local
123 variable that has not yet been bound to a value at the point where the name is
124 used, an :exc:`UnboundLocalError` exception is raised.
125 :exc:`UnboundLocalError` is a subclass of :exc:`NameError`.
126
127 If a name binding operation occurs anywhere within a code block, all uses of the
128 name within the block are treated as references to the current block. This can
129 lead to errors when a name is used within a block before it is bound. This rule
130 is subtle. Python lacks declarations and allows name binding operations to
131 occur anywhere within a code block. The local variables of a code block can be
132 determined by scanning the entire text of the block for name binding operations.
133 See :ref:`the FAQ entry on UnboundLocalError <faq-unboundlocalerror>`
134 for examples.
135
136 If the :keyword:`global` statement occurs within a block, all uses of the names
137 specified in the statement refer to the bindings of those names in the top-level
138 namespace. Names are resolved in the top-level namespace by searching the
139 global namespace, i.e. the namespace of the module containing the code block,
140 and the builtins namespace, the namespace of the module :mod:`builtins`. The
141 global namespace is searched first. If the names are not found there, the
142 builtins namespace is searched next. If the names are also not found in the
143 builtins namespace, new variables are created in the global namespace.
144 The global statement must precede all uses of the listed names.
145
146 The :keyword:`global` statement has the same scope as a name binding operation
147 in the same block. If the nearest enclosing scope for a free variable contains
148 a global statement, the free variable is treated as a global.
149
150 .. XXX say more about "nonlocal" semantics here
151
152 The :keyword:`nonlocal` statement causes corresponding names to refer
153 to previously bound variables in the nearest enclosing function scope.
154 :exc:`SyntaxError` is raised at compile time if the given name does not
155 exist in any enclosing function scope. :ref:`Type parameters <type-params>`
156 cannot be rebound with the :keyword:`!nonlocal` statement.
157
158 .. index:: pair: module; __main__
159
160 The namespace for a module is automatically created the first time a module is
161 imported. The main module for a script is always called :mod:`__main__`.
162
163 Class definition blocks and arguments to :func:`exec` and :func:`eval` are
164 special in the context of name resolution.
165 A class definition is an executable statement that may use and define names.
166 These references follow the normal rules for name resolution with an exception
167 that unbound local variables are looked up in the global namespace.
168 The namespace of the class definition becomes the attribute dictionary of
169 the class. The scope of names defined in a class block is limited to the
170 class block; it does not extend to the code blocks of methods. This includes
171 comprehensions and generator expressions, but it does not include
172 :ref:`annotation scopes <annotation-scopes>`,
173 which have access to their enclosing class scopes.
174 This means that the following will fail::
175
176 class A:
177 a = 42
178 b = list(a + i for i in range(10))
179
180 However, the following will succeed::
181
182 class A:
183 type Alias = Nested
184 class Nested: pass
185
186 print(A.Alias.__value__) # <type 'A.Nested'>
187
188 .. _annotation-scopes:
189
190 Annotation scopes
191 -----------------
192
193 :term:`Annotations <annotation>`, :ref:`type parameter lists <type-params>`
194 and :keyword:`type` statements
195 introduce *annotation scopes*, which behave mostly like function scopes,
196 but with some exceptions discussed below.
197
198 Annotation scopes are used in the following contexts:
199
200 * :term:`Function annotations <function annotation>`.
201 * :term:`Variable annotations <variable annotation>`.
202 * Type parameter lists for :ref:`generic type aliases <generic-type-aliases>`.
203 * Type parameter lists for :ref:`generic functions <generic-functions>`.
204 A generic function's annotations are
205 executed within the annotation scope, but its defaults and decorators are not.
206 * Type parameter lists for :ref:`generic classes <generic-classes>`.
207 A generic class's base classes and
208 keyword arguments are executed within the annotation scope, but its decorators are not.
209 * The bounds, constraints, and default values for type parameters
210 (:ref:`lazily evaluated <lazy-evaluation>`).
211 * The value of type aliases (:ref:`lazily evaluated <lazy-evaluation>`).
212
213 Annotation scopes differ from function scopes in the following ways:
214
215 * Annotation scopes have access to their enclosing class namespace.
216 If an annotation scope is immediately within a class scope, or within another
217 annotation scope that is immediately within a class scope, the code in the
218 annotation scope can use names defined in the class scope as if it were
219 executed directly within the class body. This contrasts with regular
220 functions defined within classes, which cannot access names defined in the class scope.
221 * Expressions in annotation scopes cannot contain :keyword:`yield`, ``yield from``,
222 :keyword:`await`, or :token:`:= <python-grammar:assignment_expression>`
223 expressions. (These expressions are allowed in other scopes contained within the
224 annotation scope.)
225 * Names defined in annotation scopes cannot be rebound with :keyword:`nonlocal`
226 statements in inner scopes. This includes only type parameters, as no other
227 syntactic elements that can appear within annotation scopes can introduce new names.
228 * While annotation scopes have an internal name, that name is not reflected in the
229 :term:`qualified name` of objects defined within the scope.
230 Instead, the :attr:`~definition.__qualname__`
231 of such objects is as if the object were defined in the enclosing scope.
232
233 .. versionadded:: 3.12
234 Annotation scopes were introduced in Python 3.12 as part of :pep:`695`.
235
236 .. versionchanged:: 3.13
237 Annotation scopes are also used for type parameter defaults, as
238 introduced by :pep:`696`.
239
240 .. versionchanged:: 3.14
241 Annotation scopes are now also used for annotations, as specified in
242 :pep:`649` and :pep:`749`.
243
244 .. _lazy-evaluation:
245
246 Lazy evaluation
247 ---------------
248
249 Most annotation scopes are *lazily evaluated*. This includes annotations,
250 the values of type aliases created through the :keyword:`type` statement, and
251 the bounds, constraints, and default values of type
252 variables created through the :ref:`type parameter syntax <type-params>`.
253 This means that they are not evaluated when the type alias or type variable is
254 created, or when the object carrying annotations is created. Instead, they
255 are only evaluated when necessary, for example when the ``__value__``
256 attribute on a type alias is accessed.
257
258 Example:
259
260 .. doctest::
261
262 >>> type Alias = 1/0
263 >>> Alias.__value__
264 Traceback (most recent call last):
265 ...
266 ZeroDivisionError: division by zero
267 >>> def func[T: 1/0](): pass
268 >>> T = func.__type_params__[0]
269 >>> T.__bound__
270 Traceback (most recent call last):
271 ...
272 ZeroDivisionError: division by zero
273
274 Here the exception is raised only when the ``__value__`` attribute
275 of the type alias or the ``__bound__`` attribute of the type variable
276 is accessed.
277
278 This behavior is primarily useful for references to types that have not
279 yet been defined when the type alias or type variable is created. For example,
280 lazy evaluation enables creation of mutually recursive type aliases::
281
282 from typing import Literal
283
284 type SimpleExpr = int | Parenthesized
285 type Parenthesized = tuple[Literal["("], Expr, Literal[")"]]
286 type Expr = SimpleExpr | tuple[SimpleExpr, Literal["+", "-"], Expr]
287
288 Lazily evaluated values are evaluated in :ref:`annotation scope <annotation-scopes>`,
289 which means that names that appear inside the lazily evaluated value are looked up
290 as if they were used in the immediately enclosing scope.
291
292 .. versionadded:: 3.12
293
294 .. _restrict_exec:
295
296 Builtins and restricted execution
297 ---------------------------------
298
299 .. index:: pair: restricted; execution
300
301 .. impl-detail::
302
303 Users should not touch ``__builtins__``; it is strictly an implementation
304 detail. Users wanting to override values in the builtins namespace should
305 :keyword:`import` the :mod:`builtins` module and modify its
306 attributes appropriately.
307
308 The builtins namespace associated with the execution of a code block
309 is actually found by looking up the name ``__builtins__`` in its
310 global namespace; this should be a dictionary or a module (in the
311 latter case the module's dictionary is used). By default, when in the
312 :mod:`__main__` module, ``__builtins__`` is the built-in module
313 :mod:`builtins`; when in any other module, ``__builtins__`` is an
314 alias for the dictionary of the :mod:`builtins` module itself.
315
316
317 .. _dynamic-features:
318
319 Interaction with dynamic features
320 ---------------------------------
321
322 Name resolution of free variables occurs at runtime, not at compile time.
323 This means that the following code will print 42::
324
325 i = 10
326 def f():
327 print(i)
328 i = 42
329 f()
330
331 .. XXX from * also invalid with relative imports (at least currently)
332
333 The :func:`eval` and :func:`exec` functions do not have access to the full
334 environment for resolving names. Names may be resolved in the local and global
335 namespaces of the caller. Free variables are not resolved in the nearest
336 enclosing namespace, but in the global namespace. [#]_ The :func:`exec` and
337 :func:`eval` functions have optional arguments to override the global and local
338 namespace. If only one namespace is specified, it is used for both.
339
340 .. XXX(ncoghlan) above is only accurate for string execution. When executing code objects,
341 closure cells may now be passed explicitly to resolve co_freevars references.
342 Docs issue: https://github.com/python/cpython/issues/122826
343
344 .. _exceptions:
345
346 Exceptions
347 ==========
348
349 .. index:: single: exception
350
351 .. index::
352 single: raise an exception
353 single: handle an exception
354 single: exception handler
355 single: errors
356 single: error handling
357
358 Exceptions are a means of breaking out of the normal flow of control of a code
359 block in order to handle errors or other exceptional conditions. An exception
360 is *raised* at the point where the error is detected; it may be *handled* by the
361 surrounding code block or by any code block that directly or indirectly invoked
362 the code block where the error occurred.
363
364 The Python interpreter raises an exception when it detects a run-time error
365 (such as division by zero). A Python program can also explicitly raise an
366 exception with the :keyword:`raise` statement. Exception handlers are specified
367 with the :keyword:`try` ... :keyword:`except` statement. The :keyword:`finally`
368 clause of such a statement can be used to specify cleanup code which does not
369 handle the exception, but is executed whether an exception occurred or not in
370 the preceding code.
371
372 .. index:: single: termination model
373
374 Python uses the "termination" model of error handling: an exception handler can
375 find out what happened and continue execution at an outer level, but it cannot
376 repair the cause of the error and retry the failing operation (except by
377 re-entering the offending piece of code from the top).
378
379 .. index:: single: SystemExit (built-in exception)
380
381 When an exception is not handled at all, the interpreter terminates execution of
382 the program, or returns to its interactive main loop. In either case, it prints
383 a stack traceback, except when the exception is :exc:`SystemExit`.
384
385 Exceptions are identified by class instances. The :keyword:`except` clause is
386 selected depending on the class of the instance: it must reference the class of
387 the instance or a :term:`non-virtual base class <abstract base class>` thereof.
388 The instance can be received by the handler and can carry additional information
389 about the exceptional condition.
390
391 .. note::
392
393 Exception messages are not part of the Python API. Their contents may change
394 from one version of Python to the next without warning and should not be
395 relied on by code which will run under multiple versions of the interpreter.
396
397 See also the description of the :keyword:`try` statement in section :ref:`try`
398 and :keyword:`raise` statement in section :ref:`raise`.
399
400
401 .. _execcomponents:
402
403 Runtime Components
404 ==================
405
406 General Computing Model
407 -----------------------
408
409 Python's execution model does not operate in a vacuum. It runs on
410 a host machine and through that host's runtime environment, including
411 its operating system (OS), if there is one. When a program runs,
412 the conceptual layers of how it runs on the host look something
413 like this:
414
415 | **host machine**
416 | **process** (global resources)
417 | **thread** (runs machine code)
418
419 Each process represents a program running on the host. Think of each
420 process itself as the data part of its program. Think of the process'
421 threads as the execution part of the program. This distinction will
422 be important to understand the conceptual Python runtime.
423
424 The process, as the data part, is the execution context in which the
425 program runs. It mostly consists of the set of resources assigned to
426 the program by the host, including memory, signals, file handles,
427 sockets, and environment variables.
428
429 Processes are isolated and independent from one another. (The same
430 is true for hosts.) The host manages the process' access to its
431 assigned resources, in addition to coordinating between processes.
432
433 Each thread represents the actual execution of the program's machine
434 code, running relative to the resources assigned to the program's
435 process. It's strictly up to the host how and when that execution
436 takes place.
437
438 From the point of view of Python, a program always starts with exactly
439 one thread. However, the program may grow to run in multiple
440 simultaneous threads. Not all hosts support multiple threads per
441 process, but most do. Unlike processes, threads in a process are not
442 isolated and independent from one another. Specifically, all threads
443 in a process share all of the process' resources.
444
445 The fundamental point of threads is that each one does *run*
446 independently, at the same time as the others. That may be only
447 conceptually at the same time ("concurrently") or physically
448 ("in parallel"). Either way, the threads effectively run
449 at a non-synchronized rate.
450
451 .. note::
452
453 That non-synchronized rate means none of the process' memory is
454 guaranteed to stay consistent for the code running in any given
455 thread. Thus multi-threaded programs must take care to coordinate
456 access to intentionally shared resources. Likewise, they must take
457 care to be absolutely diligent about not accessing any *other*
458 resources in multiple threads; otherwise two threads running at the
459 same time might accidentally interfere with each other's use of some
460 shared data. All this is true for both Python programs and the
461 Python runtime.
462
463 The cost of this broad, unstructured requirement is the tradeoff for
464 the kind of raw concurrency that threads provide. The alternative
465 to the required discipline generally means dealing with
466 non-deterministic bugs and data corruption.
467
468 Python Runtime Model
469 --------------------
470
471 The same conceptual layers apply to each Python program, with some
472 extra data layers specific to Python:
473
474 | **host machine**
475 | **process** (global resources)
476 | Python global runtime (*state*)
477 | Python interpreter (*state*)
478 | **thread** (runs Python bytecode and "C-API")
479 | Python thread *state*
480
481 At the conceptual level: when a Python program starts, it looks exactly
482 like that diagram, with one of each. The runtime may grow to include
483 multiple interpreters, and each interpreter may grow to include
484 multiple thread states.
485
486 .. note::
487
488 A Python implementation won't necessarily implement the runtime
489 layers distinctly or even concretely. The only exception is places
490 where distinct layers are directly specified or exposed to users,
491 like through the :mod:`threading` module.
492
493 .. note::
494
495 The initial interpreter is typically called the "main" interpreter.
496 Some Python implementations, like CPython, assign special roles
497 to the main interpreter.
498
499 Likewise, the host thread where the runtime was initialized is known
500 as the "main" thread. It may be different from the process' initial
501 thread, though they are often the same. In some cases "main thread"
502 may be even more specific and refer to the initial thread state.
503 A Python runtime might assign specific responsibilities
504 to the main thread, such as handling signals.
505
506 As a whole, the Python runtime consists of the global runtime state,
507 interpreters, and thread states. The runtime ensures all that state
508 stays consistent over its lifetime, particularly when used with
509 multiple host threads.
510
511 The global runtime, at the conceptual level, is just a set of
512 interpreters. While those interpreters are otherwise isolated and
513 independent from one another, they may share some data or other
514 resources. The runtime is responsible for managing these global
515 resources safely. The actual nature and management of these resources
516 is implementation-specific. Ultimately, the external utility of the
517 global runtime is limited to managing interpreters.
518
519 In contrast, an "interpreter" is conceptually what we would normally
520 think of as the (full-featured) "Python runtime". When machine code
521 executing in a host thread interacts with the Python runtime, it calls
522 into Python in the context of a specific interpreter.
523
524 .. note::
525
526 The term "interpreter" here is not the same as the "bytecode
527 interpreter", which is what regularly runs in threads, executing
528 compiled Python code.
529
530 In an ideal world, "Python runtime" would refer to what we currently
531 call "interpreter". However, it's been called "interpreter" at least
532 since introduced in 1997 (`CPython:a027efa5b`_).
533
534 .. _CPython:a027efa5b: https://github.com/python/cpython/commit/a027efa5b
535
536 Each interpreter completely encapsulates all of the non-process-global,
537 non-thread-specific state needed for the Python runtime to work.
538 Notably, the interpreter's state persists between uses. It includes
539 fundamental data like :data:`sys.modules`. The runtime ensures
540 multiple threads using the same interpreter will safely
541 share it between them.
542
543 A Python implementation may support using multiple interpreters at the
544 same time in the same process. They are independent and isolated from
545 one another. For example, each interpreter has its own
546 :data:`sys.modules`.
547
548 For thread-specific runtime state, each interpreter has a set of thread
549 states, which it manages, in the same way the global runtime contains
550 a set of interpreters. It can have thread states for as many host
551 threads as it needs. It may even have multiple thread states for
552 the same host thread, though that isn't as common.
553
554 Each thread state, conceptually, has all the thread-specific runtime
555 data an interpreter needs to operate in one host thread. The thread
556 state includes the current raised exception and the thread's Python
557 call stack. It may include other thread-specific resources.
558
559 .. note::
560
561 The term "Python thread" can sometimes refer to a thread state, but
562 normally it means a thread created using the :mod:`threading` module.
563
564 Each thread state, over its lifetime, is always tied to exactly one
565 interpreter and exactly one host thread. It will only ever be used in
566 that thread and with that interpreter.
567
568 Multiple thread states may be tied to the same host thread, whether for
569 different interpreters or even the same interpreter. However, for any
570 given host thread, only one of the thread states tied to it can be used
571 by the thread at a time.
572
573 Thread states are isolated and independent from one another and don't
574 share any data, except for possibly sharing an interpreter and objects
575 or other resources belonging to that interpreter.
576
577 Once a program is running, new Python threads can be created using the
578 :mod:`threading` module (on platforms and Python implementations that
579 support threads). Additional processes can be created using the
580 :mod:`os`, :mod:`subprocess`, and :mod:`multiprocessing` modules.
581 Interpreters can be created and used with the
582 :mod:`~concurrent.interpreters` module. Coroutines (async) can
583 be run using :mod:`asyncio` in each interpreter, typically only
584 in a single thread (often the main thread).
585
586
587 .. rubric:: Footnotes
588
589 .. [#] This limitation occurs because the code that is executed by these operations
590 is not available at the time the module is compiled.