Debian Codegraph @

Doc/reference/executionmodel.rst

raw
1
2 .. _execmodel:
3
4 ***************
5 Execution model
6 ***************
7
8 .. index::
9 single: execution model
10 pair: code; block
11
12 .. _prog_structure:
13
14 Structure of a program
15 ======================
16
17 .. index:: block
18
19 A Python program is constructed from code blocks.
20 A :dfn:`block` is a piece of Python program text that is executed as a unit.
21 The following are blocks: a module, a function body, and a class definition.
22 Each command typed interactively is a block. A script file (a file given as
23 standard input to the interpreter or specified as a command line argument to the
24 interpreter) is a code block. A script command (a command specified on the
25 interpreter command line with the :option:`-c` option) is a code block.
26 A module run as a top level script (as module ``__main__``) from the command
27 line using a :option:`-m` argument is also a code block. The string
28 argument passed to the built-in functions :func:`eval` and :func:`exec` is a
29 code block.
30
31 .. index:: pair: execution; frame
32
33 A code block is executed in an :dfn:`execution frame`. A frame contains some
34 administrative information (used for debugging) and determines where and how
35 execution continues after the code block's execution has completed.
36
37 .. _naming:
38
39 Naming and binding
40 ==================
41
42 .. index::
43 single: namespace
44 single: scope
45
46 .. _bind_names:
47
48 Binding of names
49 ----------------
50
51 .. index::
52 single: name
53 pair: binding; name
54
55 :dfn:`Names` refer to objects. Names are introduced by name binding operations.
56
57 .. index:: single: from; import statement
58
59 The following constructs bind names:
60
61 * formal parameters to functions,
62 * class definitions,
63 * function definitions,
64 * assignment expressions,
65 * :ref:`targets <assignment>` that are identifiers if occurring in
66 an assignment:
67
68 + :keyword:`for` loop header,
69 + after :keyword:`!as` in a :keyword:`with` statement, :keyword:`except`
70 clause, :keyword:`except* <except_star>` clause, or in the as-pattern in structural pattern matching,
71 + in a capture pattern in structural pattern matching
72
73 * :keyword:`import` statements.
74 * :keyword:`type` statements.
75 * :ref:`type parameter lists <type-params>`.
76
77 The :keyword:`!import` statement of the form ``from ... import *`` binds all
78 names defined in the imported module, except those beginning with an underscore.
79 This form may only be used at the module level.
80
81 A target occurring in a :keyword:`del` statement is also considered bound for
82 this purpose (though the actual semantics are to unbind the name).
83
84 Each assignment or import statement occurs within a block defined by a class or
85 function definition or at the module level (the top-level code block).
86
87 .. index:: pair: free; variable
88
89 If a name is bound in a block, it is a local variable of that block, unless
90 declared as :keyword:`nonlocal` or :keyword:`global`. If a name is bound at
91 the module level, it is a global variable. (The variables of the module code
92 block are local and global.) If a variable is used in a code block but not
93 defined there, it is a :term:`free variable`.
94
95 Each occurrence of a name in the program text refers to the :dfn:`binding` of
96 that name established by the following name resolution rules.
97
98 .. _resolve_names:
99
100 Resolution of names
101 -------------------
102
103 .. index:: scope
104
105 A :dfn:`scope` defines the visibility of a name within a block. If a local
106 variable is defined in a block, its scope includes that block. If the
107 definition occurs in a function block, the scope extends to any blocks contained
108 within the defining one, unless a contained block introduces a different binding
109 for the name.
110
111 .. index:: single: environment
112
113 When a name is used in a code block, it is resolved using the nearest enclosing
114 scope. The set of all such scopes visible to a code block is called the block's
115 :dfn:`environment`.
116
117 .. index::
118 single: NameError (built-in exception)
119 single: UnboundLocalError
120
121 When a name is not found at all, a :exc:`NameError` exception is raised.
122 If the current scope is a function scope, and the name refers to a local
123 variable that has not yet been bound to a value at the point where the name is
124 used, an :exc:`UnboundLocalError` exception is raised.
125 :exc:`UnboundLocalError` is a subclass of :exc:`NameError`.
126
127 If a name binding operation occurs anywhere within a code block, all uses of the
128 name within the block are treated as references to the current block. This can
129 lead to errors when a name is used within a block before it is bound. This rule
130 is subtle. Python lacks declarations and allows name binding operations to
131 occur anywhere within a code block. The local variables of a code block can be
132 determined by scanning the entire text of the block for name binding operations.
133 See :ref:`the FAQ entry on UnboundLocalError <faq-unboundlocalerror>`
134 for examples.
135
136 If the :keyword:`global` statement occurs within a block, all uses of the names
137 specified in the statement refer to the bindings of those names in the top-level
138 namespace. Names are resolved in the top-level namespace by searching the
139 global namespace, i.e. the namespace of the module containing the code block,
140 and the builtins namespace, the namespace of the module :mod:`builtins`. The
141 global namespace is searched first. If the names are not found there, the
142 builtins namespace is searched next. If the names are also not found in the
143 builtins namespace, new variables are created in the global namespace.
144 The global statement must precede all uses of the listed names.
145
146 The :keyword:`global` statement has the same scope as a name binding operation
147 in the same block. If the nearest enclosing scope for a free variable contains
148 a global statement, the free variable is treated as a global.
149
150 .. XXX say more about "nonlocal" semantics here
151
152 The :keyword:`nonlocal` statement causes corresponding names to refer
153 to previously bound variables in the nearest enclosing function scope.
154 :exc:`SyntaxError` is raised at compile time if the given name does not
155 exist in any enclosing function scope. :ref:`Type parameters <type-params>`
156 cannot be rebound with the :keyword:`!nonlocal` statement.
157
158 .. index:: pair: module; __main__
159
160 The namespace for a module is automatically created the first time a module is
161 imported. The main module for a script is always called :mod:`__main__`.
162
163 Class definition blocks and arguments to :func:`exec` and :func:`eval` are
164 special in the context of name resolution.
165 A class definition is an executable statement that may use and define names.
166 These references follow the normal rules for name resolution with an exception
167 that unbound local variables are looked up in the global namespace.
168 The namespace of the class definition becomes the attribute dictionary of
169 the class. The scope of names defined in a class block is limited to the
170 class block; it does not extend to the code blocks of methods. This includes
171 comprehensions and generator expressions, but it does not include
172 :ref:`annotation scopes <annotation-scopes>`,
173 which have access to their enclosing class scopes.
174 This means that the following will fail::
175
176 class A:
177 a = 42
178 b = list(a + i for i in range(10))
179
180 However, the following will succeed::
181
182 class A:
183 type Alias = Nested
184 class Nested: pass
185
186 print(A.Alias.__value__) # <type 'A.Nested'>
187
188 .. _annotation-scopes:
189
190 Annotation scopes
191 -----------------
192
193 :ref:`Type parameter lists <type-params>` and :keyword:`type` statements
194 introduce *annotation scopes*, which behave mostly like function scopes,
195 but with some exceptions discussed below. :term:`Annotations <annotation>`
196 currently do not use annotation scopes, but they are expected to use
197 annotation scopes in Python 3.13 when :pep:`649` is implemented.
198
199 Annotation scopes are used in the following contexts:
200
201 * Type parameter lists for :ref:`generic type aliases <generic-type-aliases>`.
202 * Type parameter lists for :ref:`generic functions <generic-functions>`.
203 A generic function's annotations are
204 executed within the annotation scope, but its defaults and decorators are not.
205 * Type parameter lists for :ref:`generic classes <generic-classes>`.
206 A generic class's base classes and
207 keyword arguments are executed within the annotation scope, but its decorators are not.
208 * The bounds, constraints, and default values for type parameters
209 (:ref:`lazily evaluated <lazy-evaluation>`).
210 * The value of type aliases (:ref:`lazily evaluated <lazy-evaluation>`).
211
212 Annotation scopes differ from function scopes in the following ways:
213
214 * Annotation scopes have access to their enclosing class namespace.
215 If an annotation scope is immediately within a class scope, or within another
216 annotation scope that is immediately within a class scope, the code in the
217 annotation scope can use names defined in the class scope as if it were
218 executed directly within the class body. This contrasts with regular
219 functions defined within classes, which cannot access names defined in the class scope.
220 * Expressions in annotation scopes cannot contain :keyword:`yield`, ``yield from``,
221 :keyword:`await`, or :token:`:= <python-grammar:assignment_expression>`
222 expressions. (These expressions are allowed in other scopes contained within the
223 annotation scope.)
224 * Names defined in annotation scopes cannot be rebound with :keyword:`nonlocal`
225 statements in inner scopes. This includes only type parameters, as no other
226 syntactic elements that can appear within annotation scopes can introduce new names.
227 * While annotation scopes have an internal name, that name is not reflected in the
228 :term:`qualified name` of objects defined within the scope.
229 Instead, the :attr:`~definition.__qualname__`
230 of such objects is as if the object were defined in the enclosing scope.
231
232 .. versionadded:: 3.12
233 Annotation scopes were introduced in Python 3.12 as part of :pep:`695`.
234
235 .. versionchanged:: 3.13
236 Annotation scopes are also used for type parameter defaults, as
237 introduced by :pep:`696`.
238
239 .. _lazy-evaluation:
240
241 Lazy evaluation
242 ---------------
243
244 The values of type aliases created through the :keyword:`type` statement are
245 *lazily evaluated*. The same applies to the bounds, constraints, and default values of type
246 variables created through the :ref:`type parameter syntax <type-params>`.
247 This means that they are not evaluated when the type alias or type variable is
248 created. Instead, they are only evaluated when doing so is necessary to resolve
249 an attribute access.
250
251 Example:
252
253 .. doctest::
254
255 >>> type Alias = 1/0
256 >>> Alias.__value__
257 Traceback (most recent call last):
258 ...
259 ZeroDivisionError: division by zero
260 >>> def func[T: 1/0](): pass
261 >>> T = func.__type_params__[0]
262 >>> T.__bound__
263 Traceback (most recent call last):
264 ...
265 ZeroDivisionError: division by zero
266
267 Here the exception is raised only when the ``__value__`` attribute
268 of the type alias or the ``__bound__`` attribute of the type variable
269 is accessed.
270
271 This behavior is primarily useful for references to types that have not
272 yet been defined when the type alias or type variable is created. For example,
273 lazy evaluation enables creation of mutually recursive type aliases::
274
275 from typing import Literal
276
277 type SimpleExpr = int | Parenthesized
278 type Parenthesized = tuple[Literal["("], Expr, Literal[")"]]
279 type Expr = SimpleExpr | tuple[SimpleExpr, Literal["+", "-"], Expr]
280
281 Lazily evaluated values are evaluated in :ref:`annotation scope <annotation-scopes>`,
282 which means that names that appear inside the lazily evaluated value are looked up
283 as if they were used in the immediately enclosing scope.
284
285 .. versionadded:: 3.12
286
287 .. _restrict_exec:
288
289 Builtins and restricted execution
290 ---------------------------------
291
292 .. index:: pair: restricted; execution
293
294 .. impl-detail::
295
296 Users should not touch ``__builtins__``; it is strictly an implementation
297 detail. Users wanting to override values in the builtins namespace should
298 :keyword:`import` the :mod:`builtins` module and modify its
299 attributes appropriately.
300
301 The builtins namespace associated with the execution of a code block
302 is actually found by looking up the name ``__builtins__`` in its
303 global namespace; this should be a dictionary or a module (in the
304 latter case the module's dictionary is used). By default, when in the
305 :mod:`__main__` module, ``__builtins__`` is the built-in module
306 :mod:`builtins`; when in any other module, ``__builtins__`` is an
307 alias for the dictionary of the :mod:`builtins` module itself.
308
309
310 .. _dynamic-features:
311
312 Interaction with dynamic features
313 ---------------------------------
314
315 Name resolution of free variables occurs at runtime, not at compile time.
316 This means that the following code will print 42::
317
318 i = 10
319 def f():
320 print(i)
321 i = 42
322 f()
323
324 .. XXX from * also invalid with relative imports (at least currently)
325
326 The :func:`eval` and :func:`exec` functions do not have access to the full
327 environment for resolving names. Names may be resolved in the local and global
328 namespaces of the caller. Free variables are not resolved in the nearest
329 enclosing namespace, but in the global namespace. [#]_ The :func:`exec` and
330 :func:`eval` functions have optional arguments to override the global and local
331 namespace. If only one namespace is specified, it is used for both.
332
333 .. XXX(ncoghlan) above is only accurate for string execution. When executing code objects,
334 closure cells may now be passed explicitly to resolve co_freevars references.
335 Docs issue: https://github.com/python/cpython/issues/122826
336
337 .. _exceptions:
338
339 Exceptions
340 ==========
341
342 .. index:: single: exception
343
344 .. index::
345 single: raise an exception
346 single: handle an exception
347 single: exception handler
348 single: errors
349 single: error handling
350
351 Exceptions are a means of breaking out of the normal flow of control of a code
352 block in order to handle errors or other exceptional conditions. An exception
353 is *raised* at the point where the error is detected; it may be *handled* by the
354 surrounding code block or by any code block that directly or indirectly invoked
355 the code block where the error occurred.
356
357 The Python interpreter raises an exception when it detects a run-time error
358 (such as division by zero). A Python program can also explicitly raise an
359 exception with the :keyword:`raise` statement. Exception handlers are specified
360 with the :keyword:`try` ... :keyword:`except` statement. The :keyword:`finally`
361 clause of such a statement can be used to specify cleanup code which does not
362 handle the exception, but is executed whether an exception occurred or not in
363 the preceding code.
364
365 .. index:: single: termination model
366
367 Python uses the "termination" model of error handling: an exception handler can
368 find out what happened and continue execution at an outer level, but it cannot
369 repair the cause of the error and retry the failing operation (except by
370 re-entering the offending piece of code from the top).
371
372 .. index:: single: SystemExit (built-in exception)
373
374 When an exception is not handled at all, the interpreter terminates execution of
375 the program, or returns to its interactive main loop. In either case, it prints
376 a stack traceback, except when the exception is :exc:`SystemExit`.
377
378 Exceptions are identified by class instances. The :keyword:`except` clause is
379 selected depending on the class of the instance: it must reference the class of
380 the instance or a :term:`non-virtual base class <abstract base class>` thereof.
381 The instance can be received by the handler and can carry additional information
382 about the exceptional condition.
383
384 .. note::
385
386 Exception messages are not part of the Python API. Their contents may change
387 from one version of Python to the next without warning and should not be
388 relied on by code which will run under multiple versions of the interpreter.
389
390 See also the description of the :keyword:`try` statement in section :ref:`try`
391 and :keyword:`raise` statement in section :ref:`raise`.
392
393
394 .. _execcomponents:
395
396 Runtime Components
397 ==================
398
399 General Computing Model
400 -----------------------
401
402 Python's execution model does not operate in a vacuum. It runs on
403 a host machine and through that host's runtime environment, including
404 its operating system (OS), if there is one. When a program runs,
405 the conceptual layers of how it runs on the host look something
406 like this:
407
408 | **host machine**
409 | **process** (global resources)
410 | **thread** (runs machine code)
411
412 Each process represents a program running on the host. Think of each
413 process itself as the data part of its program. Think of the process'
414 threads as the execution part of the program. This distinction will
415 be important to understand the conceptual Python runtime.
416
417 The process, as the data part, is the execution context in which the
418 program runs. It mostly consists of the set of resources assigned to
419 the program by the host, including memory, signals, file handles,
420 sockets, and environment variables.
421
422 Processes are isolated and independent from one another. (The same
423 is true for hosts.) The host manages the process' access to its
424 assigned resources, in addition to coordinating between processes.
425
426 Each thread represents the actual execution of the program's machine
427 code, running relative to the resources assigned to the program's
428 process. It's strictly up to the host how and when that execution
429 takes place.
430
431 From the point of view of Python, a program always starts with exactly
432 one thread. However, the program may grow to run in multiple
433 simultaneous threads. Not all hosts support multiple threads per
434 process, but most do. Unlike processes, threads in a process are not
435 isolated and independent from one another. Specifically, all threads
436 in a process share all of the process' resources.
437
438 The fundamental point of threads is that each one does *run*
439 independently, at the same time as the others. That may be only
440 conceptually at the same time ("concurrently") or physically
441 ("in parallel"). Either way, the threads effectively run
442 at a non-synchronized rate.
443
444 .. note::
445
446 That non-synchronized rate means none of the process' memory is
447 guaranteed to stay consistent for the code running in any given
448 thread. Thus multi-threaded programs must take care to coordinate
449 access to intentionally shared resources. Likewise, they must take
450 care to be absolutely diligent about not accessing any *other*
451 resources in multiple threads; otherwise two threads running at the
452 same time might accidentally interfere with each other's use of some
453 shared data. All this is true for both Python programs and the
454 Python runtime.
455
456 The cost of this broad, unstructured requirement is the tradeoff for
457 the kind of raw concurrency that threads provide. The alternative
458 to the required discipline generally means dealing with
459 non-deterministic bugs and data corruption.
460
461 Python Runtime Model
462 --------------------
463
464 The same conceptual layers apply to each Python program, with some
465 extra data layers specific to Python:
466
467 | **host machine**
468 | **process** (global resources)
469 | Python global runtime (*state*)
470 | Python interpreter (*state*)
471 | **thread** (runs Python bytecode and "C-API")
472 | Python thread *state*
473
474 At the conceptual level: when a Python program starts, it looks exactly
475 like that diagram, with one of each. The runtime may grow to include
476 multiple interpreters, and each interpreter may grow to include
477 multiple thread states.
478
479 .. note::
480
481 A Python implementation won't necessarily implement the runtime
482 layers distinctly or even concretely. The only exception is places
483 where distinct layers are directly specified or exposed to users,
484 like through the :mod:`threading` module.
485
486 .. note::
487
488 The initial interpreter is typically called the "main" interpreter.
489 Some Python implementations, like CPython, assign special roles
490 to the main interpreter.
491
492 Likewise, the host thread where the runtime was initialized is known
493 as the "main" thread. It may be different from the process' initial
494 thread, though they are often the same. In some cases "main thread"
495 may be even more specific and refer to the initial thread state.
496 A Python runtime might assign specific responsibilities
497 to the main thread, such as handling signals.
498
499 As a whole, the Python runtime consists of the global runtime state,
500 interpreters, and thread states. The runtime ensures all that state
501 stays consistent over its lifetime, particularly when used with
502 multiple host threads.
503
504 The global runtime, at the conceptual level, is just a set of
505 interpreters. While those interpreters are otherwise isolated and
506 independent from one another, they may share some data or other
507 resources. The runtime is responsible for managing these global
508 resources safely. The actual nature and management of these resources
509 is implementation-specific. Ultimately, the external utility of the
510 global runtime is limited to managing interpreters.
511
512 In contrast, an "interpreter" is conceptually what we would normally
513 think of as the (full-featured) "Python runtime". When machine code
514 executing in a host thread interacts with the Python runtime, it calls
515 into Python in the context of a specific interpreter.
516
517 .. note::
518
519 The term "interpreter" here is not the same as the "bytecode
520 interpreter", which is what regularly runs in threads, executing
521 compiled Python code.
522
523 In an ideal world, "Python runtime" would refer to what we currently
524 call "interpreter". However, it's been called "interpreter" at least
525 since introduced in 1997 (`CPython:a027efa5b`_).
526
527 .. _CPython:a027efa5b: https://github.com/python/cpython/commit/a027efa5b
528
529 Each interpreter completely encapsulates all of the non-process-global,
530 non-thread-specific state needed for the Python runtime to work.
531 Notably, the interpreter's state persists between uses. It includes
532 fundamental data like :data:`sys.modules`. The runtime ensures
533 multiple threads using the same interpreter will safely
534 share it between them.
535
536 A Python implementation may support using multiple interpreters at the
537 same time in the same process. They are independent and isolated from
538 one another. For example, each interpreter has its own
539 :data:`sys.modules`.
540
541 For thread-specific runtime state, each interpreter has a set of thread
542 states, which it manages, in the same way the global runtime contains
543 a set of interpreters. It can have thread states for as many host
544 threads as it needs. It may even have multiple thread states for
545 the same host thread, though that isn't as common.
546
547 Each thread state, conceptually, has all the thread-specific runtime
548 data an interpreter needs to operate in one host thread. The thread
549 state includes the current raised exception and the thread's Python
550 call stack. It may include other thread-specific resources.
551
552 .. note::
553
554 The term "Python thread" can sometimes refer to a thread state, but
555 normally it means a thread created using the :mod:`threading` module.
556
557 Each thread state, over its lifetime, is always tied to exactly one
558 interpreter and exactly one host thread. It will only ever be used in
559 that thread and with that interpreter.
560
561 Multiple thread states may be tied to the same host thread, whether for
562 different interpreters or even the same interpreter. However, for any
563 given host thread, only one of the thread states tied to it can be used
564 by the thread at a time.
565
566 Thread states are isolated and independent from one another and don't
567 share any data, except for possibly sharing an interpreter and objects
568 or other resources belonging to that interpreter.
569
570 Once a program is running, new Python threads can be created using the
571 :mod:`threading` module (on platforms and Python implementations that
572 support threads). Additional processes can be created using the
573 :mod:`os`, :mod:`subprocess`, and :mod:`multiprocessing` modules.
574 Coroutines (async) can
575 be run using :mod:`asyncio` in each interpreter, typically only
576 in a single thread (often the main thread).
577
578
579 .. rubric:: Footnotes
580
581 .. [#] This limitation occurs because the code that is executed by these operations
582 is not available at the time the module is compiled.