| 1 |
|
| 2 |
.. _execmodel: |
| 3 |
|
| 4 |
*************** |
| 5 |
Execution model |
| 6 |
*************** |
| 7 |
|
| 8 |
.. index:: |
| 9 |
single: execution model |
| 10 |
pair: code; block |
| 11 |
|
| 12 |
.. _prog_structure: |
| 13 |
|
| 14 |
Structure of a program |
| 15 |
====================== |
| 16 |
|
| 17 |
.. index:: block |
| 18 |
|
| 19 |
A Python program is constructed from code blocks. |
| 20 |
A :dfn:`block` is a piece of Python program text that is executed as a unit. |
| 21 |
The following are blocks: a module, a function body, and a class definition. |
| 22 |
Each command typed interactively is a block. A script file (a file given as |
| 23 |
standard input to the interpreter or specified as a command line argument to the |
| 24 |
interpreter) is a code block. A script command (a command specified on the |
| 25 |
interpreter command line with the :option:`-c` option) is a code block. |
| 26 |
A module run as a top level script (as module ``__main__``) from the command |
| 27 |
line using a :option:`-m` argument is also a code block. The string |
| 28 |
argument passed to the built-in functions :func:`eval` and :func:`exec` is a |
| 29 |
code block. |
| 30 |
|
| 31 |
.. index:: pair: execution; frame |
| 32 |
|
| 33 |
A code block is executed in an :dfn:`execution frame`. A frame contains some |
| 34 |
administrative information (used for debugging) and determines where and how |
| 35 |
execution continues after the code block's execution has completed. |
| 36 |
|
| 37 |
.. _naming: |
| 38 |
|
| 39 |
Naming and binding |
| 40 |
================== |
| 41 |
|
| 42 |
.. index:: |
| 43 |
single: namespace |
| 44 |
single: scope |
| 45 |
|
| 46 |
.. _bind_names: |
| 47 |
|
| 48 |
Binding of names |
| 49 |
---------------- |
| 50 |
|
| 51 |
.. index:: |
| 52 |
single: name |
| 53 |
pair: binding; name |
| 54 |
|
| 55 |
:dfn:`Names` refer to objects. Names are introduced by name binding operations. |
| 56 |
|
| 57 |
.. index:: single: from; import statement |
| 58 |
|
| 59 |
The following constructs bind names: |
| 60 |
|
| 61 |
* formal parameters to functions, |
| 62 |
* class definitions, |
| 63 |
* function definitions, |
| 64 |
* assignment expressions, |
| 65 |
* :ref:`targets <assignment>` that are identifiers if occurring in |
| 66 |
an assignment: |
| 67 |
|
| 68 |
+ :keyword:`for` loop header, |
| 69 |
+ after :keyword:`!as` in a :keyword:`with` statement, :keyword:`except` |
| 70 |
clause, :keyword:`except* <except_star>` clause, or in the as-pattern in structural pattern matching, |
| 71 |
+ in a capture pattern in structural pattern matching |
| 72 |
|
| 73 |
* :keyword:`import` statements. |
| 74 |
* :keyword:`type` statements. |
| 75 |
* :ref:`type parameter lists <type-params>`. |
| 76 |
|
| 77 |
The :keyword:`!import` statement of the form ``from ... import *`` binds all |
| 78 |
names defined in the imported module, except those beginning with an underscore. |
| 79 |
This form may only be used at the module level. |
| 80 |
|
| 81 |
A target occurring in a :keyword:`del` statement is also considered bound for |
| 82 |
this purpose (though the actual semantics are to unbind the name). |
| 83 |
|
| 84 |
Each assignment or import statement occurs within a block defined by a class or |
| 85 |
function definition or at the module level (the top-level code block). |
| 86 |
|
| 87 |
.. index:: pair: free; variable |
| 88 |
|
| 89 |
If a name is bound in a block, it is a local variable of that block, unless |
| 90 |
declared as :keyword:`nonlocal` or :keyword:`global`. If a name is bound at |
| 91 |
the module level, it is a global variable. (The variables of the module code |
| 92 |
block are local and global.) If a variable is used in a code block but not |
| 93 |
defined there, it is a :term:`free variable`. |
| 94 |
|
| 95 |
Each occurrence of a name in the program text refers to the :dfn:`binding` of |
| 96 |
that name established by the following name resolution rules. |
| 97 |
|
| 98 |
.. _resolve_names: |
| 99 |
|
| 100 |
Resolution of names |
| 101 |
------------------- |
| 102 |
|
| 103 |
.. index:: scope |
| 104 |
|
| 105 |
A :dfn:`scope` defines the visibility of a name within a block. If a local |
| 106 |
variable is defined in a block, its scope includes that block. If the |
| 107 |
definition occurs in a function block, the scope extends to any blocks contained |
| 108 |
within the defining one, unless a contained block introduces a different binding |
| 109 |
for the name. |
| 110 |
|
| 111 |
.. index:: single: environment |
| 112 |
|
| 113 |
When a name is used in a code block, it is resolved using the nearest enclosing |
| 114 |
scope. The set of all such scopes visible to a code block is called the block's |
| 115 |
:dfn:`environment`. |
| 116 |
|
| 117 |
.. index:: |
| 118 |
single: NameError (built-in exception) |
| 119 |
single: UnboundLocalError |
| 120 |
|
| 121 |
When a name is not found at all, a :exc:`NameError` exception is raised. |
| 122 |
If the current scope is a function scope, and the name refers to a local |
| 123 |
variable that has not yet been bound to a value at the point where the name is |
| 124 |
used, an :exc:`UnboundLocalError` exception is raised. |
| 125 |
:exc:`UnboundLocalError` is a subclass of :exc:`NameError`. |
| 126 |
|
| 127 |
If a name binding operation occurs anywhere within a code block, all uses of the |
| 128 |
name within the block are treated as references to the current block. This can |
| 129 |
lead to errors when a name is used within a block before it is bound. This rule |
| 130 |
is subtle. Python lacks declarations and allows name binding operations to |
| 131 |
occur anywhere within a code block. The local variables of a code block can be |
| 132 |
determined by scanning the entire text of the block for name binding operations. |
| 133 |
See :ref:`the FAQ entry on UnboundLocalError <faq-unboundlocalerror>` |
| 134 |
for examples. |
| 135 |
|
| 136 |
If the :keyword:`global` statement occurs within a block, all uses of the names |
| 137 |
specified in the statement refer to the bindings of those names in the top-level |
| 138 |
namespace. Names are resolved in the top-level namespace by searching the |
| 139 |
global namespace, i.e. the namespace of the module containing the code block, |
| 140 |
and the builtins namespace, the namespace of the module :mod:`builtins`. The |
| 141 |
global namespace is searched first. If the names are not found there, the |
| 142 |
builtins namespace is searched next. If the names are also not found in the |
| 143 |
builtins namespace, new variables are created in the global namespace. |
| 144 |
The global statement must precede all uses of the listed names. |
| 145 |
|
| 146 |
The :keyword:`global` statement has the same scope as a name binding operation |
| 147 |
in the same block. If the nearest enclosing scope for a free variable contains |
| 148 |
a global statement, the free variable is treated as a global. |
| 149 |
|
| 150 |
.. XXX say more about "nonlocal" semantics here |
| 151 |
|
| 152 |
The :keyword:`nonlocal` statement causes corresponding names to refer |
| 153 |
to previously bound variables in the nearest enclosing function scope. |
| 154 |
:exc:`SyntaxError` is raised at compile time if the given name does not |
| 155 |
exist in any enclosing function scope. :ref:`Type parameters <type-params>` |
| 156 |
cannot be rebound with the :keyword:`!nonlocal` statement. |
| 157 |
|
| 158 |
.. index:: pair: module; __main__ |
| 159 |
|
| 160 |
The namespace for a module is automatically created the first time a module is |
| 161 |
imported. The main module for a script is always called :mod:`__main__`. |
| 162 |
|
| 163 |
Class definition blocks and arguments to :func:`exec` and :func:`eval` are |
| 164 |
special in the context of name resolution. |
| 165 |
A class definition is an executable statement that may use and define names. |
| 166 |
These references follow the normal rules for name resolution with an exception |
| 167 |
that unbound local variables are looked up in the global namespace. |
| 168 |
The namespace of the class definition becomes the attribute dictionary of |
| 169 |
the class. The scope of names defined in a class block is limited to the |
| 170 |
class block; it does not extend to the code blocks of methods. This includes |
| 171 |
comprehensions and generator expressions, but it does not include |
| 172 |
:ref:`annotation scopes <annotation-scopes>`, |
| 173 |
which have access to their enclosing class scopes. |
| 174 |
This means that the following will fail:: |
| 175 |
|
| 176 |
class A: |
| 177 |
a = 42 |
| 178 |
b = list(a + i for i in range(10)) |
| 179 |
|
| 180 |
However, the following will succeed:: |
| 181 |
|
| 182 |
class A: |
| 183 |
type Alias = Nested |
| 184 |
class Nested: pass |
| 185 |
|
| 186 |
print(A.Alias.__value__) # <type 'A.Nested'> |
| 187 |
|
| 188 |
.. _annotation-scopes: |
| 189 |
|
| 190 |
Annotation scopes |
| 191 |
----------------- |
| 192 |
|
| 193 |
:term:`Annotations <annotation>`, :ref:`type parameter lists <type-params>` |
| 194 |
and :keyword:`type` statements |
| 195 |
introduce *annotation scopes*, which behave mostly like function scopes, |
| 196 |
but with some exceptions discussed below. |
| 197 |
|
| 198 |
Annotation scopes are used in the following contexts: |
| 199 |
|
| 200 |
* :term:`Function annotations <function annotation>`. |
| 201 |
* :term:`Variable annotations <variable annotation>`. |
| 202 |
* Type parameter lists for :ref:`generic type aliases <generic-type-aliases>`. |
| 203 |
* Type parameter lists for :ref:`generic functions <generic-functions>`. |
| 204 |
A generic function's annotations are |
| 205 |
executed within the annotation scope, but its defaults and decorators are not. |
| 206 |
* Type parameter lists for :ref:`generic classes <generic-classes>`. |
| 207 |
A generic class's base classes and |
| 208 |
keyword arguments are executed within the annotation scope, but its decorators are not. |
| 209 |
* The bounds, constraints, and default values for type parameters |
| 210 |
(:ref:`lazily evaluated <lazy-evaluation>`). |
| 211 |
* The value of type aliases (:ref:`lazily evaluated <lazy-evaluation>`). |
| 212 |
|
| 213 |
Annotation scopes differ from function scopes in the following ways: |
| 214 |
|
| 215 |
* Annotation scopes have access to their enclosing class namespace. |
| 216 |
If an annotation scope is immediately within a class scope, or within another |
| 217 |
annotation scope that is immediately within a class scope, the code in the |
| 218 |
annotation scope can use names defined in the class scope as if it were |
| 219 |
executed directly within the class body. This contrasts with regular |
| 220 |
functions defined within classes, which cannot access names defined in the class scope. |
| 221 |
* Expressions in annotation scopes cannot contain :keyword:`yield`, ``yield from``, |
| 222 |
:keyword:`await`, or :token:`:= <python-grammar:assignment_expression>` |
| 223 |
expressions. (These expressions are allowed in other scopes contained within the |
| 224 |
annotation scope.) |
| 225 |
* Names defined in annotation scopes cannot be rebound with :keyword:`nonlocal` |
| 226 |
statements in inner scopes. This includes only type parameters, as no other |
| 227 |
syntactic elements that can appear within annotation scopes can introduce new names. |
| 228 |
* While annotation scopes have an internal name, that name is not reflected in the |
| 229 |
:term:`qualified name` of objects defined within the scope. |
| 230 |
Instead, the :attr:`~definition.__qualname__` |
| 231 |
of such objects is as if the object were defined in the enclosing scope. |
| 232 |
|
| 233 |
.. versionadded:: 3.12 |
| 234 |
Annotation scopes were introduced in Python 3.12 as part of :pep:`695`. |
| 235 |
|
| 236 |
.. versionchanged:: 3.13 |
| 237 |
Annotation scopes are also used for type parameter defaults, as |
| 238 |
introduced by :pep:`696`. |
| 239 |
|
| 240 |
.. versionchanged:: 3.14 |
| 241 |
Annotation scopes are now also used for annotations, as specified in |
| 242 |
:pep:`649` and :pep:`749`. |
| 243 |
|
| 244 |
.. _lazy-evaluation: |
| 245 |
|
| 246 |
Lazy evaluation |
| 247 |
--------------- |
| 248 |
|
| 249 |
Most annotation scopes are *lazily evaluated*. This includes annotations, |
| 250 |
the values of type aliases created through the :keyword:`type` statement, and |
| 251 |
the bounds, constraints, and default values of type |
| 252 |
variables created through the :ref:`type parameter syntax <type-params>`. |
| 253 |
This means that they are not evaluated when the type alias or type variable is |
| 254 |
created, or when the object carrying annotations is created. Instead, they |
| 255 |
are only evaluated when necessary, for example when the ``__value__`` |
| 256 |
attribute on a type alias is accessed. |
| 257 |
|
| 258 |
Example: |
| 259 |
|
| 260 |
.. doctest:: |
| 261 |
|
| 262 |
>>> type Alias = 1/0 |
| 263 |
>>> Alias.__value__ |
| 264 |
Traceback (most recent call last): |
| 265 |
... |
| 266 |
ZeroDivisionError: division by zero |
| 267 |
>>> def func[T: 1/0](): pass |
| 268 |
>>> T = func.__type_params__[0] |
| 269 |
>>> T.__bound__ |
| 270 |
Traceback (most recent call last): |
| 271 |
... |
| 272 |
ZeroDivisionError: division by zero |
| 273 |
|
| 274 |
Here the exception is raised only when the ``__value__`` attribute |
| 275 |
of the type alias or the ``__bound__`` attribute of the type variable |
| 276 |
is accessed. |
| 277 |
|
| 278 |
This behavior is primarily useful for references to types that have not |
| 279 |
yet been defined when the type alias or type variable is created. For example, |
| 280 |
lazy evaluation enables creation of mutually recursive type aliases:: |
| 281 |
|
| 282 |
from typing import Literal |
| 283 |
|
| 284 |
type SimpleExpr = int | Parenthesized |
| 285 |
type Parenthesized = tuple[Literal["("], Expr, Literal[")"]] |
| 286 |
type Expr = SimpleExpr | tuple[SimpleExpr, Literal["+", "-"], Expr] |
| 287 |
|
| 288 |
Lazily evaluated values are evaluated in :ref:`annotation scope <annotation-scopes>`, |
| 289 |
which means that names that appear inside the lazily evaluated value are looked up |
| 290 |
as if they were used in the immediately enclosing scope. |
| 291 |
|
| 292 |
.. versionadded:: 3.12 |
| 293 |
|
| 294 |
.. _restrict_exec: |
| 295 |
|
| 296 |
Builtins and restricted execution |
| 297 |
--------------------------------- |
| 298 |
|
| 299 |
.. index:: pair: restricted; execution |
| 300 |
|
| 301 |
.. impl-detail:: |
| 302 |
|
| 303 |
Users should not touch ``__builtins__``; it is strictly an implementation |
| 304 |
detail. Users wanting to override values in the builtins namespace should |
| 305 |
:keyword:`import` the :mod:`builtins` module and modify its |
| 306 |
attributes appropriately. |
| 307 |
|
| 308 |
The builtins namespace associated with the execution of a code block |
| 309 |
is actually found by looking up the name ``__builtins__`` in its |
| 310 |
global namespace; this should be a dictionary or a module (in the |
| 311 |
latter case the module's dictionary is used). By default, when in the |
| 312 |
:mod:`__main__` module, ``__builtins__`` is the built-in module |
| 313 |
:mod:`builtins`; when in any other module, ``__builtins__`` is an |
| 314 |
alias for the dictionary of the :mod:`builtins` module itself. |
| 315 |
|
| 316 |
|
| 317 |
.. _dynamic-features: |
| 318 |
|
| 319 |
Interaction with dynamic features |
| 320 |
--------------------------------- |
| 321 |
|
| 322 |
Name resolution of free variables occurs at runtime, not at compile time. |
| 323 |
This means that the following code will print 42:: |
| 324 |
|
| 325 |
i = 10 |
| 326 |
def f(): |
| 327 |
print(i) |
| 328 |
i = 42 |
| 329 |
f() |
| 330 |
|
| 331 |
.. XXX from * also invalid with relative imports (at least currently) |
| 332 |
|
| 333 |
The :func:`eval` and :func:`exec` functions do not have access to the full |
| 334 |
environment for resolving names. Names may be resolved in the local and global |
| 335 |
namespaces of the caller. Free variables are not resolved in the nearest |
| 336 |
enclosing namespace, but in the global namespace. [#]_ The :func:`exec` and |
| 337 |
:func:`eval` functions have optional arguments to override the global and local |
| 338 |
namespace. If only one namespace is specified, it is used for both. |
| 339 |
|
| 340 |
.. XXX(ncoghlan) above is only accurate for string execution. When executing code objects, |
| 341 |
closure cells may now be passed explicitly to resolve co_freevars references. |
| 342 |
Docs issue: https://github.com/python/cpython/issues/122826 |
| 343 |
|
| 344 |
.. _exceptions: |
| 345 |
|
| 346 |
Exceptions |
| 347 |
========== |
| 348 |
|
| 349 |
.. index:: single: exception |
| 350 |
|
| 351 |
.. index:: |
| 352 |
single: raise an exception |
| 353 |
single: handle an exception |
| 354 |
single: exception handler |
| 355 |
single: errors |
| 356 |
single: error handling |
| 357 |
|
| 358 |
Exceptions are a means of breaking out of the normal flow of control of a code |
| 359 |
block in order to handle errors or other exceptional conditions. An exception |
| 360 |
is *raised* at the point where the error is detected; it may be *handled* by the |
| 361 |
surrounding code block or by any code block that directly or indirectly invoked |
| 362 |
the code block where the error occurred. |
| 363 |
|
| 364 |
The Python interpreter raises an exception when it detects a run-time error |
| 365 |
(such as division by zero). A Python program can also explicitly raise an |
| 366 |
exception with the :keyword:`raise` statement. Exception handlers are specified |
| 367 |
with the :keyword:`try` ... :keyword:`except` statement. The :keyword:`finally` |
| 368 |
clause of such a statement can be used to specify cleanup code which does not |
| 369 |
handle the exception, but is executed whether an exception occurred or not in |
| 370 |
the preceding code. |
| 371 |
|
| 372 |
.. index:: single: termination model |
| 373 |
|
| 374 |
Python uses the "termination" model of error handling: an exception handler can |
| 375 |
find out what happened and continue execution at an outer level, but it cannot |
| 376 |
repair the cause of the error and retry the failing operation (except by |
| 377 |
re-entering the offending piece of code from the top). |
| 378 |
|
| 379 |
.. index:: single: SystemExit (built-in exception) |
| 380 |
|
| 381 |
When an exception is not handled at all, the interpreter terminates execution of |
| 382 |
the program, or returns to its interactive main loop. In either case, it prints |
| 383 |
a stack traceback, except when the exception is :exc:`SystemExit`. |
| 384 |
|
| 385 |
Exceptions are identified by class instances. The :keyword:`except` clause is |
| 386 |
selected depending on the class of the instance: it must reference the class of |
| 387 |
the instance or a :term:`non-virtual base class <abstract base class>` thereof. |
| 388 |
The instance can be received by the handler and can carry additional information |
| 389 |
about the exceptional condition. |
| 390 |
|
| 391 |
.. note:: |
| 392 |
|
| 393 |
Exception messages are not part of the Python API. Their contents may change |
| 394 |
from one version of Python to the next without warning and should not be |
| 395 |
relied on by code which will run under multiple versions of the interpreter. |
| 396 |
|
| 397 |
See also the description of the :keyword:`try` statement in section :ref:`try` |
| 398 |
and :keyword:`raise` statement in section :ref:`raise`. |
| 399 |
|
| 400 |
|
| 401 |
.. _execcomponents: |
| 402 |
|
| 403 |
Runtime Components |
| 404 |
================== |
| 405 |
|
| 406 |
General Computing Model |
| 407 |
----------------------- |
| 408 |
|
| 409 |
Python's execution model does not operate in a vacuum. It runs on |
| 410 |
a host machine and through that host's runtime environment, including |
| 411 |
its operating system (OS), if there is one. When a program runs, |
| 412 |
the conceptual layers of how it runs on the host look something |
| 413 |
like this: |
| 414 |
|
| 415 |
| **host machine** |
| 416 |
| **process** (global resources) |
| 417 |
| **thread** (runs machine code) |
| 418 |
|
| 419 |
Each process represents a program running on the host. Think of each |
| 420 |
process itself as the data part of its program. Think of the process' |
| 421 |
threads as the execution part of the program. This distinction will |
| 422 |
be important to understand the conceptual Python runtime. |
| 423 |
|
| 424 |
The process, as the data part, is the execution context in which the |
| 425 |
program runs. It mostly consists of the set of resources assigned to |
| 426 |
the program by the host, including memory, signals, file handles, |
| 427 |
sockets, and environment variables. |
| 428 |
|
| 429 |
Processes are isolated and independent from one another. (The same |
| 430 |
is true for hosts.) The host manages the process' access to its |
| 431 |
assigned resources, in addition to coordinating between processes. |
| 432 |
|
| 433 |
Each thread represents the actual execution of the program's machine |
| 434 |
code, running relative to the resources assigned to the program's |
| 435 |
process. It's strictly up to the host how and when that execution |
| 436 |
takes place. |
| 437 |
|
| 438 |
From the point of view of Python, a program always starts with exactly |
| 439 |
one thread. However, the program may grow to run in multiple |
| 440 |
simultaneous threads. Not all hosts support multiple threads per |
| 441 |
process, but most do. Unlike processes, threads in a process are not |
| 442 |
isolated and independent from one another. Specifically, all threads |
| 443 |
in a process share all of the process' resources. |
| 444 |
|
| 445 |
The fundamental point of threads is that each one does *run* |
| 446 |
independently, at the same time as the others. That may be only |
| 447 |
conceptually at the same time ("concurrently") or physically |
| 448 |
("in parallel"). Either way, the threads effectively run |
| 449 |
at a non-synchronized rate. |
| 450 |
|
| 451 |
.. note:: |
| 452 |
|
| 453 |
That non-synchronized rate means none of the process' memory is |
| 454 |
guaranteed to stay consistent for the code running in any given |
| 455 |
thread. Thus multi-threaded programs must take care to coordinate |
| 456 |
access to intentionally shared resources. Likewise, they must take |
| 457 |
care to be absolutely diligent about not accessing any *other* |
| 458 |
resources in multiple threads; otherwise two threads running at the |
| 459 |
same time might accidentally interfere with each other's use of some |
| 460 |
shared data. All this is true for both Python programs and the |
| 461 |
Python runtime. |
| 462 |
|
| 463 |
The cost of this broad, unstructured requirement is the tradeoff for |
| 464 |
the kind of raw concurrency that threads provide. The alternative |
| 465 |
to the required discipline generally means dealing with |
| 466 |
non-deterministic bugs and data corruption. |
| 467 |
|
| 468 |
Python Runtime Model |
| 469 |
-------------------- |
| 470 |
|
| 471 |
The same conceptual layers apply to each Python program, with some |
| 472 |
extra data layers specific to Python: |
| 473 |
|
| 474 |
| **host machine** |
| 475 |
| **process** (global resources) |
| 476 |
| Python global runtime (*state*) |
| 477 |
| Python interpreter (*state*) |
| 478 |
| **thread** (runs Python bytecode and "C-API") |
| 479 |
| Python thread *state* |
| 480 |
|
| 481 |
At the conceptual level: when a Python program starts, it looks exactly |
| 482 |
like that diagram, with one of each. The runtime may grow to include |
| 483 |
multiple interpreters, and each interpreter may grow to include |
| 484 |
multiple thread states. |
| 485 |
|
| 486 |
.. note:: |
| 487 |
|
| 488 |
A Python implementation won't necessarily implement the runtime |
| 489 |
layers distinctly or even concretely. The only exception is places |
| 490 |
where distinct layers are directly specified or exposed to users, |
| 491 |
like through the :mod:`threading` module. |
| 492 |
|
| 493 |
.. note:: |
| 494 |
|
| 495 |
The initial interpreter is typically called the "main" interpreter. |
| 496 |
Some Python implementations, like CPython, assign special roles |
| 497 |
to the main interpreter. |
| 498 |
|
| 499 |
Likewise, the host thread where the runtime was initialized is known |
| 500 |
as the "main" thread. It may be different from the process' initial |
| 501 |
thread, though they are often the same. In some cases "main thread" |
| 502 |
may be even more specific and refer to the initial thread state. |
| 503 |
A Python runtime might assign specific responsibilities |
| 504 |
to the main thread, such as handling signals. |
| 505 |
|
| 506 |
As a whole, the Python runtime consists of the global runtime state, |
| 507 |
interpreters, and thread states. The runtime ensures all that state |
| 508 |
stays consistent over its lifetime, particularly when used with |
| 509 |
multiple host threads. |
| 510 |
|
| 511 |
The global runtime, at the conceptual level, is just a set of |
| 512 |
interpreters. While those interpreters are otherwise isolated and |
| 513 |
independent from one another, they may share some data or other |
| 514 |
resources. The runtime is responsible for managing these global |
| 515 |
resources safely. The actual nature and management of these resources |
| 516 |
is implementation-specific. Ultimately, the external utility of the |
| 517 |
global runtime is limited to managing interpreters. |
| 518 |
|
| 519 |
In contrast, an "interpreter" is conceptually what we would normally |
| 520 |
think of as the (full-featured) "Python runtime". When machine code |
| 521 |
executing in a host thread interacts with the Python runtime, it calls |
| 522 |
into Python in the context of a specific interpreter. |
| 523 |
|
| 524 |
.. note:: |
| 525 |
|
| 526 |
The term "interpreter" here is not the same as the "bytecode |
| 527 |
interpreter", which is what regularly runs in threads, executing |
| 528 |
compiled Python code. |
| 529 |
|
| 530 |
In an ideal world, "Python runtime" would refer to what we currently |
| 531 |
call "interpreter". However, it's been called "interpreter" at least |
| 532 |
since introduced in 1997 (`CPython:a027efa5b`_). |
| 533 |
|
| 534 |
.. _CPython:a027efa5b: https://github.com/python/cpython/commit/a027efa5b |
| 535 |
|
| 536 |
Each interpreter completely encapsulates all of the non-process-global, |
| 537 |
non-thread-specific state needed for the Python runtime to work. |
| 538 |
Notably, the interpreter's state persists between uses. It includes |
| 539 |
fundamental data like :data:`sys.modules`. The runtime ensures |
| 540 |
multiple threads using the same interpreter will safely |
| 541 |
share it between them. |
| 542 |
|
| 543 |
A Python implementation may support using multiple interpreters at the |
| 544 |
same time in the same process. They are independent and isolated from |
| 545 |
one another. For example, each interpreter has its own |
| 546 |
:data:`sys.modules`. |
| 547 |
|
| 548 |
For thread-specific runtime state, each interpreter has a set of thread |
| 549 |
states, which it manages, in the same way the global runtime contains |
| 550 |
a set of interpreters. It can have thread states for as many host |
| 551 |
threads as it needs. It may even have multiple thread states for |
| 552 |
the same host thread, though that isn't as common. |
| 553 |
|
| 554 |
Each thread state, conceptually, has all the thread-specific runtime |
| 555 |
data an interpreter needs to operate in one host thread. The thread |
| 556 |
state includes the current raised exception and the thread's Python |
| 557 |
call stack. It may include other thread-specific resources. |
| 558 |
|
| 559 |
.. note:: |
| 560 |
|
| 561 |
The term "Python thread" can sometimes refer to a thread state, but |
| 562 |
normally it means a thread created using the :mod:`threading` module. |
| 563 |
|
| 564 |
Each thread state, over its lifetime, is always tied to exactly one |
| 565 |
interpreter and exactly one host thread. It will only ever be used in |
| 566 |
that thread and with that interpreter. |
| 567 |
|
| 568 |
Multiple thread states may be tied to the same host thread, whether for |
| 569 |
different interpreters or even the same interpreter. However, for any |
| 570 |
given host thread, only one of the thread states tied to it can be used |
| 571 |
by the thread at a time. |
| 572 |
|
| 573 |
Thread states are isolated and independent from one another and don't |
| 574 |
share any data, except for possibly sharing an interpreter and objects |
| 575 |
or other resources belonging to that interpreter. |
| 576 |
|
| 577 |
Once a program is running, new Python threads can be created using the |
| 578 |
:mod:`threading` module (on platforms and Python implementations that |
| 579 |
support threads). Additional processes can be created using the |
| 580 |
:mod:`os`, :mod:`subprocess`, and :mod:`multiprocessing` modules. |
| 581 |
Interpreters can be created and used with the |
| 582 |
:mod:`~concurrent.interpreters` module. Coroutines (async) can |
| 583 |
be run using :mod:`asyncio` in each interpreter, typically only |
| 584 |
in a single thread (often the main thread). |
| 585 |
|
| 586 |
|
| 587 |
.. rubric:: Footnotes |
| 588 |
|
| 589 |
.. [#] This limitation occurs because the code that is executed by these operations |
| 590 |
is not available at the time the module is compiled. |