Debian Codegraph @

Doc/c-api/subinterpreters.rst

raw
1 .. highlight:: c
2
3 .. _sub-interpreter-support:
4
5 Multiple interpreters in a Python process
6 =========================================
7
8 While in most uses, you will only embed a single Python interpreter, there
9 are cases where you need to create several independent interpreters in the
10 same process and perhaps even in the same thread. Sub-interpreters allow
11 you to do that.
12
13 The "main" interpreter is the first one created when the runtime initializes.
14 It is usually the only Python interpreter in a process. Unlike sub-interpreters,
15 the main interpreter has unique process-global responsibilities like signal
16 handling. It is also responsible for execution during runtime initialization and
17 is usually the active interpreter during runtime finalization. The
18 :c:func:`PyInterpreterState_Main` function returns a pointer to its state.
19
20 You can switch between sub-interpreters using the :c:func:`PyThreadState_Swap`
21 function. You can create and destroy them using the following functions:
22
23
24 .. c:type:: PyInterpreterConfig
25
26 Structure containing most parameters to configure a sub-interpreter.
27 Its values are used only in :c:func:`Py_NewInterpreterFromConfig` and
28 never modified by the runtime.
29
30 .. versionadded:: 3.12
31
32 Structure fields:
33
34 .. c:member:: int use_main_obmalloc
35
36 If this is ``0`` then the sub-interpreter will use its own
37 "object" allocator state.
38 Otherwise it will use (share) the main interpreter's.
39
40 If this is ``0`` then
41 :c:member:`~PyInterpreterConfig.check_multi_interp_extensions`
42 must be ``1`` (non-zero).
43 If this is ``1`` then :c:member:`~PyInterpreterConfig.gil`
44 must not be :c:macro:`PyInterpreterConfig_OWN_GIL`.
45
46 .. c:member:: int allow_fork
47
48 If this is ``0`` then the runtime will not support forking the
49 process in any thread where the sub-interpreter is currently active.
50 Otherwise fork is unrestricted.
51
52 Note that the :mod:`subprocess` module still works
53 when fork is disallowed.
54
55 .. c:member:: int allow_exec
56
57 If this is ``0`` then the runtime will not support replacing the
58 current process via exec (e.g. :func:`os.execv`) in any thread
59 where the sub-interpreter is currently active.
60 Otherwise exec is unrestricted.
61
62 Note that the :mod:`subprocess` module still works
63 when exec is disallowed.
64
65 .. c:member:: int allow_threads
66
67 If this is ``0`` then the sub-interpreter's :mod:`threading` module
68 won't create threads.
69 Otherwise threads are allowed.
70
71 .. c:member:: int allow_daemon_threads
72
73 If this is ``0`` then the sub-interpreter's :mod:`threading` module
74 won't create daemon threads.
75 Otherwise daemon threads are allowed (as long as
76 :c:member:`~PyInterpreterConfig.allow_threads` is non-zero).
77
78 .. c:member:: int check_multi_interp_extensions
79
80 If this is ``0`` then all extension modules may be imported,
81 including legacy (single-phase init) modules,
82 in any thread where the sub-interpreter is currently active.
83 Otherwise only multi-phase init extension modules
84 (see :pep:`489`) may be imported.
85 (Also see :c:macro:`Py_mod_multiple_interpreters`.)
86
87 This must be ``1`` (non-zero) if
88 :c:member:`~PyInterpreterConfig.use_main_obmalloc` is ``0``.
89
90 .. c:member:: int gil
91
92 This determines the operation of the GIL for the sub-interpreter.
93 It may be one of the following:
94
95 .. c:namespace:: NULL
96
97 .. c:macro:: PyInterpreterConfig_DEFAULT_GIL
98
99 Use the default selection (:c:macro:`PyInterpreterConfig_SHARED_GIL`).
100
101 .. c:macro:: PyInterpreterConfig_SHARED_GIL
102
103 Use (share) the main interpreter's GIL.
104
105 .. c:macro:: PyInterpreterConfig_OWN_GIL
106
107 Use the sub-interpreter's own GIL.
108
109 If this is :c:macro:`PyInterpreterConfig_OWN_GIL` then
110 :c:member:`PyInterpreterConfig.use_main_obmalloc` must be ``0``.
111
112
113 .. c:function:: PyStatus Py_NewInterpreterFromConfig(PyThreadState **tstate_p, const PyInterpreterConfig *config)
114
115 .. index::
116 pair: module; builtins
117 pair: module; __main__
118 pair: module; sys
119 single: stdout (in module sys)
120 single: stderr (in module sys)
121 single: stdin (in module sys)
122
123 Create a new sub-interpreter. This is an (almost) totally separate environment
124 for the execution of Python code. In particular, the new interpreter has
125 separate, independent versions of all imported modules, including the
126 fundamental modules :mod:`builtins`, :mod:`__main__` and :mod:`sys`. The
127 table of loaded modules (``sys.modules``) and the module search path
128 (``sys.path``) are also separate. The new environment has no ``sys.argv``
129 variable. It has new standard I/O stream file objects ``sys.stdin``,
130 ``sys.stdout`` and ``sys.stderr`` (however these refer to the same underlying
131 file descriptors).
132
133 The given *config* controls the options with which the interpreter
134 is initialized.
135
136 Upon success, *tstate_p* will be set to the first :term:`thread state`
137 created in the new sub-interpreter. This thread state is
138 :term:`attached <attached thread state>`.
139 Note that no actual thread is created; see the discussion of thread states
140 below. If creation of the new interpreter is unsuccessful,
141 *tstate_p* is set to ``NULL``;
142 no exception is set since the exception state is stored in the
143 :term:`attached thread state`, which might not exist.
144
145 Like all other Python/C API functions, an :term:`attached thread state`
146 must be present before calling this function, but it might be detached upon
147 returning. On success, the returned thread state will be :term:`attached <attached thread state>`.
148 If the sub-interpreter is created with its own :term:`GIL` then the
149 :term:`attached thread state` of the calling interpreter will be detached.
150 When the function returns, the new interpreter's :term:`thread state`
151 will be :term:`attached <attached thread state>` to the current thread and
152 the previous interpreter's :term:`attached thread state` will remain detached.
153
154 .. versionadded:: 3.12
155
156 Sub-interpreters are most effective when isolated from each other,
157 with certain functionality restricted::
158
159 PyInterpreterConfig config = {
160 .use_main_obmalloc = 0,
161 .allow_fork = 0,
162 .allow_exec = 0,
163 .allow_threads = 1,
164 .allow_daemon_threads = 0,
165 .check_multi_interp_extensions = 1,
166 .gil = PyInterpreterConfig_OWN_GIL,
167 };
168 PyThreadState *tstate = NULL;
169 PyStatus status = Py_NewInterpreterFromConfig(&tstate, &config);
170 if (PyStatus_Exception(status)) {
171 Py_ExitStatusException(status);
172 }
173
174 Note that the config is used only briefly and does not get modified.
175 During initialization the config's values are converted into various
176 :c:type:`PyInterpreterState` values. A read-only copy of the config
177 may be stored internally on the :c:type:`PyInterpreterState`.
178
179 .. index::
180 single: Py_FinalizeEx (C function)
181 single: Py_Initialize (C function)
182
183 Extension modules are shared between (sub-)interpreters as follows:
184
185 * For modules using multi-phase initialization,
186 e.g. :c:func:`PyModule_FromDefAndSpec`, a separate module object is
187 created and initialized for each interpreter.
188 Only C-level static and global variables are shared between these
189 module objects.
190
191 * For modules using legacy
192 :ref:`single-phase initialization <single-phase-initialization>`,
193 e.g. :c:func:`PyModule_Create`, the first time a particular extension
194 is imported, it is initialized normally, and a (shallow) copy of its
195 module's dictionary is squirreled away.
196 When the same extension is imported by another (sub-)interpreter, a new
197 module is initialized and filled with the contents of this copy; the
198 extension's ``init`` function is not called.
199 Objects in the module's dictionary thus end up shared across
200 (sub-)interpreters, which might cause unwanted behavior (see
201 `Bugs and caveats`_ below).
202
203 Note that this is different from what happens when an extension is
204 imported after the interpreter has been completely re-initialized by
205 calling :c:func:`Py_FinalizeEx` and :c:func:`Py_Initialize`; in that
206 case, the extension's ``initmodule`` function *is* called again.
207 As with multi-phase initialization, this means that only C-level static
208 and global variables are shared between these modules.
209
210 .. index:: single: close (in module os)
211
212
213 .. c:function:: PyThreadState* Py_NewInterpreter(void)
214
215 .. index::
216 pair: module; builtins
217 pair: module; __main__
218 pair: module; sys
219 single: stdout (in module sys)
220 single: stderr (in module sys)
221 single: stdin (in module sys)
222
223 Create a new sub-interpreter. This is essentially just a wrapper
224 around :c:func:`Py_NewInterpreterFromConfig` with a config that
225 preserves the existing behavior. The result is an unisolated
226 sub-interpreter that shares the main interpreter's GIL, allows
227 fork/exec, allows daemon threads, and allows single-phase init
228 modules.
229
230
231 .. c:function:: void Py_EndInterpreter(PyThreadState *tstate)
232
233 .. index:: single: Py_FinalizeEx (C function)
234
235 Destroy the (sub-)interpreter represented by the given :term:`thread state`.
236 The given thread state must be :term:`attached <attached thread state>`.
237 When the call returns, there will be no :term:`attached thread state`.
238 All thread states associated with this interpreter are destroyed.
239
240 :c:func:`Py_FinalizeEx` will destroy all sub-interpreters that
241 haven't been explicitly destroyed at that point.
242
243
244 .. _per-interpreter-gil:
245
246 A per-interpreter GIL
247 ---------------------
248
249 .. versionadded:: 3.12
250
251 Using :c:func:`Py_NewInterpreterFromConfig` you can create
252 a sub-interpreter that is completely isolated from other interpreters,
253 including having its own GIL. The most important benefit of this
254 isolation is that such an interpreter can execute Python code without
255 being blocked by other interpreters or blocking any others. Thus a
256 single Python process can truly take advantage of multiple CPU cores
257 when running Python code. The isolation also encourages a different
258 approach to concurrency than that of just using threads.
259 (See :pep:`554` and :pep:`684`.)
260
261 Using an isolated interpreter requires vigilance in preserving that
262 isolation. That especially means not sharing any objects or mutable
263 state without guarantees about thread-safety. Even objects that are
264 otherwise immutable (e.g. ``None``, ``(1, 5)``) can't normally be shared
265 because of the refcount. One simple but less-efficient approach around
266 this is to use a global lock around all use of some state (or object).
267 Alternately, effectively immutable objects (like integers or strings)
268 can be made safe in spite of their refcounts by making them :term:`immortal`.
269 In fact, this has been done for the builtin singletons, small integers,
270 and a number of other builtin objects.
271
272 If you preserve isolation then you will have access to proper multi-core
273 computing without the complications that come with free-threading.
274 Failure to preserve isolation will expose you to the full consequences
275 of free-threading, including races and hard-to-debug crashes.
276
277 Aside from that, one of the main challenges of using multiple isolated
278 interpreters is how to communicate between them safely (not break
279 isolation) and efficiently. The runtime and stdlib do not provide
280 any standard approach to this yet. A future stdlib module would help
281 mitigate the effort of preserving isolation and expose effective tools
282 for communicating (and sharing) data between interpreters.
283
284
285 Bugs and caveats
286 ----------------
287
288 Because sub-interpreters (and the main interpreter) are part of the same
289 process, the insulation between them isn't perfect --- for example, using
290 low-level file operations like :func:`os.close` they can
291 (accidentally or maliciously) affect each other's open files. Because of the
292 way extensions are shared between (sub-)interpreters, some extensions may not
293 work properly; this is especially likely when using single-phase initialization
294 or (static) global variables.
295 It is possible to insert objects created in one sub-interpreter into
296 a namespace of another (sub-)interpreter; this should be avoided if possible.
297
298 Special care should be taken to avoid sharing user-defined functions,
299 methods, instances or classes between sub-interpreters, since import
300 operations executed by such objects may affect the wrong (sub-)interpreter's
301 dictionary of loaded modules. It is equally important to avoid sharing
302 objects from which the above are reachable.
303
304 Also note that combining this functionality with ``PyGILState_*`` APIs
305 is delicate, because these APIs assume a bijection between Python thread states
306 and OS-level threads, an assumption broken by the presence of sub-interpreters.
307 It is highly recommended that you don't switch sub-interpreters between a pair
308 of matching :c:func:`PyGILState_Ensure` and :c:func:`PyGILState_Release` calls.
309 Furthermore, extensions (such as :mod:`ctypes`) using these APIs to allow calling
310 of Python code from non-Python created threads will probably be broken when using
311 sub-interpreters.
312
313
314 High-level APIs
315 ---------------
316
317 .. c:type:: PyInterpreterState
318
319 This data structure represents the state shared by a number of cooperating
320 threads. Threads belonging to the same interpreter share their module
321 administration and a few other internal items. There are no public members in
322 this structure.
323
324 Threads belonging to different interpreters initially share nothing, except
325 process state like available memory, open file descriptors and such. The global
326 interpreter lock is also shared by all threads, regardless of to which
327 interpreter they belong.
328
329 .. versionchanged:: 3.12
330
331 :pep:`684` introduced the possibility
332 of a :ref:`per-interpreter GIL <per-interpreter-gil>`.
333 See :c:func:`Py_NewInterpreterFromConfig`.
334
335
336 .. c:function:: PyInterpreterState* PyInterpreterState_Get(void)
337
338 Get the current interpreter.
339
340 Issue a fatal error if there is no :term:`attached thread state`.
341 It cannot return NULL.
342
343 .. versionadded:: 3.9
344
345
346 .. c:function:: int64_t PyInterpreterState_GetID(PyInterpreterState *interp)
347
348 Return the interpreter's unique ID. If there was any error in doing
349 so then ``-1`` is returned and an error is set.
350
351 The caller must have an :term:`attached thread state`.
352
353 .. versionadded:: 3.7
354
355
356 .. c:function:: PyObject* PyInterpreterState_GetDict(PyInterpreterState *interp)
357
358 Return a dictionary in which interpreter-specific data may be stored.
359 If this function returns ``NULL`` then no exception has been raised and
360 the caller should assume no interpreter-specific dict is available.
361
362 This is not a replacement for :c:func:`PyModule_GetState()`, which
363 extensions should use to store interpreter-specific state information.
364
365 The returned dictionary is borrowed from the interpreter and is valid until
366 interpreter shutdown.
367
368 .. versionadded:: 3.8
369
370
371 .. c:type:: PyObject* (*_PyFrameEvalFunction)(PyThreadState *tstate, _PyInterpreterFrame *frame, int throwflag)
372
373 Type of a frame evaluation function.
374
375 The *throwflag* parameter is used by the ``throw()`` method of generators:
376 if non-zero, handle the current exception.
377
378 .. versionchanged:: 3.9
379 The function now takes a *tstate* parameter.
380
381 .. versionchanged:: 3.11
382 The *frame* parameter changed from ``PyFrameObject*`` to ``_PyInterpreterFrame*``.
383
384
385 .. c:function:: _PyFrameEvalFunction _PyInterpreterState_GetEvalFrameFunc(PyInterpreterState *interp)
386
387 Get the frame evaluation function.
388
389 See the :pep:`523` "Adding a frame evaluation API to CPython".
390
391 .. versionadded:: 3.9
392
393
394 .. c:function:: void _PyInterpreterState_SetEvalFrameFunc(PyInterpreterState *interp, _PyFrameEvalFunction eval_frame)
395
396 Set the frame evaluation function.
397
398 See the :pep:`523` "Adding a frame evaluation API to CPython".
399
400 .. versionadded:: 3.9
401
402
403 Low-level APIs
404 --------------
405
406 All of the following functions must be called after :c:func:`Py_Initialize`.
407
408 .. versionchanged:: 3.7
409 :c:func:`Py_Initialize()` now initializes the :term:`GIL`
410 and sets an :term:`attached thread state`.
411
412
413 .. c:function:: PyInterpreterState* PyInterpreterState_New()
414
415 Create a new interpreter state object. An :term:`attached thread state` is not needed,
416 but may optionally exist if it is necessary to serialize calls to this
417 function.
418
419 .. audit-event:: cpython.PyInterpreterState_New "" c.PyInterpreterState_New
420
421
422 .. c:function:: void PyInterpreterState_Clear(PyInterpreterState *interp)
423
424 Reset all information in an interpreter state object. There must be
425 an :term:`attached thread state` for the interpreter.
426
427 .. audit-event:: cpython.PyInterpreterState_Clear "" c.PyInterpreterState_Clear
428
429
430 .. c:function:: void PyInterpreterState_Delete(PyInterpreterState *interp)
431
432 Destroy an interpreter state object. There **should not** be an
433 :term:`attached thread state` for the target interpreter. The interpreter
434 state must have been reset with a previous call to :c:func:`PyInterpreterState_Clear`.
435
436
437 .. _advanced-debugging:
438
439 Advanced debugger support
440 -------------------------
441
442 These functions are only intended to be used by advanced debugging tools.
443
444
445 .. c:function:: PyInterpreterState* PyInterpreterState_Head()
446
447 Return the interpreter state object at the head of the list of all such objects.
448
449
450 .. c:function:: PyInterpreterState* PyInterpreterState_Main()
451
452 Return the main interpreter state object.
453
454
455 .. c:function:: PyInterpreterState* PyInterpreterState_Next(PyInterpreterState *interp)
456
457 Return the next interpreter state object after *interp* from the list of all
458 such objects.
459
460
461 .. c:function:: PyThreadState * PyInterpreterState_ThreadHead(PyInterpreterState *interp)
462
463 Return the pointer to the first :c:type:`PyThreadState` object in the list of
464 threads associated with the interpreter *interp*.
465
466
467 .. c:function:: PyThreadState* PyThreadState_Next(PyThreadState *tstate)
468
469 Return the next thread state object after *tstate* from the list of all such
470 objects belonging to the same :c:type:`PyInterpreterState` object.