Debian Codegraph @

Doc/howto/isolating-extensions.rst

raw
1 .. highlight:: c
2
3 .. _isolating-extensions-howto:
4
5 ***************************
6 Isolating Extension Modules
7 ***************************
8
9 .. topic:: Abstract
10
11 Traditionally, state belonging to Python extension modules was kept in C
12 ``static`` variables, which have process-wide scope. This document
13 describes problems of such per-process state and shows a safer way:
14 per-module state.
15
16 The document also describes how to switch to per-module state where
17 possible. This transition involves allocating space for that state, potentially
18 switching from static types to heap types, and—perhaps most
19 importantly—accessing per-module state from code.
20
21
22 Who should read this
23 ====================
24
25 This guide is written for maintainers of :ref:`C-API <c-api-index>` extensions
26 who would like to make that extension safer to use in applications where
27 Python itself is used as a library.
28
29
30 Background
31 ==========
32
33 An *interpreter* is the context in which Python code runs. It contains
34 configuration (e.g. the import path) and runtime state (e.g. the set of
35 imported modules).
36
37 Python supports running multiple interpreters in one process. There are
38 two cases to think about—users may run interpreters:
39
40 - in sequence, with several :c:func:`Py_InitializeEx`/:c:func:`Py_FinalizeEx`
41 cycles, and
42 - in parallel, managing "sub-interpreters" using
43 :c:func:`Py_NewInterpreter`/:c:func:`Py_EndInterpreter`.
44
45 Both cases (and combinations of them) would be most useful when
46 embedding Python within a library. Libraries generally shouldn't make
47 assumptions about the application that uses them, which include
48 assuming a process-wide "main Python interpreter".
49
50 Historically, Python extension modules don't handle this use case well.
51 Many extension modules (and even some stdlib modules) use *per-process*
52 global state, because C ``static`` variables are extremely easy to use.
53 Thus, data that should be specific to an interpreter ends up being shared
54 between interpreters. Unless the extension developer is careful, it is very
55 easy to introduce edge cases that lead to crashes when a module is loaded in
56 more than one interpreter in the same process.
57
58 Unfortunately, *per-interpreter* state is not easy to achieve. Extension
59 authors tend to not keep multiple interpreters in mind when developing,
60 and it is currently cumbersome to test the behavior.
61
62 Enter Per-Module State
63 ----------------------
64
65 Instead of focusing on per-interpreter state, Python's C API is evolving
66 to better support the more granular *per-module* state.
67 This means that C-level data should be attached to a *module object*.
68 Each interpreter creates its own module object, keeping the data separate.
69 For testing the isolation, multiple module objects corresponding to a single
70 extension can even be loaded in a single interpreter.
71
72 Per-module state provides an easy way to think about lifetime and
73 resource ownership: the extension module will initialize when a
74 module object is created, and clean up when it's freed. In this regard,
75 a module is just like any other :c:expr:`PyObject *`; there are no "on
76 interpreter shutdown" hooks to think—or forget—about.
77
78 Note that there are use cases for different kinds of "globals":
79 per-process, per-interpreter, per-thread or per-task state.
80 With per-module state as the default, these are still possible,
81 but you should treat them as exceptional cases:
82 if you need them, you should give them additional care and testing.
83 (Note that this guide does not cover them.)
84
85
86 Isolated Module Objects
87 -----------------------
88
89 The key point to keep in mind when developing an extension module is
90 that several module objects can be created from a single shared library.
91 For example:
92
93 .. code-block:: pycon
94
95 >>> import sys
96 >>> import binascii
97 >>> old_binascii = binascii
98 >>> del sys.modules['binascii']
99 >>> import binascii # create a new module object
100 >>> old_binascii == binascii
101 False
102
103 As a rule of thumb, the two modules should be completely independent.
104 All objects and state specific to the module should be encapsulated
105 within the module object, not shared with other module objects, and
106 cleaned up when the module object is deallocated.
107 Since this just is a rule of thumb, exceptions are possible
108 (see `Managing Global State`_), but they will need more
109 thought and attention to edge cases.
110
111 While some modules could do with less stringent restrictions, isolated
112 modules make it easier to set clear expectations and guidelines that
113 work across a variety of use cases.
114
115
116 Surprising Edge Cases
117 ---------------------
118
119 Note that isolated modules do create some surprising edge cases. Most
120 notably, each module object will typically not share its classes and
121 exceptions with other similar modules. Continuing from the
122 `example above <Isolated Module Objects_>`__,
123 note that ``old_binascii.Error`` and ``binascii.Error`` are
124 separate objects. In the following code, the exception is *not* caught:
125
126 .. code-block:: pycon
127
128 >>> old_binascii.Error == binascii.Error
129 False
130 >>> try:
131 ... old_binascii.unhexlify(b'qwertyuiop')
132 ... except binascii.Error:
133 ... print('boo')
134 ...
135 Traceback (most recent call last):
136 File "<stdin>", line 2, in <module>
137 binascii.Error: Non-hexadecimal digit found
138
139 This is expected. Notice that pure-Python modules behave the same way:
140 it is a part of how Python works.
141
142 The goal is to make extension modules safe at the C level, not to make
143 hacks behave intuitively. Mutating ``sys.modules`` "manually" counts
144 as a hack.
145
146
147 Making Modules Safe with Multiple Interpreters
148 ==============================================
149
150
151 Managing Global State
152 ---------------------
153
154 Sometimes, the state associated with a Python module is not specific to that module, but
155 to the entire process (or something else "more global" than a module).
156 For example:
157
158 - The ``readline`` module manages *the* terminal.
159 - A module running on a circuit board wants to control *the* on-board
160 LED.
161
162 In these cases, the Python module should provide *access* to the global
163 state, rather than *own* it. If possible, write the module so that
164 multiple copies of it can access the state independently (along with
165 other libraries, whether for Python or other languages). If that is not
166 possible, consider explicit locking.
167
168 If it is necessary to use process-global state, the simplest way to
169 avoid issues with multiple interpreters is to explicitly prevent a
170 module from being loaded more than once per process—see
171 :ref:`isolating-extensions-optout`.
172
173
174 Managing Per-Module State
175 -------------------------
176
177 To use per-module state, use
178 :ref:`multi-phase extension module initialization <multi-phase-initialization>`.
179 This signals that your module supports multiple interpreters correctly.
180
181 Set ``PyModuleDef.m_size`` to a positive number to request that many
182 bytes of storage local to the module. Usually, this will be set to the
183 size of some module-specific ``struct``, which can store all of the
184 module's C-level state. In particular, it is where you should put
185 pointers to classes (including exceptions, but excluding static types)
186 and settings (e.g. ``csv``'s :py:data:`~csv.field_size_limit`)
187 which the C code needs to function.
188
189 .. note::
190 Another option is to store state in the module's ``__dict__``,
191 but you must avoid crashing when users modify ``__dict__`` from
192 Python code. This usually means error- and type-checking at the C level,
193 which is easy to get wrong and hard to test sufficiently.
194
195 However, if module state is not needed in C code, storing it in
196 ``__dict__`` only is a good idea.
197
198 If the module state includes ``PyObject`` pointers, the module object
199 must hold references to those objects and implement the module-level hooks
200 ``m_traverse``, ``m_clear`` and ``m_free``. These work like
201 ``tp_traverse``, ``tp_clear`` and ``tp_free`` of a class. Adding them will
202 require some work and make the code longer; this is the price for
203 modules which can be unloaded cleanly.
204
205 An example of a module with per-module state is currently available as
206 `xxlimited <https://github.com/python/cpython/blob/master/Modules/xxlimited.c>`__;
207 example module initialization shown at the bottom of the file.
208
209
210 .. _isolating-extensions-optout:
211
212 Opt-Out: Limiting to One Module Object per Process
213 --------------------------------------------------
214
215 A non-negative ``PyModuleDef.m_size`` signals that a module supports
216 multiple interpreters correctly. If this is not yet the case for your
217 module, you can explicitly make your module loadable only once per
218 process. For example::
219
220 // A process-wide flag
221 static int loaded = 0;
222
223 // Mutex to provide thread safety (only needed for free-threaded Python)
224 static PyMutex modinit_mutex = {0};
225
226 static int
227 exec_module(PyObject* module)
228 {
229 PyMutex_Lock(&modinit_mutex);
230 if (loaded) {
231 PyMutex_Unlock(&modinit_mutex);
232 PyErr_SetString(PyExc_ImportError,
233 "cannot load module more than once per process");
234 return -1;
235 }
236 loaded = 1;
237 PyMutex_Unlock(&modinit_mutex);
238 // ... rest of initialization
239 }
240
241
242 If your module's :c:member:`PyModuleDef.m_clear` function is able to prepare
243 for future re-initialization, it should clear the ``loaded`` flag.
244 In this case, your module won't support multiple instances existing
245 *concurrently*, but it will, for example, support being loaded after
246 Python runtime shutdown (:c:func:`Py_FinalizeEx`) and re-initialization
247 (:c:func:`Py_Initialize`).
248
249
250 Module State Access from Functions
251 ----------------------------------
252
253 Accessing the state from module-level functions is straightforward.
254 Functions get the module object as their first argument; for extracting
255 the state, you can use ``PyModule_GetState``::
256
257 static PyObject *
258 func(PyObject *module, PyObject *args)
259 {
260 my_struct *state = (my_struct*)PyModule_GetState(module);
261 if (state == NULL) {
262 return NULL;
263 }
264 // ... rest of logic
265 }
266
267 .. note::
268 ``PyModule_GetState`` may return ``NULL`` without setting an
269 exception if there is no module state, i.e. ``PyModuleDef.m_size`` was
270 zero. In your own module, you're in control of ``m_size``, so this is
271 easy to prevent.
272
273
274 Heap Types
275 ==========
276
277 Traditionally, types defined in C code are *static*; that is,
278 ``static PyTypeObject`` structures defined directly in code and
279 initialized using ``PyType_Ready()``.
280
281 Such types are necessarily shared across the process. Sharing them
282 between module objects requires paying attention to any state they own
283 or access. To limit the possible issues, static types are immutable at
284 the Python level: for example, you can't set ``str.myattribute = 123``.
285
286 .. impl-detail::
287 Sharing truly immutable objects between interpreters is fine,
288 as long as they don't provide access to mutable objects.
289 However, in CPython, every Python object has a mutable implementation
290 detail: the reference count. Changes to the refcount are guarded by the GIL.
291 Thus, code that shares any Python objects across interpreters implicitly
292 depends on CPython's current, process-wide GIL.
293
294 Because they are immutable and process-global, static types cannot access
295 "their" module state.
296 If any method of such a type requires access to module state,
297 the type must be converted to a *heap-allocated type*, or *heap type*
298 for short. These correspond more closely to classes created by Python's
299 ``class`` statement.
300
301 For new modules, using heap types by default is a good rule of thumb.
302
303
304 Changing Static Types to Heap Types
305 -----------------------------------
306
307 Static types can be converted to heap types, but note that
308 the heap type API was not designed for "lossless" conversion
309 from static types—that is, creating a type that works exactly like a given
310 static type.
311 So, when rewriting the class definition in a new API,
312 you are likely to unintentionally change a few details (e.g. pickleability
313 or inherited slots).
314 Always test the details that are important to you.
315
316 Watch out for the following two points in particular (but note that this is not
317 a comprehensive list):
318
319 * Unlike static types, heap type objects are mutable by default.
320 Use the :c:macro:`Py_TPFLAGS_IMMUTABLETYPE` flag to prevent mutability.
321 * Heap types inherit :c:member:`~PyTypeObject.tp_new` by default,
322 so it may become possible to instantiate them from Python code.
323 You can prevent this with the :c:macro:`Py_TPFLAGS_DISALLOW_INSTANTIATION` flag.
324
325
326 Defining Heap Types
327 -------------------
328
329 Heap types can be created by filling a :c:struct:`PyType_Spec` structure, a
330 description or "blueprint" of a class, and calling
331 :c:func:`PyType_FromModuleAndSpec` to construct a new class object.
332
333 .. note::
334 Other functions, like :c:func:`PyType_FromSpec`, can also create
335 heap types, but :c:func:`PyType_FromModuleAndSpec` associates the module
336 with the class, allowing access to the module state from methods.
337
338 The class should generally be stored in *both* the module state (for
339 safe access from C) and the module's ``__dict__`` (for access from
340 Python code).
341
342
343 Garbage-Collection Protocol
344 ---------------------------
345
346 Instances of heap types hold a reference to their type.
347 This ensures that the type isn't destroyed before all its instances are,
348 but may result in reference cycles that need to be broken by the
349 garbage collector.
350
351 To avoid memory leaks, instances of heap types must implement the
352 garbage collection protocol.
353 That is, heap types should:
354
355 - Have the :c:macro:`Py_TPFLAGS_HAVE_GC` flag.
356 - Define a traverse function using :c:data:`Py_tp_traverse`, which
357 visits the type (e.g. using ``Py_VISIT(Py_TYPE(self))``).
358
359 Please refer to the documentation of
360 :c:macro:`Py_TPFLAGS_HAVE_GC` and :c:member:`~PyTypeObject.tp_traverse`
361 for additional considerations.
362
363 The API for defining heap types grew organically, leaving it
364 somewhat awkward to use in its current state.
365 The following sections will guide you through common issues.
366
367
368 ``tp_traverse`` in Python 3.8 and lower
369 .......................................
370
371 The requirement to visit the type from ``tp_traverse`` was added in Python 3.9.
372 If you support Python 3.8 and lower, the traverse function must *not*
373 visit the type, so it must be more complicated::
374
375 static int my_traverse(PyObject *self, visitproc visit, void *arg)
376 {
377 if (Py_Version >= 0x03090000) {
378 Py_VISIT(Py_TYPE(self));
379 }
380 return 0;
381 }
382
383 Unfortunately, :c:data:`Py_Version` was only added in Python 3.11.
384 As a replacement, use:
385
386 * :c:macro:`PY_VERSION_HEX`, if not using the stable ABI, or
387 * :py:data:`sys.version_info` (via :c:func:`PySys_GetObject` and
388 :c:func:`PyArg_ParseTuple`).
389
390
391 Delegating ``tp_traverse``
392 ..........................
393
394 If your traverse function delegates to the :c:member:`~PyTypeObject.tp_traverse`
395 of its base class (or another type), ensure that ``Py_TYPE(self)`` is visited
396 only once.
397 Note that only heap type are expected to visit the type in ``tp_traverse``.
398
399 For example, if your traverse function includes::
400
401 base->tp_traverse(self, visit, arg)
402
403 ...and ``base`` may be a static type, then it should also include::
404
405 if (base->tp_flags & Py_TPFLAGS_HEAPTYPE) {
406 // a heap type's tp_traverse already visited Py_TYPE(self)
407 } else {
408 if (Py_Version >= 0x03090000) {
409 Py_VISIT(Py_TYPE(self));
410 }
411 }
412
413 It is not necessary to handle the type's reference count in
414 :c:member:`~PyTypeObject.tp_new` and :c:member:`~PyTypeObject.tp_clear`.
415
416
417 Defining ``tp_dealloc``
418 .......................
419
420 If your type has a custom :c:member:`~PyTypeObject.tp_dealloc` function,
421 it needs to:
422
423 - call :c:func:`PyObject_GC_UnTrack` before any fields are invalidated, and
424 - decrement the reference count of the type.
425
426 To keep the type valid while ``tp_free`` is called, the type's refcount needs
427 to be decremented *after* the instance is deallocated. For example::
428
429 static void my_dealloc(PyObject *self)
430 {
431 PyObject_GC_UnTrack(self);
432 ...
433 PyTypeObject *type = Py_TYPE(self);
434 type->tp_free(self);
435 Py_DECREF(type);
436 }
437
438 The default ``tp_dealloc`` function does this, so
439 if your type does *not* override
440 ``tp_dealloc`` you don't need to add it.
441
442
443 Not overriding ``tp_free``
444 ..........................
445
446 The :c:member:`~PyTypeObject.tp_free` slot of a heap type must be set to
447 :c:func:`PyObject_GC_Del`.
448 This is the default; do not override it.
449
450
451 Avoiding ``PyObject_New``
452 .........................
453
454 GC-tracked objects need to be allocated using GC-aware functions.
455
456 If you use :c:func:`PyObject_New` or :c:func:`PyObject_NewVar`:
457
458 - Get and call type's :c:member:`~PyTypeObject.tp_alloc` slot, if possible.
459 That is, replace ``TYPE *o = PyObject_New(TYPE, typeobj)`` with::
460
461 TYPE *o = typeobj->tp_alloc(typeobj, 0);
462
463 Replace ``o = PyObject_NewVar(TYPE, typeobj, size)`` with the same,
464 but use size instead of the 0.
465
466 - If the above is not possible (e.g. inside a custom ``tp_alloc``),
467 call :c:func:`PyObject_GC_New` or :c:func:`PyObject_GC_NewVar`::
468
469 TYPE *o = PyObject_GC_New(TYPE, typeobj);
470
471 TYPE *o = PyObject_GC_NewVar(TYPE, typeobj, size);
472
473
474 Module State Access from Classes
475 --------------------------------
476
477 If you have a type object defined with :c:func:`PyType_FromModuleAndSpec`,
478 you can call :c:func:`PyType_GetModule` to get the associated module, and then
479 :c:func:`PyModule_GetState` to get the module's state.
480
481 To save a some tedious error-handling boilerplate code, you can combine
482 these two steps with :c:func:`PyType_GetModuleState`, resulting in::
483
484 my_struct *state = (my_struct*)PyType_GetModuleState(type);
485 if (state == NULL) {
486 return NULL;
487 }
488
489
490 Module State Access from Regular Methods
491 ----------------------------------------
492
493 Accessing the module-level state from methods of a class is somewhat more
494 complicated, but is possible thanks to API introduced in Python 3.9.
495 To get the state, you need to first get the *defining class*, and then
496 get the module state from it.
497
498 The largest roadblock is getting *the class a method was defined in*, or
499 that method's "defining class" for short. The defining class can have a
500 reference to the module it is part of.
501
502 Do not confuse the defining class with ``Py_TYPE(self)``. If the method
503 is called on a *subclass* of your type, ``Py_TYPE(self)`` will refer to
504 that subclass, which may be defined in different module than yours.
505
506 .. note::
507 The following Python code can illustrate the concept.
508 ``Base.get_defining_class`` returns ``Base`` even
509 if ``type(self) == Sub``:
510
511 .. code-block:: python
512
513 class Base:
514 def get_type_of_self(self):
515 return type(self)
516
517 def get_defining_class(self):
518 return __class__
519
520 class Sub(Base):
521 pass
522
523 For a method to get its "defining class", it must use the
524 :ref:`METH_METHOD | METH_FASTCALL | METH_KEYWORDS <METH_METHOD-METH_FASTCALL-METH_KEYWORDS>`
525 :c:type:`calling convention <PyMethodDef>`
526 and the corresponding :c:type:`PyCMethod` signature::
527
528 PyObject *PyCMethod(
529 PyObject *self, // object the method was called on
530 PyTypeObject *defining_class, // defining class
531 PyObject *const *args, // C array of arguments
532 Py_ssize_t nargs, // length of "args"
533 PyObject *kwnames) // NULL, or dict of keyword arguments
534
535 Once you have the defining class, call :c:func:`PyType_GetModuleState` to get
536 the state of its associated module.
537
538 For example::
539
540 static PyObject *
541 example_method(PyObject *self,
542 PyTypeObject *defining_class,
543 PyObject *const *args,
544 Py_ssize_t nargs,
545 PyObject *kwnames)
546 {
547 my_struct *state = (my_struct*)PyType_GetModuleState(defining_class);
548 if (state == NULL) {
549 return NULL;
550 }
551 ... // rest of logic
552 }
553
554 PyDoc_STRVAR(example_method_doc, "...");
555
556 static PyMethodDef my_methods[] = {
557 {"example_method",
558 (PyCFunction)(void(*)(void))example_method,
559 METH_METHOD|METH_FASTCALL|METH_KEYWORDS,
560 example_method_doc}
561 {NULL},
562 }
563
564
565 Module State Access from Slot Methods, Getters and Setters
566 ----------------------------------------------------------
567
568 .. note::
569
570 This is new in Python 3.11.
571
572 .. After adding to limited API:
573
574 If you use the :ref:`limited API <limited-c-api>`,
575 you must update ``Py_LIMITED_API`` to ``0x030b0000``, losing ABI
576 compatibility with earlier versions.
577
578 Slot methods—the fast C equivalents for special methods, such as
579 :c:member:`~PyNumberMethods.nb_add` for :py:attr:`~object.__add__` or
580 :c:member:`~PyTypeObject.tp_new` for initialization—have a very simple API that
581 doesn't allow passing in the defining class, unlike with :c:type:`PyCMethod`.
582 The same goes for getters and setters defined with
583 :c:type:`PyGetSetDef`.
584
585 To access the module state in these cases, use the
586 :c:func:`PyType_GetModuleByDef` function, and pass in the module definition.
587 Once you have the module, call :c:func:`PyModule_GetState`
588 to get the state::
589
590 PyObject *module = PyType_GetModuleByDef(Py_TYPE(self), &module_def);
591 my_struct *state = (my_struct*)PyModule_GetState(module);
592 if (state == NULL) {
593 return NULL;
594 }
595
596 :c:func:`!PyType_GetModuleByDef` works by searching the
597 :term:`method resolution order` (i.e. all superclasses) for the first
598 superclass that has a corresponding module.
599
600 .. note::
601
602 In very exotic cases (inheritance chains spanning multiple modules
603 created from the same definition), :c:func:`!PyType_GetModuleByDef` might not
604 return the module of the true defining class. However, it will always
605 return a module with the same definition, ensuring a compatible
606 C memory layout.
607
608
609 Lifetime of the Module State
610 ----------------------------
611
612 When a module object is garbage-collected, its module state is freed.
613 For each pointer to (a part of) the module state, you must hold a reference
614 to the module object.
615
616 Usually this is not an issue, because types created with
617 :c:func:`PyType_FromModuleAndSpec`, and their instances, hold a reference
618 to the module.
619 However, you must be careful in reference counting when you reference
620 module state from other places, such as callbacks for external
621 libraries.
622
623
624 Open Issues
625 ===========
626
627 Several issues around per-module state and heap types are still open.
628
629 Discussions about improving the situation are best held on the `discuss forum under c-api tag <https://discuss.python.org/c/core-dev/c-api/30>`__.
630
631
632 Per-Class Scope
633 ---------------
634
635 It is currently (as of Python 3.11) not possible to attach state to individual
636 *types* without relying on CPython implementation details (which may change
637 in the future—perhaps, ironically, to allow a proper solution for
638 per-class scope).
639
640
641 Lossless Conversion to Heap Types
642 ---------------------------------
643
644 The heap type API was not designed for "lossless" conversion from static types;
645 that is, creating a type that works exactly like a given static type.