| 1 |
.. highlight:: c |
| 2 |
|
| 3 |
.. _isolating-extensions-howto: |
| 4 |
|
| 5 |
*************************** |
| 6 |
Isolating Extension Modules |
| 7 |
*************************** |
| 8 |
|
| 9 |
.. topic:: Abstract |
| 10 |
|
| 11 |
Traditionally, state belonging to Python extension modules was kept in C |
| 12 |
``static`` variables, which have process-wide scope. This document |
| 13 |
describes problems of such per-process state and shows a safer way: |
| 14 |
per-module state. |
| 15 |
|
| 16 |
The document also describes how to switch to per-module state where |
| 17 |
possible. This transition involves allocating space for that state, potentially |
| 18 |
switching from static types to heap types, and—perhaps most |
| 19 |
importantly—accessing per-module state from code. |
| 20 |
|
| 21 |
|
| 22 |
Who should read this |
| 23 |
==================== |
| 24 |
|
| 25 |
This guide is written for maintainers of :ref:`C-API <c-api-index>` extensions |
| 26 |
who would like to make that extension safer to use in applications where |
| 27 |
Python itself is used as a library. |
| 28 |
|
| 29 |
|
| 30 |
Background |
| 31 |
========== |
| 32 |
|
| 33 |
An *interpreter* is the context in which Python code runs. It contains |
| 34 |
configuration (e.g. the import path) and runtime state (e.g. the set of |
| 35 |
imported modules). |
| 36 |
|
| 37 |
Python supports running multiple interpreters in one process. There are |
| 38 |
two cases to think about—users may run interpreters: |
| 39 |
|
| 40 |
- in sequence, with several :c:func:`Py_InitializeEx`/:c:func:`Py_FinalizeEx` |
| 41 |
cycles, and |
| 42 |
- in parallel, managing "sub-interpreters" using |
| 43 |
:c:func:`Py_NewInterpreter`/:c:func:`Py_EndInterpreter`. |
| 44 |
|
| 45 |
Both cases (and combinations of them) would be most useful when |
| 46 |
embedding Python within a library. Libraries generally shouldn't make |
| 47 |
assumptions about the application that uses them, which include |
| 48 |
assuming a process-wide "main Python interpreter". |
| 49 |
|
| 50 |
Historically, Python extension modules don't handle this use case well. |
| 51 |
Many extension modules (and even some stdlib modules) use *per-process* |
| 52 |
global state, because C ``static`` variables are extremely easy to use. |
| 53 |
Thus, data that should be specific to an interpreter ends up being shared |
| 54 |
between interpreters. Unless the extension developer is careful, it is very |
| 55 |
easy to introduce edge cases that lead to crashes when a module is loaded in |
| 56 |
more than one interpreter in the same process. |
| 57 |
|
| 58 |
Unfortunately, *per-interpreter* state is not easy to achieve. Extension |
| 59 |
authors tend to not keep multiple interpreters in mind when developing, |
| 60 |
and it is currently cumbersome to test the behavior. |
| 61 |
|
| 62 |
Enter Per-Module State |
| 63 |
---------------------- |
| 64 |
|
| 65 |
Instead of focusing on per-interpreter state, Python's C API is evolving |
| 66 |
to better support the more granular *per-module* state. |
| 67 |
This means that C-level data should be attached to a *module object*. |
| 68 |
Each interpreter creates its own module object, keeping the data separate. |
| 69 |
For testing the isolation, multiple module objects corresponding to a single |
| 70 |
extension can even be loaded in a single interpreter. |
| 71 |
|
| 72 |
Per-module state provides an easy way to think about lifetime and |
| 73 |
resource ownership: the extension module will initialize when a |
| 74 |
module object is created, and clean up when it's freed. In this regard, |
| 75 |
a module is just like any other :c:expr:`PyObject *`; there are no "on |
| 76 |
interpreter shutdown" hooks to think—or forget—about. |
| 77 |
|
| 78 |
Note that there are use cases for different kinds of "globals": |
| 79 |
per-process, per-interpreter, per-thread or per-task state. |
| 80 |
With per-module state as the default, these are still possible, |
| 81 |
but you should treat them as exceptional cases: |
| 82 |
if you need them, you should give them additional care and testing. |
| 83 |
(Note that this guide does not cover them.) |
| 84 |
|
| 85 |
|
| 86 |
Isolated Module Objects |
| 87 |
----------------------- |
| 88 |
|
| 89 |
The key point to keep in mind when developing an extension module is |
| 90 |
that several module objects can be created from a single shared library. |
| 91 |
For example: |
| 92 |
|
| 93 |
.. code-block:: pycon |
| 94 |
|
| 95 |
>>> import sys |
| 96 |
>>> import binascii |
| 97 |
>>> old_binascii = binascii |
| 98 |
>>> del sys.modules['binascii'] |
| 99 |
>>> import binascii # create a new module object |
| 100 |
>>> old_binascii == binascii |
| 101 |
False |
| 102 |
|
| 103 |
As a rule of thumb, the two modules should be completely independent. |
| 104 |
All objects and state specific to the module should be encapsulated |
| 105 |
within the module object, not shared with other module objects, and |
| 106 |
cleaned up when the module object is deallocated. |
| 107 |
Since this just is a rule of thumb, exceptions are possible |
| 108 |
(see `Managing Global State`_), but they will need more |
| 109 |
thought and attention to edge cases. |
| 110 |
|
| 111 |
While some modules could do with less stringent restrictions, isolated |
| 112 |
modules make it easier to set clear expectations and guidelines that |
| 113 |
work across a variety of use cases. |
| 114 |
|
| 115 |
|
| 116 |
Surprising Edge Cases |
| 117 |
--------------------- |
| 118 |
|
| 119 |
Note that isolated modules do create some surprising edge cases. Most |
| 120 |
notably, each module object will typically not share its classes and |
| 121 |
exceptions with other similar modules. Continuing from the |
| 122 |
`example above <Isolated Module Objects_>`__, |
| 123 |
note that ``old_binascii.Error`` and ``binascii.Error`` are |
| 124 |
separate objects. In the following code, the exception is *not* caught: |
| 125 |
|
| 126 |
.. code-block:: pycon |
| 127 |
|
| 128 |
>>> old_binascii.Error == binascii.Error |
| 129 |
False |
| 130 |
>>> try: |
| 131 |
... old_binascii.unhexlify(b'qwertyuiop') |
| 132 |
... except binascii.Error: |
| 133 |
... print('boo') |
| 134 |
... |
| 135 |
Traceback (most recent call last): |
| 136 |
File "<stdin>", line 2, in <module> |
| 137 |
binascii.Error: Non-hexadecimal digit found |
| 138 |
|
| 139 |
This is expected. Notice that pure-Python modules behave the same way: |
| 140 |
it is a part of how Python works. |
| 141 |
|
| 142 |
The goal is to make extension modules safe at the C level, not to make |
| 143 |
hacks behave intuitively. Mutating ``sys.modules`` "manually" counts |
| 144 |
as a hack. |
| 145 |
|
| 146 |
|
| 147 |
Making Modules Safe with Multiple Interpreters |
| 148 |
============================================== |
| 149 |
|
| 150 |
|
| 151 |
Managing Global State |
| 152 |
--------------------- |
| 153 |
|
| 154 |
Sometimes, the state associated with a Python module is not specific to that module, but |
| 155 |
to the entire process (or something else "more global" than a module). |
| 156 |
For example: |
| 157 |
|
| 158 |
- The ``readline`` module manages *the* terminal. |
| 159 |
- A module running on a circuit board wants to control *the* on-board |
| 160 |
LED. |
| 161 |
|
| 162 |
In these cases, the Python module should provide *access* to the global |
| 163 |
state, rather than *own* it. If possible, write the module so that |
| 164 |
multiple copies of it can access the state independently (along with |
| 165 |
other libraries, whether for Python or other languages). If that is not |
| 166 |
possible, consider explicit locking. |
| 167 |
|
| 168 |
If it is necessary to use process-global state, the simplest way to |
| 169 |
avoid issues with multiple interpreters is to explicitly prevent a |
| 170 |
module from being loaded more than once per process—see |
| 171 |
:ref:`isolating-extensions-optout`. |
| 172 |
|
| 173 |
|
| 174 |
Managing Per-Module State |
| 175 |
------------------------- |
| 176 |
|
| 177 |
To use per-module state, use |
| 178 |
:ref:`multi-phase extension module initialization <multi-phase-initialization>`. |
| 179 |
This signals that your module supports multiple interpreters correctly. |
| 180 |
|
| 181 |
Set ``PyModuleDef.m_size`` to a positive number to request that many |
| 182 |
bytes of storage local to the module. Usually, this will be set to the |
| 183 |
size of some module-specific ``struct``, which can store all of the |
| 184 |
module's C-level state. In particular, it is where you should put |
| 185 |
pointers to classes (including exceptions, but excluding static types) |
| 186 |
and settings (e.g. ``csv``'s :py:data:`~csv.field_size_limit`) |
| 187 |
which the C code needs to function. |
| 188 |
|
| 189 |
.. note:: |
| 190 |
Another option is to store state in the module's ``__dict__``, |
| 191 |
but you must avoid crashing when users modify ``__dict__`` from |
| 192 |
Python code. This usually means error- and type-checking at the C level, |
| 193 |
which is easy to get wrong and hard to test sufficiently. |
| 194 |
|
| 195 |
However, if module state is not needed in C code, storing it in |
| 196 |
``__dict__`` only is a good idea. |
| 197 |
|
| 198 |
If the module state includes ``PyObject`` pointers, the module object |
| 199 |
must hold references to those objects and implement the module-level hooks |
| 200 |
``m_traverse``, ``m_clear`` and ``m_free``. These work like |
| 201 |
``tp_traverse``, ``tp_clear`` and ``tp_free`` of a class. Adding them will |
| 202 |
require some work and make the code longer; this is the price for |
| 203 |
modules which can be unloaded cleanly. |
| 204 |
|
| 205 |
An example of a module with per-module state is currently available as |
| 206 |
`xxlimited <https://github.com/python/cpython/blob/master/Modules/xxlimited.c>`__; |
| 207 |
example module initialization shown at the bottom of the file. |
| 208 |
|
| 209 |
|
| 210 |
.. _isolating-extensions-optout: |
| 211 |
|
| 212 |
Opt-Out: Limiting to One Module Object per Process |
| 213 |
-------------------------------------------------- |
| 214 |
|
| 215 |
A non-negative ``PyModuleDef.m_size`` signals that a module supports |
| 216 |
multiple interpreters correctly. If this is not yet the case for your |
| 217 |
module, you can explicitly make your module loadable only once per |
| 218 |
process. For example:: |
| 219 |
|
| 220 |
// A process-wide flag |
| 221 |
static int loaded = 0; |
| 222 |
|
| 223 |
// Mutex to provide thread safety (only needed for free-threaded Python) |
| 224 |
static PyMutex modinit_mutex = {0}; |
| 225 |
|
| 226 |
static int |
| 227 |
exec_module(PyObject* module) |
| 228 |
{ |
| 229 |
PyMutex_Lock(&modinit_mutex); |
| 230 |
if (loaded) { |
| 231 |
PyMutex_Unlock(&modinit_mutex); |
| 232 |
PyErr_SetString(PyExc_ImportError, |
| 233 |
"cannot load module more than once per process"); |
| 234 |
return -1; |
| 235 |
} |
| 236 |
loaded = 1; |
| 237 |
PyMutex_Unlock(&modinit_mutex); |
| 238 |
// ... rest of initialization |
| 239 |
} |
| 240 |
|
| 241 |
|
| 242 |
If your module's :c:member:`PyModuleDef.m_clear` function is able to prepare |
| 243 |
for future re-initialization, it should clear the ``loaded`` flag. |
| 244 |
In this case, your module won't support multiple instances existing |
| 245 |
*concurrently*, but it will, for example, support being loaded after |
| 246 |
Python runtime shutdown (:c:func:`Py_FinalizeEx`) and re-initialization |
| 247 |
(:c:func:`Py_Initialize`). |
| 248 |
|
| 249 |
|
| 250 |
Module State Access from Functions |
| 251 |
---------------------------------- |
| 252 |
|
| 253 |
Accessing the state from module-level functions is straightforward. |
| 254 |
Functions get the module object as their first argument; for extracting |
| 255 |
the state, you can use ``PyModule_GetState``:: |
| 256 |
|
| 257 |
static PyObject * |
| 258 |
func(PyObject *module, PyObject *args) |
| 259 |
{ |
| 260 |
my_struct *state = (my_struct*)PyModule_GetState(module); |
| 261 |
if (state == NULL) { |
| 262 |
return NULL; |
| 263 |
} |
| 264 |
// ... rest of logic |
| 265 |
} |
| 266 |
|
| 267 |
.. note:: |
| 268 |
``PyModule_GetState`` may return ``NULL`` without setting an |
| 269 |
exception if there is no module state, i.e. ``PyModuleDef.m_size`` was |
| 270 |
zero. In your own module, you're in control of ``m_size``, so this is |
| 271 |
easy to prevent. |
| 272 |
|
| 273 |
|
| 274 |
Heap Types |
| 275 |
========== |
| 276 |
|
| 277 |
Traditionally, types defined in C code are *static*; that is, |
| 278 |
``static PyTypeObject`` structures defined directly in code and |
| 279 |
initialized using ``PyType_Ready()``. |
| 280 |
|
| 281 |
Such types are necessarily shared across the process. Sharing them |
| 282 |
between module objects requires paying attention to any state they own |
| 283 |
or access. To limit the possible issues, static types are immutable at |
| 284 |
the Python level: for example, you can't set ``str.myattribute = 123``. |
| 285 |
|
| 286 |
.. impl-detail:: |
| 287 |
Sharing truly immutable objects between interpreters is fine, |
| 288 |
as long as they don't provide access to mutable objects. |
| 289 |
However, in CPython, every Python object has a mutable implementation |
| 290 |
detail: the reference count. Changes to the refcount are guarded by the GIL. |
| 291 |
Thus, code that shares any Python objects across interpreters implicitly |
| 292 |
depends on CPython's current, process-wide GIL. |
| 293 |
|
| 294 |
Because they are immutable and process-global, static types cannot access |
| 295 |
"their" module state. |
| 296 |
If any method of such a type requires access to module state, |
| 297 |
the type must be converted to a *heap-allocated type*, or *heap type* |
| 298 |
for short. These correspond more closely to classes created by Python's |
| 299 |
``class`` statement. |
| 300 |
|
| 301 |
For new modules, using heap types by default is a good rule of thumb. |
| 302 |
|
| 303 |
|
| 304 |
Changing Static Types to Heap Types |
| 305 |
----------------------------------- |
| 306 |
|
| 307 |
Static types can be converted to heap types, but note that |
| 308 |
the heap type API was not designed for "lossless" conversion |
| 309 |
from static types—that is, creating a type that works exactly like a given |
| 310 |
static type. |
| 311 |
So, when rewriting the class definition in a new API, |
| 312 |
you are likely to unintentionally change a few details (e.g. pickleability |
| 313 |
or inherited slots). |
| 314 |
Always test the details that are important to you. |
| 315 |
|
| 316 |
Watch out for the following two points in particular (but note that this is not |
| 317 |
a comprehensive list): |
| 318 |
|
| 319 |
* Unlike static types, heap type objects are mutable by default. |
| 320 |
Use the :c:macro:`Py_TPFLAGS_IMMUTABLETYPE` flag to prevent mutability. |
| 321 |
* Heap types inherit :c:member:`~PyTypeObject.tp_new` by default, |
| 322 |
so it may become possible to instantiate them from Python code. |
| 323 |
You can prevent this with the :c:macro:`Py_TPFLAGS_DISALLOW_INSTANTIATION` flag. |
| 324 |
|
| 325 |
|
| 326 |
Defining Heap Types |
| 327 |
------------------- |
| 328 |
|
| 329 |
Heap types can be created by filling a :c:struct:`PyType_Spec` structure, a |
| 330 |
description or "blueprint" of a class, and calling |
| 331 |
:c:func:`PyType_FromModuleAndSpec` to construct a new class object. |
| 332 |
|
| 333 |
.. note:: |
| 334 |
Other functions, like :c:func:`PyType_FromSpec`, can also create |
| 335 |
heap types, but :c:func:`PyType_FromModuleAndSpec` associates the module |
| 336 |
with the class, allowing access to the module state from methods. |
| 337 |
|
| 338 |
The class should generally be stored in *both* the module state (for |
| 339 |
safe access from C) and the module's ``__dict__`` (for access from |
| 340 |
Python code). |
| 341 |
|
| 342 |
|
| 343 |
Garbage-Collection Protocol |
| 344 |
--------------------------- |
| 345 |
|
| 346 |
Instances of heap types hold a reference to their type. |
| 347 |
This ensures that the type isn't destroyed before all its instances are, |
| 348 |
but may result in reference cycles that need to be broken by the |
| 349 |
garbage collector. |
| 350 |
|
| 351 |
To avoid memory leaks, instances of heap types must implement the |
| 352 |
garbage collection protocol. |
| 353 |
That is, heap types should: |
| 354 |
|
| 355 |
- Have the :c:macro:`Py_TPFLAGS_HAVE_GC` flag. |
| 356 |
- Define a traverse function using :c:data:`Py_tp_traverse`, which |
| 357 |
visits the type (e.g. using ``Py_VISIT(Py_TYPE(self))``). |
| 358 |
|
| 359 |
Please refer to the documentation of |
| 360 |
:c:macro:`Py_TPFLAGS_HAVE_GC` and :c:member:`~PyTypeObject.tp_traverse` |
| 361 |
for additional considerations. |
| 362 |
|
| 363 |
The API for defining heap types grew organically, leaving it |
| 364 |
somewhat awkward to use in its current state. |
| 365 |
The following sections will guide you through common issues. |
| 366 |
|
| 367 |
|
| 368 |
``tp_traverse`` in Python 3.8 and lower |
| 369 |
....................................... |
| 370 |
|
| 371 |
The requirement to visit the type from ``tp_traverse`` was added in Python 3.9. |
| 372 |
If you support Python 3.8 and lower, the traverse function must *not* |
| 373 |
visit the type, so it must be more complicated:: |
| 374 |
|
| 375 |
static int my_traverse(PyObject *self, visitproc visit, void *arg) |
| 376 |
{ |
| 377 |
if (Py_Version >= 0x03090000) { |
| 378 |
Py_VISIT(Py_TYPE(self)); |
| 379 |
} |
| 380 |
return 0; |
| 381 |
} |
| 382 |
|
| 383 |
Unfortunately, :c:data:`Py_Version` was only added in Python 3.11. |
| 384 |
As a replacement, use: |
| 385 |
|
| 386 |
* :c:macro:`PY_VERSION_HEX`, if not using the stable ABI, or |
| 387 |
* :py:data:`sys.version_info` (via :c:func:`PySys_GetObject` and |
| 388 |
:c:func:`PyArg_ParseTuple`). |
| 389 |
|
| 390 |
|
| 391 |
Delegating ``tp_traverse`` |
| 392 |
.......................... |
| 393 |
|
| 394 |
If your traverse function delegates to the :c:member:`~PyTypeObject.tp_traverse` |
| 395 |
of its base class (or another type), ensure that ``Py_TYPE(self)`` is visited |
| 396 |
only once. |
| 397 |
Note that only heap type are expected to visit the type in ``tp_traverse``. |
| 398 |
|
| 399 |
For example, if your traverse function includes:: |
| 400 |
|
| 401 |
base->tp_traverse(self, visit, arg) |
| 402 |
|
| 403 |
...and ``base`` may be a static type, then it should also include:: |
| 404 |
|
| 405 |
if (base->tp_flags & Py_TPFLAGS_HEAPTYPE) { |
| 406 |
// a heap type's tp_traverse already visited Py_TYPE(self) |
| 407 |
} else { |
| 408 |
if (Py_Version >= 0x03090000) { |
| 409 |
Py_VISIT(Py_TYPE(self)); |
| 410 |
} |
| 411 |
} |
| 412 |
|
| 413 |
It is not necessary to handle the type's reference count in |
| 414 |
:c:member:`~PyTypeObject.tp_new` and :c:member:`~PyTypeObject.tp_clear`. |
| 415 |
|
| 416 |
|
| 417 |
Defining ``tp_dealloc`` |
| 418 |
....................... |
| 419 |
|
| 420 |
If your type has a custom :c:member:`~PyTypeObject.tp_dealloc` function, |
| 421 |
it needs to: |
| 422 |
|
| 423 |
- call :c:func:`PyObject_GC_UnTrack` before any fields are invalidated, and |
| 424 |
- decrement the reference count of the type. |
| 425 |
|
| 426 |
To keep the type valid while ``tp_free`` is called, the type's refcount needs |
| 427 |
to be decremented *after* the instance is deallocated. For example:: |
| 428 |
|
| 429 |
static void my_dealloc(PyObject *self) |
| 430 |
{ |
| 431 |
PyObject_GC_UnTrack(self); |
| 432 |
... |
| 433 |
PyTypeObject *type = Py_TYPE(self); |
| 434 |
type->tp_free(self); |
| 435 |
Py_DECREF(type); |
| 436 |
} |
| 437 |
|
| 438 |
The default ``tp_dealloc`` function does this, so |
| 439 |
if your type does *not* override |
| 440 |
``tp_dealloc`` you don't need to add it. |
| 441 |
|
| 442 |
|
| 443 |
Not overriding ``tp_free`` |
| 444 |
.......................... |
| 445 |
|
| 446 |
The :c:member:`~PyTypeObject.tp_free` slot of a heap type must be set to |
| 447 |
:c:func:`PyObject_GC_Del`. |
| 448 |
This is the default; do not override it. |
| 449 |
|
| 450 |
|
| 451 |
Avoiding ``PyObject_New`` |
| 452 |
......................... |
| 453 |
|
| 454 |
GC-tracked objects need to be allocated using GC-aware functions. |
| 455 |
|
| 456 |
If you use :c:func:`PyObject_New` or :c:func:`PyObject_NewVar`: |
| 457 |
|
| 458 |
- Get and call type's :c:member:`~PyTypeObject.tp_alloc` slot, if possible. |
| 459 |
That is, replace ``TYPE *o = PyObject_New(TYPE, typeobj)`` with:: |
| 460 |
|
| 461 |
TYPE *o = typeobj->tp_alloc(typeobj, 0); |
| 462 |
|
| 463 |
Replace ``o = PyObject_NewVar(TYPE, typeobj, size)`` with the same, |
| 464 |
but use size instead of the 0. |
| 465 |
|
| 466 |
- If the above is not possible (e.g. inside a custom ``tp_alloc``), |
| 467 |
call :c:func:`PyObject_GC_New` or :c:func:`PyObject_GC_NewVar`:: |
| 468 |
|
| 469 |
TYPE *o = PyObject_GC_New(TYPE, typeobj); |
| 470 |
|
| 471 |
TYPE *o = PyObject_GC_NewVar(TYPE, typeobj, size); |
| 472 |
|
| 473 |
|
| 474 |
Module State Access from Classes |
| 475 |
-------------------------------- |
| 476 |
|
| 477 |
If you have a type object defined with :c:func:`PyType_FromModuleAndSpec`, |
| 478 |
you can call :c:func:`PyType_GetModule` to get the associated module, and then |
| 479 |
:c:func:`PyModule_GetState` to get the module's state. |
| 480 |
|
| 481 |
To save a some tedious error-handling boilerplate code, you can combine |
| 482 |
these two steps with :c:func:`PyType_GetModuleState`, resulting in:: |
| 483 |
|
| 484 |
my_struct *state = (my_struct*)PyType_GetModuleState(type); |
| 485 |
if (state == NULL) { |
| 486 |
return NULL; |
| 487 |
} |
| 488 |
|
| 489 |
|
| 490 |
Module State Access from Regular Methods |
| 491 |
---------------------------------------- |
| 492 |
|
| 493 |
Accessing the module-level state from methods of a class is somewhat more |
| 494 |
complicated, but is possible thanks to API introduced in Python 3.9. |
| 495 |
To get the state, you need to first get the *defining class*, and then |
| 496 |
get the module state from it. |
| 497 |
|
| 498 |
The largest roadblock is getting *the class a method was defined in*, or |
| 499 |
that method's "defining class" for short. The defining class can have a |
| 500 |
reference to the module it is part of. |
| 501 |
|
| 502 |
Do not confuse the defining class with ``Py_TYPE(self)``. If the method |
| 503 |
is called on a *subclass* of your type, ``Py_TYPE(self)`` will refer to |
| 504 |
that subclass, which may be defined in different module than yours. |
| 505 |
|
| 506 |
.. note:: |
| 507 |
The following Python code can illustrate the concept. |
| 508 |
``Base.get_defining_class`` returns ``Base`` even |
| 509 |
if ``type(self) == Sub``: |
| 510 |
|
| 511 |
.. code-block:: python |
| 512 |
|
| 513 |
class Base: |
| 514 |
def get_type_of_self(self): |
| 515 |
return type(self) |
| 516 |
|
| 517 |
def get_defining_class(self): |
| 518 |
return __class__ |
| 519 |
|
| 520 |
class Sub(Base): |
| 521 |
pass |
| 522 |
|
| 523 |
For a method to get its "defining class", it must use the |
| 524 |
:ref:`METH_METHOD | METH_FASTCALL | METH_KEYWORDS <METH_METHOD-METH_FASTCALL-METH_KEYWORDS>` |
| 525 |
:c:type:`calling convention <PyMethodDef>` |
| 526 |
and the corresponding :c:type:`PyCMethod` signature:: |
| 527 |
|
| 528 |
PyObject *PyCMethod( |
| 529 |
PyObject *self, // object the method was called on |
| 530 |
PyTypeObject *defining_class, // defining class |
| 531 |
PyObject *const *args, // C array of arguments |
| 532 |
Py_ssize_t nargs, // length of "args" |
| 533 |
PyObject *kwnames) // NULL, or dict of keyword arguments |
| 534 |
|
| 535 |
Once you have the defining class, call :c:func:`PyType_GetModuleState` to get |
| 536 |
the state of its associated module. |
| 537 |
|
| 538 |
For example:: |
| 539 |
|
| 540 |
static PyObject * |
| 541 |
example_method(PyObject *self, |
| 542 |
PyTypeObject *defining_class, |
| 543 |
PyObject *const *args, |
| 544 |
Py_ssize_t nargs, |
| 545 |
PyObject *kwnames) |
| 546 |
{ |
| 547 |
my_struct *state = (my_struct*)PyType_GetModuleState(defining_class); |
| 548 |
if (state == NULL) { |
| 549 |
return NULL; |
| 550 |
} |
| 551 |
... // rest of logic |
| 552 |
} |
| 553 |
|
| 554 |
PyDoc_STRVAR(example_method_doc, "..."); |
| 555 |
|
| 556 |
static PyMethodDef my_methods[] = { |
| 557 |
{"example_method", |
| 558 |
(PyCFunction)(void(*)(void))example_method, |
| 559 |
METH_METHOD|METH_FASTCALL|METH_KEYWORDS, |
| 560 |
example_method_doc} |
| 561 |
{NULL}, |
| 562 |
} |
| 563 |
|
| 564 |
|
| 565 |
Module State Access from Slot Methods, Getters and Setters |
| 566 |
---------------------------------------------------------- |
| 567 |
|
| 568 |
.. note:: |
| 569 |
|
| 570 |
This is new in Python 3.11. |
| 571 |
|
| 572 |
.. After adding to limited API: |
| 573 |
|
| 574 |
If you use the :ref:`limited API <limited-c-api>`, |
| 575 |
you must update ``Py_LIMITED_API`` to ``0x030b0000``, losing ABI |
| 576 |
compatibility with earlier versions. |
| 577 |
|
| 578 |
Slot methods—the fast C equivalents for special methods, such as |
| 579 |
:c:member:`~PyNumberMethods.nb_add` for :py:attr:`~object.__add__` or |
| 580 |
:c:member:`~PyTypeObject.tp_new` for initialization—have a very simple API that |
| 581 |
doesn't allow passing in the defining class, unlike with :c:type:`PyCMethod`. |
| 582 |
The same goes for getters and setters defined with |
| 583 |
:c:type:`PyGetSetDef`. |
| 584 |
|
| 585 |
To access the module state in these cases, use the |
| 586 |
:c:func:`PyType_GetModuleByDef` function, and pass in the module definition. |
| 587 |
Once you have the module, call :c:func:`PyModule_GetState` |
| 588 |
to get the state:: |
| 589 |
|
| 590 |
PyObject *module = PyType_GetModuleByDef(Py_TYPE(self), &module_def); |
| 591 |
my_struct *state = (my_struct*)PyModule_GetState(module); |
| 592 |
if (state == NULL) { |
| 593 |
return NULL; |
| 594 |
} |
| 595 |
|
| 596 |
:c:func:`!PyType_GetModuleByDef` works by searching the |
| 597 |
:term:`method resolution order` (i.e. all superclasses) for the first |
| 598 |
superclass that has a corresponding module. |
| 599 |
|
| 600 |
.. note:: |
| 601 |
|
| 602 |
In very exotic cases (inheritance chains spanning multiple modules |
| 603 |
created from the same definition), :c:func:`!PyType_GetModuleByDef` might not |
| 604 |
return the module of the true defining class. However, it will always |
| 605 |
return a module with the same definition, ensuring a compatible |
| 606 |
C memory layout. |
| 607 |
|
| 608 |
|
| 609 |
Lifetime of the Module State |
| 610 |
---------------------------- |
| 611 |
|
| 612 |
When a module object is garbage-collected, its module state is freed. |
| 613 |
For each pointer to (a part of) the module state, you must hold a reference |
| 614 |
to the module object. |
| 615 |
|
| 616 |
Usually this is not an issue, because types created with |
| 617 |
:c:func:`PyType_FromModuleAndSpec`, and their instances, hold a reference |
| 618 |
to the module. |
| 619 |
However, you must be careful in reference counting when you reference |
| 620 |
module state from other places, such as callbacks for external |
| 621 |
libraries. |
| 622 |
|
| 623 |
|
| 624 |
Open Issues |
| 625 |
=========== |
| 626 |
|
| 627 |
Several issues around per-module state and heap types are still open. |
| 628 |
|
| 629 |
Discussions about improving the situation are best held on the `discuss forum under c-api tag <https://discuss.python.org/c/core-dev/c-api/30>`__. |
| 630 |
|
| 631 |
|
| 632 |
Per-Class Scope |
| 633 |
--------------- |
| 634 |
|
| 635 |
It is currently (as of Python 3.11) not possible to attach state to individual |
| 636 |
*types* without relying on CPython implementation details (which may change |
| 637 |
in the future—perhaps, ironically, to allow a proper solution for |
| 638 |
per-class scope). |
| 639 |
|
| 640 |
|
| 641 |
Lossless Conversion to Heap Types |
| 642 |
--------------------------------- |
| 643 |
|
| 644 |
The heap type API was not designed for "lossless" conversion from static types; |
| 645 |
that is, creating a type that works exactly like a given static type. |