| 1 |
.. highlight:: c |
| 2 |
|
| 3 |
.. _extension-modules: |
| 4 |
|
| 5 |
Defining extension modules |
| 6 |
-------------------------- |
| 7 |
|
| 8 |
A C extension for CPython is a shared library (for example, a ``.so`` file |
| 9 |
on Linux, ``.pyd`` DLL on Windows), which is loadable into the Python process |
| 10 |
(for example, it is compiled with compatible compiler settings), and which |
| 11 |
exports an :ref:`initialization function <extension-export-hook>`. |
| 12 |
|
| 13 |
To be importable by default (that is, by |
| 14 |
:py:class:`importlib.machinery.ExtensionFileLoader`), |
| 15 |
the shared library must be available on :py:attr:`sys.path`, |
| 16 |
and must be named after the module name plus an extension listed in |
| 17 |
:py:attr:`importlib.machinery.EXTENSION_SUFFIXES`. |
| 18 |
|
| 19 |
.. note:: |
| 20 |
|
| 21 |
Building, packaging and distributing extension modules is best done with |
| 22 |
third-party tools, and is out of scope of this document. |
| 23 |
One suitable tool is Setuptools, whose documentation can be found at |
| 24 |
https://setuptools.pypa.io/en/latest/setuptools.html. |
| 25 |
|
| 26 |
Normally, the initialization function returns a module definition initialized |
| 27 |
using :c:func:`PyModuleDef_Init`. |
| 28 |
This allows splitting the creation process into several phases: |
| 29 |
|
| 30 |
- Before any substantial code is executed, Python can determine which |
| 31 |
capabilities the module supports, and it can adjust the environment or |
| 32 |
refuse loading an incompatible extension. |
| 33 |
- By default, Python itself creates the module object -- that is, it does |
| 34 |
the equivalent of :py:meth:`object.__new__` for classes. |
| 35 |
It also sets initial attributes like :attr:`~module.__package__` and |
| 36 |
:attr:`~module.__loader__`. |
| 37 |
- Afterwards, the module object is initialized using extension-specific |
| 38 |
code -- the equivalent of :py:meth:`~object.__init__` on classes. |
| 39 |
|
| 40 |
This is called *multi-phase initialization* to distinguish it from the legacy |
| 41 |
(but still supported) *single-phase initialization* scheme, |
| 42 |
where the initialization function returns a fully constructed module. |
| 43 |
See the :ref:`single-phase-initialization section below <single-phase-initialization>` |
| 44 |
for details. |
| 45 |
|
| 46 |
.. versionchanged:: 3.5 |
| 47 |
|
| 48 |
Added support for multi-phase initialization (:pep:`489`). |
| 49 |
|
| 50 |
|
| 51 |
Multiple module instances |
| 52 |
......................... |
| 53 |
|
| 54 |
By default, extension modules are not singletons. |
| 55 |
For example, if the :py:attr:`sys.modules` entry is removed and the module |
| 56 |
is re-imported, a new module object is created, and typically populated with |
| 57 |
fresh method and type objects. |
| 58 |
The old module is subject to normal garbage collection. |
| 59 |
This mirrors the behavior of pure-Python modules. |
| 60 |
|
| 61 |
Additional module instances may be created in |
| 62 |
:ref:`sub-interpreters <sub-interpreter-support>` |
| 63 |
or after Python runtime reinitialization |
| 64 |
(:c:func:`Py_Finalize` and :c:func:`Py_Initialize`). |
| 65 |
In these cases, sharing Python objects between module instances would likely |
| 66 |
cause crashes or undefined behavior. |
| 67 |
|
| 68 |
To avoid such issues, each instance of an extension module should |
| 69 |
be *isolated*: changes to one instance should not implicitly affect the others, |
| 70 |
and all state owned by the module, including references to Python objects, |
| 71 |
should be specific to a particular module instance. |
| 72 |
See :ref:`isolating-extensions-howto` for more details and a practical guide. |
| 73 |
|
| 74 |
A simpler way to avoid these issues is |
| 75 |
:ref:`raising an error on repeated initialization <isolating-extensions-optout>`. |
| 76 |
|
| 77 |
All modules are expected to support |
| 78 |
:ref:`sub-interpreters <sub-interpreter-support>`, or otherwise explicitly |
| 79 |
signal a lack of support. |
| 80 |
This is usually achieved by isolation or blocking repeated initialization, |
| 81 |
as above. |
| 82 |
A module may also be limited to the main interpreter using |
| 83 |
the :c:data:`Py_mod_multiple_interpreters` slot. |
| 84 |
|
| 85 |
|
| 86 |
.. _extension-export-hook: |
| 87 |
|
| 88 |
Initialization function |
| 89 |
....................... |
| 90 |
|
| 91 |
The initialization function defined by an extension module has the |
| 92 |
following signature: |
| 93 |
|
| 94 |
.. c:function:: PyObject* PyInit_modulename(void) |
| 95 |
|
| 96 |
Its name should be :samp:`PyInit_{<name>}`, with ``<name>`` replaced by the |
| 97 |
name of the module. |
| 98 |
|
| 99 |
For modules with ASCII-only names, the function must instead be named |
| 100 |
:samp:`PyInit_{<name>}`, with ``<name>`` replaced by the name of the module. |
| 101 |
When using :ref:`multi-phase-initialization`, non-ASCII module names |
| 102 |
are allowed. In this case, the initialization function name is |
| 103 |
:samp:`PyInitU_{<name>}`, with ``<name>`` encoded using Python's |
| 104 |
*punycode* encoding with hyphens replaced by underscores. In Python: |
| 105 |
|
| 106 |
.. code-block:: python |
| 107 |
|
| 108 |
def initfunc_name(name): |
| 109 |
try: |
| 110 |
suffix = b'_' + name.encode('ascii') |
| 111 |
except UnicodeEncodeError: |
| 112 |
suffix = b'U_' + name.encode('punycode').replace(b'-', b'_') |
| 113 |
return b'PyInit' + suffix |
| 114 |
|
| 115 |
It is recommended to define the initialization function using a helper macro: |
| 116 |
|
| 117 |
.. c:macro:: PyMODINIT_FUNC |
| 118 |
|
| 119 |
Declare an extension module initialization function. |
| 120 |
This macro: |
| 121 |
|
| 122 |
* specifies the :c:expr:`PyObject*` return type, |
| 123 |
* adds any special linkage declarations required by the platform, and |
| 124 |
* for C++, declares the function as ``extern "C"``. |
| 125 |
|
| 126 |
For example, a module called ``spam`` would be defined like this:: |
| 127 |
|
| 128 |
static struct PyModuleDef spam_module = { |
| 129 |
.m_base = PyModuleDef_HEAD_INIT, |
| 130 |
.m_name = "spam", |
| 131 |
... |
| 132 |
}; |
| 133 |
|
| 134 |
PyMODINIT_FUNC |
| 135 |
PyInit_spam(void) |
| 136 |
{ |
| 137 |
return PyModuleDef_Init(&spam_module); |
| 138 |
} |
| 139 |
|
| 140 |
It is possible to export multiple modules from a single shared library by |
| 141 |
defining multiple initialization functions. However, importing them requires |
| 142 |
using symbolic links or a custom importer, because by default only the |
| 143 |
function corresponding to the filename is found. |
| 144 |
See the `Multiple modules in one library <https://peps.python.org/pep-0489/#multiple-modules-in-one-library>`__ |
| 145 |
section in :pep:`489` for details. |
| 146 |
|
| 147 |
The initialization function is typically the only non-\ ``static`` |
| 148 |
item defined in the module's C source. |
| 149 |
|
| 150 |
|
| 151 |
.. _multi-phase-initialization: |
| 152 |
|
| 153 |
Multi-phase initialization |
| 154 |
.......................... |
| 155 |
|
| 156 |
Normally, the :ref:`initialization function <extension-export-hook>` |
| 157 |
(``PyInit_modulename``) returns a :c:type:`PyModuleDef` instance with |
| 158 |
non-``NULL`` :c:member:`~PyModuleDef.m_slots`. |
| 159 |
Before it is returned, the ``PyModuleDef`` instance must be initialized |
| 160 |
using the following function: |
| 161 |
|
| 162 |
|
| 163 |
.. c:function:: PyObject* PyModuleDef_Init(PyModuleDef *def) |
| 164 |
|
| 165 |
Ensure a module definition is a properly initialized Python object that |
| 166 |
correctly reports its type and a reference count. |
| 167 |
|
| 168 |
Return *def* cast to ``PyObject*``, or ``NULL`` if an error occurred. |
| 169 |
|
| 170 |
Calling this function is required for :ref:`multi-phase-initialization`. |
| 171 |
It should not be used in other contexts. |
| 172 |
|
| 173 |
Note that Python assumes that ``PyModuleDef`` structures are statically |
| 174 |
allocated. |
| 175 |
This function may return either a new reference or a borrowed one; |
| 176 |
this reference must not be released. |
| 177 |
|
| 178 |
.. versionadded:: 3.5 |
| 179 |
|
| 180 |
|
| 181 |
.. _single-phase-initialization: |
| 182 |
|
| 183 |
Legacy single-phase initialization |
| 184 |
.................................. |
| 185 |
|
| 186 |
.. attention:: |
| 187 |
Single-phase initialization is a legacy mechanism to initialize extension |
| 188 |
modules, with known drawbacks and design flaws. Extension module authors |
| 189 |
are encouraged to use multi-phase initialization instead. |
| 190 |
|
| 191 |
In single-phase initialization, the |
| 192 |
:ref:`initialization function <extension-export-hook>` (``PyInit_modulename``) |
| 193 |
should create, populate and return a module object. |
| 194 |
This is typically done using :c:func:`PyModule_Create` and functions like |
| 195 |
:c:func:`PyModule_AddObjectRef`. |
| 196 |
|
| 197 |
Single-phase initialization differs from the :ref:`default <multi-phase-initialization>` |
| 198 |
in the following ways: |
| 199 |
|
| 200 |
* Single-phase modules are, or rather *contain*, “singletons”. |
| 201 |
|
| 202 |
When the module is first initialized, Python saves the contents of |
| 203 |
the module's ``__dict__`` (that is, typically, the module's functions and |
| 204 |
types). |
| 205 |
|
| 206 |
For subsequent imports, Python does not call the initialization function |
| 207 |
again. |
| 208 |
Instead, it creates a new module object with a new ``__dict__``, and copies |
| 209 |
the saved contents to it. |
| 210 |
For example, given a single-phase module ``_testsinglephase`` |
| 211 |
[#testsinglephase]_ that defines a function ``sum`` and an exception class |
| 212 |
``error``: |
| 213 |
|
| 214 |
.. code-block:: python |
| 215 |
|
| 216 |
>>> import sys |
| 217 |
>>> import _testsinglephase as one |
| 218 |
>>> del sys.modules['_testsinglephase'] |
| 219 |
>>> import _testsinglephase as two |
| 220 |
>>> one is two |
| 221 |
False |
| 222 |
>>> one.__dict__ is two.__dict__ |
| 223 |
False |
| 224 |
>>> one.sum is two.sum |
| 225 |
True |
| 226 |
>>> one.error is two.error |
| 227 |
True |
| 228 |
|
| 229 |
The exact behavior should be considered a CPython implementation detail. |
| 230 |
|
| 231 |
* To work around the fact that ``PyInit_modulename`` does not take a *spec* |
| 232 |
argument, some state of the import machinery is saved and applied to the |
| 233 |
first suitable module created during the ``PyInit_modulename`` call. |
| 234 |
Specifically, when a sub-module is imported, this mechanism prepends the |
| 235 |
parent package name to the name of the module. |
| 236 |
|
| 237 |
A single-phase ``PyInit_modulename`` function should create “its” module |
| 238 |
object as soon as possible, before any other module objects can be created. |
| 239 |
|
| 240 |
* Non-ASCII module names (``PyInitU_modulename``) are not supported. |
| 241 |
|
| 242 |
* Single-phase modules support module lookup functions like |
| 243 |
:c:func:`PyState_FindModule`. |
| 244 |
|
| 245 |
.. [#testsinglephase] ``_testsinglephase`` is an internal module used |
| 246 |
in CPython's self-test suite; your installation may or may not |
| 247 |
include it. |