Debian Codegraph @

Doc/extending/extending.rst

raw
1 .. highlight:: c
2
3
4 .. _extending-intro:
5
6 ******************************
7 Extending Python with C or C++
8 ******************************
9
10 It is quite easy to add new built-in modules to Python, if you know how to
11 program in C. Such :dfn:`extension modules` can do two things that can't be
12 done directly in Python: they can implement new built-in object types, and they
13 can call C library functions and system calls.
14
15 To support extensions, the Python API (Application Programmers Interface)
16 defines a set of functions, macros and variables that provide access to most
17 aspects of the Python run-time system. The Python API is incorporated in a C
18 source file by including the header ``"Python.h"``.
19
20 The compilation of an extension module depends on its intended use as well as on
21 your system setup; details are given in later chapters.
22
23 .. note::
24
25 The C extension interface is specific to CPython, and extension modules do
26 not work on other Python implementations. In many cases, it is possible to
27 avoid writing C extensions and preserve portability to other implementations.
28 For example, if your use case is calling C library functions or system calls,
29 you should consider using the :mod:`ctypes` module or the `cffi
30 <https://cffi.readthedocs.io/>`_ library rather than writing
31 custom C code.
32 These modules let you write Python code to interface with C code and are more
33 portable between implementations of Python than writing and compiling a C
34 extension module.
35
36
37 .. _extending-simpleexample:
38
39 A Simple Example
40 ================
41
42 Let's create an extension module called ``spam`` (the favorite food of Monty
43 Python fans...) and let's say we want to create a Python interface to the C
44 library function :c:func:`system` [#]_. This function takes a null-terminated
45 character string as argument and returns an integer. We want this function to
46 be callable from Python as follows:
47
48 .. code-block:: pycon
49
50 >>> import spam
51 >>> status = spam.system("ls -l")
52
53 Begin by creating a file :file:`spammodule.c`. (Historically, if a module is
54 called ``spam``, the C file containing its implementation is called
55 :file:`spammodule.c`; if the module name is very long, like ``spammify``, the
56 module name can be just :file:`spammify.c`.)
57
58 The first two lines of our file can be::
59
60 #define PY_SSIZE_T_CLEAN
61 #include <Python.h>
62
63 which pulls in the Python API (you can add a comment describing the purpose of
64 the module and a copyright notice if you like).
65
66 .. note::
67
68 Since Python may define some pre-processor definitions which affect the standard
69 headers on some systems, you *must* include :file:`Python.h` before any standard
70 headers are included.
71
72 ``#define PY_SSIZE_T_CLEAN`` was used to indicate that ``Py_ssize_t`` should be
73 used in some APIs instead of ``int``.
74 It is not necessary since Python 3.13, but we keep it here for backward compatibility.
75 See :ref:`arg-parsing-string-and-buffers` for a description of this macro.
76
77 All user-visible symbols defined by :file:`Python.h` have a prefix of ``Py`` or
78 ``PY``, except those defined in standard header files.
79
80 .. tip::
81
82 For backward compatibility, :file:`Python.h` includes several standard header files.
83 C extensions should include the standard headers that they use,
84 and should not rely on these implicit includes.
85 If using the limited C API version 3.13 or newer, the implicit includes are:
86
87 * ``<assert.h>``
88 * ``<intrin.h>`` (on Windows)
89 * ``<inttypes.h>``
90 * ``<limits.h>``
91 * ``<math.h>``
92 * ``<stdarg.h>``
93 * ``<wchar.h>``
94 * ``<sys/types.h>`` (if present)
95
96 If :c:macro:`Py_LIMITED_API` is not defined, or is set to version 3.12 or older,
97 the headers below are also included:
98
99 * ``<ctype.h>``
100 * ``<unistd.h>`` (on POSIX)
101
102 If :c:macro:`Py_LIMITED_API` is not defined, or is set to version 3.10 or older,
103 the headers below are also included:
104
105 * ``<errno.h>``
106 * ``<stdio.h>``
107 * ``<stdlib.h>``
108 * ``<string.h>``
109
110 The next thing we add to our module file is the C function that will be called
111 when the Python expression ``spam.system(string)`` is evaluated (we'll see
112 shortly how it ends up being called)::
113
114 static PyObject *
115 spam_system(PyObject *self, PyObject *args)
116 {
117 const char *command;
118 int sts;
119
120 if (!PyArg_ParseTuple(args, "s", &command))
121 return NULL;
122 sts = system(command);
123 return PyLong_FromLong(sts);
124 }
125
126 There is a straightforward translation from the argument list in Python (for
127 example, the single expression ``"ls -l"``) to the arguments passed to the C
128 function. The C function always has two arguments, conventionally named *self*
129 and *args*.
130
131 The *self* argument points to the module object for module-level functions;
132 for a method it would point to the object instance.
133
134 The *args* argument will be a pointer to a Python tuple object containing the
135 arguments. Each item of the tuple corresponds to an argument in the call's
136 argument list. The arguments are Python objects --- in order to do anything
137 with them in our C function we have to convert them to C values. The function
138 :c:func:`PyArg_ParseTuple` in the Python API checks the argument types and
139 converts them to C values. It uses a template string to determine the required
140 types of the arguments as well as the types of the C variables into which to
141 store the converted values. More about this later.
142
143 :c:func:`PyArg_ParseTuple` returns true (nonzero) if all arguments have the right
144 type and its components have been stored in the variables whose addresses are
145 passed. It returns false (zero) if an invalid argument list was passed. In the
146 latter case it also raises an appropriate exception so the calling function can
147 return ``NULL`` immediately (as we saw in the example).
148
149
150 .. _extending-errors:
151
152 Intermezzo: Errors and Exceptions
153 =================================
154
155 An important convention throughout the Python interpreter is the following: when
156 a function fails, it should set an exception condition and return an error value
157 (usually ``-1`` or a ``NULL`` pointer). Exception information is stored in
158 three members of the interpreter's thread state. These are ``NULL`` if
159 there is no exception. Otherwise they are the C equivalents of the members
160 of the Python tuple returned by :meth:`sys.exc_info`. These are the
161 exception type, exception instance, and a traceback object. It is important
162 to know about them to understand how errors are passed around.
163
164 The Python API defines a number of functions to set various types of exceptions.
165
166 The most common one is :c:func:`PyErr_SetString`. Its arguments are an exception
167 object and a C string. The exception object is usually a predefined object like
168 :c:data:`PyExc_ZeroDivisionError`. The C string indicates the cause of the error
169 and is converted to a Python string object and stored as the "associated value"
170 of the exception.
171
172 Another useful function is :c:func:`PyErr_SetFromErrno`, which only takes an
173 exception argument and constructs the associated value by inspection of the
174 global variable :c:data:`errno`. The most general function is
175 :c:func:`PyErr_SetObject`, which takes two object arguments, the exception and
176 its associated value. You don't need to :c:func:`Py_INCREF` the objects passed
177 to any of these functions.
178
179 You can test non-destructively whether an exception has been set with
180 :c:func:`PyErr_Occurred`. This returns the current exception object, or ``NULL``
181 if no exception has occurred. You normally don't need to call
182 :c:func:`PyErr_Occurred` to see whether an error occurred in a function call,
183 since you should be able to tell from the return value.
184
185 When a function *f* that calls another function *g* detects that the latter
186 fails, *f* should itself return an error value (usually ``NULL`` or ``-1``). It
187 should *not* call one of the ``PyErr_*`` functions --- one has already
188 been called by *g*. *f*'s caller is then supposed to also return an error
189 indication to *its* caller, again *without* calling ``PyErr_*``, and so on
190 --- the most detailed cause of the error was already reported by the function
191 that first detected it. Once the error reaches the Python interpreter's main
192 loop, this aborts the currently executing Python code and tries to find an
193 exception handler specified by the Python programmer.
194
195 (There are situations where a module can actually give a more detailed error
196 message by calling another ``PyErr_*`` function, and in such cases it is
197 fine to do so. As a general rule, however, this is not necessary, and can cause
198 information about the cause of the error to be lost: most operations can fail
199 for a variety of reasons.)
200
201 To ignore an exception set by a function call that failed, the exception
202 condition must be cleared explicitly by calling :c:func:`PyErr_Clear`. The only
203 time C code should call :c:func:`PyErr_Clear` is if it doesn't want to pass the
204 error on to the interpreter but wants to handle it completely by itself
205 (possibly by trying something else, or pretending nothing went wrong).
206
207 Every failing :c:func:`malloc` call must be turned into an exception --- the
208 direct caller of :c:func:`malloc` (or :c:func:`realloc`) must call
209 :c:func:`PyErr_NoMemory` and return a failure indicator itself. All the
210 object-creating functions (for example, :c:func:`PyLong_FromLong`) already do
211 this, so this note is only relevant to those who call :c:func:`malloc` directly.
212
213 Also note that, with the important exception of :c:func:`PyArg_ParseTuple` and
214 friends, functions that return an integer status usually return a positive value
215 or zero for success and ``-1`` for failure, like Unix system calls.
216
217 Finally, be careful to clean up garbage (by making :c:func:`Py_XDECREF` or
218 :c:func:`Py_DECREF` calls for objects you have already created) when you return
219 an error indicator!
220
221 The choice of which exception to raise is entirely yours. There are predeclared
222 C objects corresponding to all built-in Python exceptions, such as
223 :c:data:`PyExc_ZeroDivisionError`, which you can use directly. Of course, you
224 should choose exceptions wisely --- don't use :c:data:`PyExc_TypeError` to mean
225 that a file couldn't be opened (that should probably be :c:data:`PyExc_OSError`).
226 If something's wrong with the argument list, the :c:func:`PyArg_ParseTuple`
227 function usually raises :c:data:`PyExc_TypeError`. If you have an argument whose
228 value must be in a particular range or must satisfy other conditions,
229 :c:data:`PyExc_ValueError` is appropriate.
230
231 You can also define a new exception that is unique to your module.
232 The simplest way to do this is to declare a static global object variable at
233 the beginning of the file::
234
235 static PyObject *SpamError = NULL;
236
237 and initialize it by calling :c:func:`PyErr_NewException` in the module's
238 :c:data:`Py_mod_exec` function (:c:func:`!spam_module_exec`)::
239
240 SpamError = PyErr_NewException("spam.error", NULL, NULL);
241
242 Since :c:data:`!SpamError` is a global variable, it will be overwritten every time
243 the module is reinitialized, when the :c:data:`Py_mod_exec` function is called.
244
245 For now, let's avoid the issue: we will block repeated initialization by raising an
246 :py:exc:`ImportError`::
247
248 static PyObject *SpamError = NULL;
249
250 static int
251 spam_module_exec(PyObject *m)
252 {
253 if (SpamError != NULL) {
254 PyErr_SetString(PyExc_ImportError,
255 "cannot initialize spam module more than once");
256 return -1;
257 }
258 SpamError = PyErr_NewException("spam.error", NULL, NULL);
259 if (PyModule_AddObjectRef(m, "SpamError", SpamError) < 0) {
260 return -1;
261 }
262
263 return 0;
264 }
265
266 static PyModuleDef_Slot spam_module_slots[] = {
267 {Py_mod_exec, spam_module_exec},
268 {0, NULL}
269 };
270
271 static struct PyModuleDef spam_module = {
272 .m_base = PyModuleDef_HEAD_INIT,
273 .m_name = "spam",
274 .m_size = 0, // non-negative
275 .m_slots = spam_module_slots,
276 };
277
278 PyMODINIT_FUNC
279 PyInit_spam(void)
280 {
281 return PyModuleDef_Init(&spam_module);
282 }
283
284 Note that the Python name for the exception object is :exc:`!spam.error`. The
285 :c:func:`PyErr_NewException` function may create a class with the base class
286 being :exc:`Exception` (unless another class is passed in instead of ``NULL``),
287 described in :ref:`bltin-exceptions`.
288
289 Note also that the :c:data:`!SpamError` variable retains a reference to the newly
290 created exception class; this is intentional! Since the exception could be
291 removed from the module by external code, an owned reference to the class is
292 needed to ensure that it will not be discarded, causing :c:data:`!SpamError` to
293 become a dangling pointer. Should it become a dangling pointer, C code which
294 raises the exception could cause a core dump or other unintended side effects.
295
296 For now, the :c:func:`Py_DECREF` call to remove this reference is missing.
297 Even when the Python interpreter shuts down, the global :c:data:`!SpamError`
298 variable will not be garbage-collected. It will "leak".
299 We did, however, ensure that this will happen at most once per process.
300
301 We discuss the use of :c:macro:`PyMODINIT_FUNC` as a function return type later in this
302 sample.
303
304 The :exc:`!spam.error` exception can be raised in your extension module using a
305 call to :c:func:`PyErr_SetString` as shown below::
306
307 static PyObject *
308 spam_system(PyObject *self, PyObject *args)
309 {
310 const char *command;
311 int sts;
312
313 if (!PyArg_ParseTuple(args, "s", &command))
314 return NULL;
315 sts = system(command);
316 if (sts < 0) {
317 PyErr_SetString(SpamError, "System command failed");
318 return NULL;
319 }
320 return PyLong_FromLong(sts);
321 }
322
323
324 .. _backtoexample:
325
326 Back to the Example
327 ===================
328
329 Going back to our example function, you should now be able to understand this
330 statement::
331
332 if (!PyArg_ParseTuple(args, "s", &command))
333 return NULL;
334
335 It returns ``NULL`` (the error indicator for functions returning object pointers)
336 if an error is detected in the argument list, relying on the exception set by
337 :c:func:`PyArg_ParseTuple`. Otherwise the string value of the argument has been
338 copied to the local variable :c:data:`!command`. This is a pointer assignment and
339 you are not supposed to modify the string to which it points (so in Standard C,
340 the variable :c:data:`!command` should properly be declared as ``const char
341 *command``).
342
343 The next statement is a call to the Unix function :c:func:`system`, passing it
344 the string we just got from :c:func:`PyArg_ParseTuple`::
345
346 sts = system(command);
347
348 Our :func:`!spam.system` function must return the value of :c:data:`!sts` as a
349 Python object. This is done using the function :c:func:`PyLong_FromLong`. ::
350
351 return PyLong_FromLong(sts);
352
353 In this case, it will return an integer object. (Yes, even integers are objects
354 on the heap in Python!)
355
356 If you have a C function that returns no useful argument (a function returning
357 :c:expr:`void`), the corresponding Python function must return ``None``. You
358 need this idiom to do so (which is implemented by the :c:macro:`Py_RETURN_NONE`
359 macro)::
360
361 Py_INCREF(Py_None);
362 return Py_None;
363
364 :c:data:`Py_None` is the C name for the special Python object ``None``. It is a
365 genuine Python object rather than a ``NULL`` pointer, which means "error" in most
366 contexts, as we have seen.
367
368
369 .. _methodtable:
370
371 The Module's Method Table and Initialization Function
372 =====================================================
373
374 I promised to show how :c:func:`!spam_system` is called from Python programs.
375 First, we need to list its name and address in a "method table"::
376
377 static PyMethodDef spam_methods[] = {
378 ...
379 {"system", spam_system, METH_VARARGS,
380 "Execute a shell command."},
381 ...
382 {NULL, NULL, 0, NULL} /* Sentinel */
383 };
384
385 Note the third entry (``METH_VARARGS``). This is a flag telling the interpreter
386 the calling convention to be used for the C function. It should normally always
387 be ``METH_VARARGS`` or ``METH_VARARGS | METH_KEYWORDS``; a value of ``0`` means
388 that an obsolete variant of :c:func:`PyArg_ParseTuple` is used.
389
390 When using only ``METH_VARARGS``, the function should expect the Python-level
391 parameters to be passed in as a tuple acceptable for parsing via
392 :c:func:`PyArg_ParseTuple`; more information on this function is provided below.
393
394 The :c:macro:`METH_KEYWORDS` bit may be set in the third field if keyword
395 arguments should be passed to the function. In this case, the C function should
396 accept a third ``PyObject *`` parameter which will be a dictionary of keywords.
397 Use :c:func:`PyArg_ParseTupleAndKeywords` to parse the arguments to such a
398 function.
399
400 The method table must be referenced in the module definition structure::
401
402 static struct PyModuleDef spam_module = {
403 ...
404 .m_methods = spam_methods,
405 ...
406 };
407
408 This structure, in turn, must be passed to the interpreter in the module's
409 initialization function. The initialization function must be named
410 :c:func:`!PyInit_name`, where *name* is the name of the module, and should be the
411 only non-\ ``static`` item defined in the module file::
412
413 PyMODINIT_FUNC
414 PyInit_spam(void)
415 {
416 return PyModuleDef_Init(&spam_module);
417 }
418
419 Note that :c:macro:`PyMODINIT_FUNC` declares the function as ``PyObject *`` return type,
420 declares any special linkage declarations required by the platform, and for C++
421 declares the function as ``extern "C"``.
422
423 :c:func:`!PyInit_spam` is called when each interpreter imports its module
424 :mod:`!spam` for the first time. (See below for comments about embedding Python.)
425 A pointer to the module definition must be returned via :c:func:`PyModuleDef_Init`,
426 so that the import machinery can create the module and store it in ``sys.modules``.
427
428 When embedding Python, the :c:func:`!PyInit_spam` function is not called
429 automatically unless there's an entry in the :c:data:`PyImport_Inittab` table.
430 To add the module to the initialization table, use :c:func:`PyImport_AppendInittab`,
431 optionally followed by an import of the module::
432
433 #define PY_SSIZE_T_CLEAN
434 #include <Python.h>
435
436 int
437 main(int argc, char *argv[])
438 {
439 PyStatus status;
440 PyConfig config;
441 PyConfig_InitPythonConfig(&config);
442
443 /* Add a built-in module, before Py_Initialize */
444 if (PyImport_AppendInittab("spam", PyInit_spam) == -1) {
445 fprintf(stderr, "Error: could not extend in-built modules table\n");
446 exit(1);
447 }
448
449 /* Pass argv[0] to the Python interpreter */
450 status = PyConfig_SetBytesString(&config, &config.program_name, argv[0]);
451 if (PyStatus_Exception(status)) {
452 goto exception;
453 }
454
455 /* Initialize the Python interpreter. Required.
456 If this step fails, it will be a fatal error. */
457 status = Py_InitializeFromConfig(&config);
458 if (PyStatus_Exception(status)) {
459 goto exception;
460 }
461 PyConfig_Clear(&config);
462
463 /* Optionally import the module; alternatively,
464 import can be deferred until the embedded script
465 imports it. */
466 PyObject *pmodule = PyImport_ImportModule("spam");
467 if (!pmodule) {
468 PyErr_Print();
469 fprintf(stderr, "Error: could not import module 'spam'\n");
470 }
471
472 // ... use Python C API here ...
473
474 return 0;
475
476 exception:
477 PyConfig_Clear(&config);
478 Py_ExitStatusException(status);
479 }
480
481 .. note::
482
483 If you declare a global variable or a local static one, the module may
484 experience unintended side-effects on re-initialisation, for example when
485 removing entries from ``sys.modules`` or importing compiled modules into
486 multiple interpreters within a process
487 (or following a :c:func:`fork` without an intervening :c:func:`exec`).
488 If module state is not yet fully :ref:`isolated <isolating-extensions-howto>`,
489 authors should consider marking the module as having no support for subinterpreters
490 (via :c:macro:`Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED`).
491
492 A more substantial example module is included in the Python source distribution
493 as :file:`Modules/xxlimited.c`. This file may be used as a template or simply
494 read as an example.
495
496
497 .. _compilation:
498
499 Compilation and Linkage
500 =======================
501
502 There are two more things to do before you can use your new extension: compiling
503 and linking it with the Python system. If you use dynamic loading, the details
504 may depend on the style of dynamic loading your system uses; see the chapters
505 about building extension modules (chapter :ref:`building`) and additional
506 information that pertains only to building on Windows (chapter
507 :ref:`building-on-windows`) for more information about this.
508
509 If you can't use dynamic loading, or if you want to make your module a permanent
510 part of the Python interpreter, you will have to change the configuration setup
511 and rebuild the interpreter. Luckily, this is very simple on Unix: just place
512 your file (:file:`spammodule.c` for example) in the :file:`Modules/` directory
513 of an unpacked source distribution, add a line to the file
514 :file:`Modules/Setup.local` describing your file:
515
516 .. code-block:: sh
517
518 spam spammodule.o
519
520 and rebuild the interpreter by running :program:`make` in the toplevel
521 directory. You can also run :program:`make` in the :file:`Modules/`
522 subdirectory, but then you must first rebuild :file:`Makefile` there by running
523 ':program:`make` Makefile'. (This is necessary each time you change the
524 :file:`Setup` file.)
525
526 If your module requires additional libraries to link with, these can be listed
527 on the line in the configuration file as well, for instance:
528
529 .. code-block:: sh
530
531 spam spammodule.o -lX11
532
533
534 .. _callingpython:
535
536 Calling Python Functions from C
537 ===============================
538
539 So far we have concentrated on making C functions callable from Python. The
540 reverse is also useful: calling Python functions from C. This is especially the
541 case for libraries that support so-called "callback" functions. If a C
542 interface makes use of callbacks, the equivalent Python often needs to provide a
543 callback mechanism to the Python programmer; the implementation will require
544 calling the Python callback functions from a C callback. Other uses are also
545 imaginable.
546
547 Fortunately, the Python interpreter is easily called recursively, and there is a
548 standard interface to call a Python function. (If you're interested in how to call the
549 Python parser with a particular string as input, see :ref:`veryhigh`.)
550
551 Calling a Python function is easy. First, the Python program must somehow pass
552 you the Python function object. You should provide a function (or some other
553 interface) to do this. When this function is called, save a pointer to the
554 Python function object (be careful to :c:func:`Py_INCREF` it!) in a global
555 variable --- or wherever you see fit. For example, the following function might
556 be part of a module definition::
557
558 static PyObject *my_callback = NULL;
559
560 static PyObject *
561 my_set_callback(PyObject *dummy, PyObject *args)
562 {
563 PyObject *result = NULL;
564 PyObject *temp;
565
566 if (PyArg_ParseTuple(args, "O:set_callback", &temp)) {
567 if (!PyCallable_Check(temp)) {
568 PyErr_SetString(PyExc_TypeError, "parameter must be callable");
569 return NULL;
570 }
571 Py_XINCREF(temp); /* Add a reference to new callback */
572 Py_XDECREF(my_callback); /* Dispose of previous callback */
573 my_callback = temp; /* Remember new callback */
574 /* Boilerplate to return "None" */
575 Py_INCREF(Py_None);
576 result = Py_None;
577 }
578 return result;
579 }
580
581 This function must be registered with the interpreter using the
582 :c:macro:`METH_VARARGS` flag; this is described in section :ref:`methodtable`. The
583 :c:func:`PyArg_ParseTuple` function and its arguments are documented in section
584 :ref:`parsetuple`.
585
586 The macros :c:func:`Py_XINCREF` and :c:func:`Py_XDECREF` increment/decrement the
587 reference count of an object and are safe in the presence of ``NULL`` pointers
588 (but note that *temp* will not be ``NULL`` in this context). More info on them
589 in section :ref:`refcounts`.
590
591 .. index:: single: PyObject_CallObject (C function)
592
593 Later, when it is time to call the function, you call the C function
594 :c:func:`PyObject_CallObject`. This function has two arguments, both pointers to
595 arbitrary Python objects: the Python function, and the argument list. The
596 argument list must always be a tuple object, whose length is the number of
597 arguments. To call the Python function with no arguments, pass in ``NULL``, or
598 an empty tuple; to call it with one argument, pass a singleton tuple.
599 :c:func:`Py_BuildValue` returns a tuple when its format string consists of zero
600 or more format codes between parentheses. For example::
601
602 int arg;
603 PyObject *arglist;
604 PyObject *result;
605 ...
606 arg = 123;
607 ...
608 /* Time to call the callback */
609 arglist = Py_BuildValue("(i)", arg);
610 result = PyObject_CallObject(my_callback, arglist);
611 Py_DECREF(arglist);
612
613 :c:func:`PyObject_CallObject` returns a Python object pointer: this is the return
614 value of the Python function. :c:func:`PyObject_CallObject` is
615 "reference-count-neutral" with respect to its arguments. In the example a new
616 tuple was created to serve as the argument list, which is
617 :c:func:`Py_DECREF`\ -ed immediately after the :c:func:`PyObject_CallObject`
618 call.
619
620 The return value of :c:func:`PyObject_CallObject` is "new": either it is a brand
621 new object, or it is an existing object whose reference count has been
622 incremented. So, unless you want to save it in a global variable, you should
623 somehow :c:func:`Py_DECREF` the result, even (especially!) if you are not
624 interested in its value.
625
626 Before you do this, however, it is important to check that the return value
627 isn't ``NULL``. If it is, the Python function terminated by raising an exception.
628 If the C code that called :c:func:`PyObject_CallObject` is called from Python, it
629 should now return an error indication to its Python caller, so the interpreter
630 can print a stack trace, or the calling Python code can handle the exception.
631 If this is not possible or desirable, the exception should be cleared by calling
632 :c:func:`PyErr_Clear`. For example::
633
634 if (result == NULL)
635 return NULL; /* Pass error back */
636 ...use result...
637 Py_DECREF(result);
638
639 Depending on the desired interface to the Python callback function, you may also
640 have to provide an argument list to :c:func:`PyObject_CallObject`. In some cases
641 the argument list is also provided by the Python program, through the same
642 interface that specified the callback function. It can then be saved and used
643 in the same manner as the function object. In other cases, you may have to
644 construct a new tuple to pass as the argument list. The simplest way to do this
645 is to call :c:func:`Py_BuildValue`. For example, if you want to pass an integral
646 event code, you might use the following code::
647
648 PyObject *arglist;
649 ...
650 arglist = Py_BuildValue("(l)", eventcode);
651 result = PyObject_CallObject(my_callback, arglist);
652 Py_DECREF(arglist);
653 if (result == NULL)
654 return NULL; /* Pass error back */
655 /* Here maybe use the result */
656 Py_DECREF(result);
657
658 Note the placement of ``Py_DECREF(arglist)`` immediately after the call, before
659 the error check! Also note that strictly speaking this code is not complete:
660 :c:func:`Py_BuildValue` may run out of memory, and this should be checked.
661
662 You may also call a function with keyword arguments by using
663 :c:func:`PyObject_Call`, which supports arguments and keyword arguments. As in
664 the above example, we use :c:func:`Py_BuildValue` to construct the dictionary. ::
665
666 PyObject *dict;
667 ...
668 dict = Py_BuildValue("{s:i}", "name", val);
669 result = PyObject_Call(my_callback, NULL, dict);
670 Py_DECREF(dict);
671 if (result == NULL)
672 return NULL; /* Pass error back */
673 /* Here maybe use the result */
674 Py_DECREF(result);
675
676
677 .. _parsetuple:
678
679 Extracting Parameters in Extension Functions
680 ============================================
681
682 .. index:: single: PyArg_ParseTuple (C function)
683
684 The :c:func:`PyArg_ParseTuple` function is declared as follows::
685
686 int PyArg_ParseTuple(PyObject *arg, const char *format, ...);
687
688 The *arg* argument must be a tuple object containing an argument list passed
689 from Python to a C function. The *format* argument must be a format string,
690 whose syntax is explained in :ref:`arg-parsing` in the Python/C API Reference
691 Manual. The remaining arguments must be addresses of variables whose type is
692 determined by the format string.
693
694 Note that while :c:func:`PyArg_ParseTuple` checks that the Python arguments have
695 the required types, it cannot check the validity of the addresses of C variables
696 passed to the call: if you make mistakes there, your code will probably crash or
697 at least overwrite random bits in memory. So be careful!
698
699 Note that any Python object references which are provided to the caller are
700 *borrowed* references; do not decrement their reference count!
701
702 Some example calls::
703
704 #define PY_SSIZE_T_CLEAN
705 #include <Python.h>
706
707 ::
708
709 int ok;
710 int i, j;
711 long k, l;
712 const char *s;
713 Py_ssize_t size;
714
715 ok = PyArg_ParseTuple(args, ""); /* No arguments */
716 /* Python call: f() */
717
718 ::
719
720 ok = PyArg_ParseTuple(args, "s", &s); /* A string */
721 /* Possible Python call: f('whoops!') */
722
723 ::
724
725 ok = PyArg_ParseTuple(args, "lls", &k, &l, &s); /* Two longs and a string */
726 /* Possible Python call: f(1, 2, 'three') */
727
728 ::
729
730 ok = PyArg_ParseTuple(args, "(ii)s#", &i, &j, &s, &size);
731 /* A pair of ints and a string, whose size is also returned */
732 /* Possible Python call: f((1, 2), 'three') */
733
734 ::
735
736 {
737 const char *file;
738 const char *mode = "r";
739 int bufsize = 0;
740 ok = PyArg_ParseTuple(args, "s|si", &file, &mode, &bufsize);
741 /* A string, and optionally another string and an integer */
742 /* Possible Python calls:
743 f('spam')
744 f('spam', 'w')
745 f('spam', 'wb', 100000) */
746 }
747
748 ::
749
750 {
751 int left, top, right, bottom, h, v;
752 ok = PyArg_ParseTuple(args, "((ii)(ii))(ii)",
753 &left, &top, &right, &bottom, &h, &v);
754 /* A rectangle and a point */
755 /* Possible Python call:
756 f(((0, 0), (400, 300)), (10, 10)) */
757 }
758
759 ::
760
761 {
762 Py_complex c;
763 ok = PyArg_ParseTuple(args, "D:myfunction", &c);
764 /* a complex, also providing a function name for errors */
765 /* Possible Python call: myfunction(1+2j) */
766 }
767
768
769 .. _parsetupleandkeywords:
770
771 Keyword Parameters for Extension Functions
772 ==========================================
773
774 .. index:: single: PyArg_ParseTupleAndKeywords (C function)
775
776 The :c:func:`PyArg_ParseTupleAndKeywords` function is declared as follows::
777
778 int PyArg_ParseTupleAndKeywords(PyObject *arg, PyObject *kwdict,
779 const char *format, char * const *kwlist, ...);
780
781 The *arg* and *format* parameters are identical to those of the
782 :c:func:`PyArg_ParseTuple` function. The *kwdict* parameter is the dictionary of
783 keywords received as the third parameter from the Python runtime. The *kwlist*
784 parameter is a ``NULL``-terminated list of strings which identify the parameters;
785 the names are matched with the type information from *format* from left to
786 right. On success, :c:func:`PyArg_ParseTupleAndKeywords` returns true, otherwise
787 it returns false and raises an appropriate exception.
788
789 .. note::
790
791 Nested tuples cannot be parsed when using keyword arguments! Keyword parameters
792 passed in which are not present in the *kwlist* will cause :exc:`TypeError` to
793 be raised.
794
795 .. index:: single: Philbrick, Geoff
796
797 Here is an example module which uses keywords, based on an example by Geoff
798 Philbrick ([email protected])::
799
800 #define PY_SSIZE_T_CLEAN
801 #include <Python.h>
802
803 static PyObject *
804 keywdarg_parrot(PyObject *self, PyObject *args, PyObject *keywds)
805 {
806 int voltage;
807 const char *state = "a stiff";
808 const char *action = "voom";
809 const char *type = "Norwegian Blue";
810
811 static char *kwlist[] = {"voltage", "state", "action", "type", NULL};
812
813 if (!PyArg_ParseTupleAndKeywords(args, keywds, "i|sss", kwlist,
814 &voltage, &state, &action, &type))
815 return NULL;
816
817 printf("-- This parrot wouldn't %s if you put %i Volts through it.\n",
818 action, voltage);
819 printf("-- Lovely plumage, the %s -- It's %s!\n", type, state);
820
821 Py_RETURN_NONE;
822 }
823
824 static PyMethodDef keywdarg_methods[] = {
825 /* The cast of the function is necessary since PyCFunction values
826 * only take two PyObject* parameters, and keywdarg_parrot() takes
827 * three.
828 */
829 {"parrot", (PyCFunction)(void(*)(void))keywdarg_parrot, METH_VARARGS | METH_KEYWORDS,
830 "Print a lovely skit to standard output."},
831 {NULL, NULL, 0, NULL} /* sentinel */
832 };
833
834 static struct PyModuleDef keywdarg_module = {
835 .m_base = PyModuleDef_HEAD_INIT,
836 .m_name = "keywdarg",
837 .m_size = 0,
838 .m_methods = keywdarg_methods,
839 };
840
841 PyMODINIT_FUNC
842 PyInit_keywdarg(void)
843 {
844 return PyModuleDef_Init(&keywdarg_module);
845 }
846
847
848 .. _buildvalue:
849
850 Building Arbitrary Values
851 =========================
852
853 This function is the counterpart to :c:func:`PyArg_ParseTuple`. It is declared
854 as follows::
855
856 PyObject *Py_BuildValue(const char *format, ...);
857
858 It recognizes a set of format units similar to the ones recognized by
859 :c:func:`PyArg_ParseTuple`, but the arguments (which are input to the function,
860 not output) must not be pointers, just values. It returns a new Python object,
861 suitable for returning from a C function called from Python.
862
863 One difference with :c:func:`PyArg_ParseTuple`: while the latter requires its
864 first argument to be a tuple (since Python argument lists are always represented
865 as tuples internally), :c:func:`Py_BuildValue` does not always build a tuple. It
866 builds a tuple only if its format string contains two or more format units. If
867 the format string is empty, it returns ``None``; if it contains exactly one
868 format unit, it returns whatever object is described by that format unit. To
869 force it to return a tuple of size 0 or one, parenthesize the format string.
870
871 Examples (to the left the call, to the right the resulting Python value):
872
873 .. code-block:: none
874
875 Py_BuildValue("") None
876 Py_BuildValue("i", 123) 123
877 Py_BuildValue("iii", 123, 456, 789) (123, 456, 789)
878 Py_BuildValue("s", "hello") 'hello'
879 Py_BuildValue("y", "hello") b'hello'
880 Py_BuildValue("ss", "hello", "world") ('hello', 'world')
881 Py_BuildValue("s#", "hello", 4) 'hell'
882 Py_BuildValue("y#", "hello", 4) b'hell'
883 Py_BuildValue("()") ()
884 Py_BuildValue("(i)", 123) (123,)
885 Py_BuildValue("(ii)", 123, 456) (123, 456)
886 Py_BuildValue("(i,i)", 123, 456) (123, 456)
887 Py_BuildValue("[i,i]", 123, 456) [123, 456]
888 Py_BuildValue("{s:i,s:i}",
889 "abc", 123, "def", 456) {'abc': 123, 'def': 456}
890 Py_BuildValue("((ii)(ii)) (ii)",
891 1, 2, 3, 4, 5, 6) (((1, 2), (3, 4)), (5, 6))
892
893
894 .. _refcounts:
895
896 Reference Counts
897 ================
898
899 In languages like C or C++, the programmer is responsible for dynamic allocation
900 and deallocation of memory on the heap. In C, this is done using the functions
901 :c:func:`malloc` and :c:func:`free`. In C++, the operators ``new`` and
902 ``delete`` are used with essentially the same meaning and we'll restrict
903 the following discussion to the C case.
904
905 Every block of memory allocated with :c:func:`malloc` should eventually be
906 returned to the pool of available memory by exactly one call to :c:func:`free`.
907 It is important to call :c:func:`free` at the right time. If a block's address
908 is forgotten but :c:func:`free` is not called for it, the memory it occupies
909 cannot be reused until the program terminates. This is called a :dfn:`memory
910 leak`. On the other hand, if a program calls :c:func:`free` for a block and then
911 continues to use the block, it creates a conflict with reuse of the block
912 through another :c:func:`malloc` call. This is called :dfn:`using freed memory`.
913 It has the same bad consequences as referencing uninitialized data --- core
914 dumps, wrong results, mysterious crashes.
915
916 Common causes of memory leaks are unusual paths through the code. For instance,
917 a function may allocate a block of memory, do some calculation, and then free
918 the block again. Now a change in the requirements for the function may add a
919 test to the calculation that detects an error condition and can return
920 prematurely from the function. It's easy to forget to free the allocated memory
921 block when taking this premature exit, especially when it is added later to the
922 code. Such leaks, once introduced, often go undetected for a long time: the
923 error exit is taken only in a small fraction of all calls, and most modern
924 machines have plenty of virtual memory, so the leak only becomes apparent in a
925 long-running process that uses the leaking function frequently. Therefore, it's
926 important to prevent leaks from happening by having a coding convention or
927 strategy that minimizes this kind of errors.
928
929 Since Python makes heavy use of :c:func:`malloc` and :c:func:`free`, it needs a
930 strategy to avoid memory leaks as well as the use of freed memory. The chosen
931 method is called :dfn:`reference counting`. The principle is simple: every
932 object contains a counter, which is incremented when a reference to the object
933 is stored somewhere, and which is decremented when a reference to it is deleted.
934 When the counter reaches zero, the last reference to the object has been deleted
935 and the object is freed.
936
937 An alternative strategy is called :dfn:`automatic garbage collection`.
938 (Sometimes, reference counting is also referred to as a garbage collection
939 strategy, hence the use of "automatic" to distinguish the two.) The big
940 advantage of automatic garbage collection is that the user doesn't need to call
941 :c:func:`free` explicitly. (Another claimed advantage is an improvement in speed
942 or memory usage --- this is no hard fact however.) The disadvantage is that for
943 C, there is no truly portable automatic garbage collector, while reference
944 counting can be implemented portably (as long as the functions :c:func:`malloc`
945 and :c:func:`free` are available --- which the C Standard guarantees). Maybe some
946 day a sufficiently portable automatic garbage collector will be available for C.
947 Until then, we'll have to live with reference counts.
948
949 While Python uses the traditional reference counting implementation, it also
950 offers a cycle detector that works to detect reference cycles. This allows
951 applications to not worry about creating direct or indirect circular references;
952 these are the weakness of garbage collection implemented using only reference
953 counting. Reference cycles consist of objects which contain (possibly indirect)
954 references to themselves, so that each object in the cycle has a reference count
955 which is non-zero. Typical reference counting implementations are not able to
956 reclaim the memory belonging to any objects in a reference cycle, or referenced
957 from the objects in the cycle, even though there are no further references to
958 the cycle itself.
959
960 The cycle detector is able to detect garbage cycles and can reclaim them.
961 The :mod:`gc` module exposes a way to run the detector (the
962 :func:`~gc.collect` function), as well as configuration
963 interfaces and the ability to disable the detector at runtime.
964
965
966 .. _refcountsinpython:
967
968 Reference Counting in Python
969 ----------------------------
970
971 There are two macros, ``Py_INCREF(x)`` and ``Py_DECREF(x)``, which handle the
972 incrementing and decrementing of the reference count. :c:func:`Py_DECREF` also
973 frees the object when the count reaches zero. For flexibility, it doesn't call
974 :c:func:`free` directly --- rather, it makes a call through a function pointer in
975 the object's :dfn:`type object`. For this purpose (and others), every object
976 also contains a pointer to its type object.
977
978 The big question now remains: when to use ``Py_INCREF(x)`` and ``Py_DECREF(x)``?
979 Let's first introduce some terms. Nobody "owns" an object; however, you can
980 :dfn:`own a reference` to an object. An object's reference count is now defined
981 as the number of owned references to it. The owner of a reference is
982 responsible for calling :c:func:`Py_DECREF` when the reference is no longer
983 needed. Ownership of a reference can be transferred. There are three ways to
984 dispose of an owned reference: pass it on, store it, or call :c:func:`Py_DECREF`.
985 Forgetting to dispose of an owned reference creates a memory leak.
986
987 It is also possible to :dfn:`borrow` [#]_ a reference to an object. The
988 borrower of a reference should not call :c:func:`Py_DECREF`. The borrower must
989 not hold on to the object longer than the owner from which it was borrowed.
990 Using a borrowed reference after the owner has disposed of it risks using freed
991 memory and should be avoided completely [#]_.
992
993 The advantage of borrowing over owning a reference is that you don't need to
994 take care of disposing of the reference on all possible paths through the code
995 --- in other words, with a borrowed reference you don't run the risk of leaking
996 when a premature exit is taken. The disadvantage of borrowing over owning is
997 that there are some subtle situations where in seemingly correct code a borrowed
998 reference can be used after the owner from which it was borrowed has in fact
999 disposed of it.
1000
1001 A borrowed reference can be changed into an owned reference by calling
1002 :c:func:`Py_INCREF`. This does not affect the status of the owner from which the
1003 reference was borrowed --- it creates a new owned reference, and gives full
1004 owner responsibilities (the new owner must dispose of the reference properly, as
1005 well as the previous owner).
1006
1007
1008 .. _ownershiprules:
1009
1010 Ownership Rules
1011 ---------------
1012
1013 Whenever an object reference is passed into or out of a function, it is part of
1014 the function's interface specification whether ownership is transferred with the
1015 reference or not.
1016
1017 Most functions that return a reference to an object pass on ownership with the
1018 reference. In particular, all functions whose function it is to create a new
1019 object, such as :c:func:`PyLong_FromLong` and :c:func:`Py_BuildValue`, pass
1020 ownership to the receiver. Even if the object is not actually new, you still
1021 receive ownership of a new reference to that object. For instance,
1022 :c:func:`PyLong_FromLong` maintains a cache of popular values and can return a
1023 reference to a cached item.
1024
1025 Many functions that extract objects from other objects also transfer ownership
1026 with the reference, for instance :c:func:`PyObject_GetAttrString`. The picture
1027 is less clear, here, however, since a few common routines are exceptions:
1028 :c:func:`PyTuple_GetItem`, :c:func:`PyList_GetItem`, :c:func:`PyDict_GetItem`, and
1029 :c:func:`PyDict_GetItemString` all return references that you borrow from the
1030 tuple, list or dictionary.
1031
1032 The function :c:func:`PyImport_AddModule` also returns a borrowed reference, even
1033 though it may actually create the object it returns: this is possible because an
1034 owned reference to the object is stored in ``sys.modules``.
1035
1036 When you pass an object reference into another function, in general, the
1037 function borrows the reference from you --- if it needs to store it, it will use
1038 :c:func:`Py_INCREF` to become an independent owner. There are exactly two
1039 important exceptions to this rule: :c:func:`PyTuple_SetItem` and
1040 :c:func:`PyList_SetItem`. These functions take over ownership of the item passed
1041 to them --- even if they fail! (Note that :c:func:`PyDict_SetItem` and friends
1042 don't take over ownership --- they are "normal.")
1043
1044 When a C function is called from Python, it borrows references to its arguments
1045 from the caller. The caller owns a reference to the object, so the borrowed
1046 reference's lifetime is guaranteed until the function returns. Only when such a
1047 borrowed reference must be stored or passed on, it must be turned into an owned
1048 reference by calling :c:func:`Py_INCREF`.
1049
1050 The object reference returned from a C function that is called from Python must
1051 be an owned reference --- ownership is transferred from the function to its
1052 caller.
1053
1054
1055 .. _thinice:
1056
1057 Thin Ice
1058 --------
1059
1060 There are a few situations where seemingly harmless use of a borrowed reference
1061 can lead to problems. These all have to do with implicit invocations of the
1062 interpreter, which can cause the owner of a reference to dispose of it.
1063
1064 The first and most important case to know about is using :c:func:`Py_DECREF` on
1065 an unrelated object while borrowing a reference to a list item. For instance::
1066
1067 void
1068 bug(PyObject *list)
1069 {
1070 PyObject *item = PyList_GetItem(list, 0);
1071
1072 PyList_SetItem(list, 1, PyLong_FromLong(0L));
1073 PyObject_Print(item, stdout, 0); /* BUG! */
1074 }
1075
1076 This function first borrows a reference to ``list[0]``, then replaces
1077 ``list[1]`` with the value ``0``, and finally prints the borrowed reference.
1078 Looks harmless, right? But it's not!
1079
1080 Let's follow the control flow into :c:func:`PyList_SetItem`. The list owns
1081 references to all its items, so when item 1 is replaced, it has to dispose of
1082 the original item 1. Now let's suppose the original item 1 was an instance of a
1083 user-defined class, and let's further suppose that the class defined a
1084 :meth:`!__del__` method. If this class instance has a reference count of 1,
1085 disposing of it will call its :meth:`!__del__` method. Internally,
1086 :c:func:`PyList_SetItem` calls :c:func:`Py_DECREF` on the replaced item,
1087 which invokes replaced item's corresponding
1088 :c:member:`~PyTypeObject.tp_dealloc` function. During
1089 deallocation, :c:member:`~PyTypeObject.tp_dealloc` calls
1090 :c:member:`~PyTypeObject.tp_finalize`, which is mapped to the
1091 :meth:`!__del__` method for class instances (see :pep:`442`). This entire
1092 sequence happens synchronously within the :c:func:`PyList_SetItem` call.
1093
1094 Since it is written in Python, the :meth:`!__del__` method can execute arbitrary
1095 Python code. Could it perhaps do something to invalidate the reference to
1096 ``item`` in :c:func:`!bug`? You bet! Assuming that the list passed into
1097 :c:func:`!bug` is accessible to the :meth:`!__del__` method, it could execute a
1098 statement to the effect of ``del list[0]``, and assuming this was the last
1099 reference to that object, it would free the memory associated with it, thereby
1100 invalidating ``item``.
1101
1102 The solution, once you know the source of the problem, is easy: temporarily
1103 increment the reference count. The correct version of the function reads::
1104
1105 void
1106 no_bug(PyObject *list)
1107 {
1108 PyObject *item = PyList_GetItem(list, 0);
1109
1110 Py_INCREF(item);
1111 PyList_SetItem(list, 1, PyLong_FromLong(0L));
1112 PyObject_Print(item, stdout, 0);
1113 Py_DECREF(item);
1114 }
1115
1116 This is a true story. An older version of Python contained variants of this bug
1117 and someone spent a considerable amount of time in a C debugger to figure out
1118 why his :meth:`!__del__` methods would fail...
1119
1120 The second case of problems with a borrowed reference is a variant involving
1121 threads. Normally, multiple threads in the Python interpreter can't get in each
1122 other's way, because there is a :term:`global lock <global interpreter lock>`
1123 protecting Python's entire object space.
1124 However, it is possible to temporarily release this lock using the macro
1125 :c:macro:`Py_BEGIN_ALLOW_THREADS`, and to re-acquire it using
1126 :c:macro:`Py_END_ALLOW_THREADS`. This is common around blocking I/O calls, to
1127 let other threads use the processor while waiting for the I/O to complete.
1128 Obviously, the following function has the same problem as the previous one::
1129
1130 void
1131 bug(PyObject *list)
1132 {
1133 PyObject *item = PyList_GetItem(list, 0);
1134 Py_BEGIN_ALLOW_THREADS
1135 ...some blocking I/O call...
1136 Py_END_ALLOW_THREADS
1137 PyObject_Print(item, stdout, 0); /* BUG! */
1138 }
1139
1140
1141 .. _nullpointers:
1142
1143 NULL Pointers
1144 -------------
1145
1146 In general, functions that take object references as arguments do not expect you
1147 to pass them ``NULL`` pointers, and will dump core (or cause later core dumps) if
1148 you do so. Functions that return object references generally return ``NULL`` only
1149 to indicate that an exception occurred. The reason for not testing for ``NULL``
1150 arguments is that functions often pass the objects they receive on to other
1151 function --- if each function were to test for ``NULL``, there would be a lot of
1152 redundant tests and the code would run more slowly.
1153
1154 It is better to test for ``NULL`` only at the "source:" when a pointer that may be
1155 ``NULL`` is received, for example, from :c:func:`malloc` or from a function that
1156 may raise an exception.
1157
1158 The macros :c:func:`Py_INCREF` and :c:func:`Py_DECREF` do not check for ``NULL``
1159 pointers --- however, their variants :c:func:`Py_XINCREF` and :c:func:`Py_XDECREF`
1160 do.
1161
1162 The macros for checking for a particular object type (``Pytype_Check()``) don't
1163 check for ``NULL`` pointers --- again, there is much code that calls several of
1164 these in a row to test an object against various different expected types, and
1165 this would generate redundant tests. There are no variants with ``NULL``
1166 checking.
1167
1168 The C function calling mechanism guarantees that the argument list passed to C
1169 functions (``args`` in the examples) is never ``NULL`` --- in fact it guarantees
1170 that it is always a tuple [#]_.
1171
1172 It is a severe error to ever let a ``NULL`` pointer "escape" to the Python user.
1173
1174 .. Frank Stajano:
1175 A pedagogically buggy example, along the lines of the previous listing, would
1176 be helpful here -- showing in more concrete terms what sort of actions could
1177 cause the problem. I can't very well imagine it from the description.
1178
1179
1180 .. _cplusplus:
1181
1182 Writing Extensions in C++
1183 =========================
1184
1185 It is possible to write extension modules in C++. Some restrictions apply. If
1186 the main program (the Python interpreter) is compiled and linked by the C
1187 compiler, global or static objects with constructors cannot be used. This is
1188 not a problem if the main program is linked by the C++ compiler. Functions that
1189 will be called by the Python interpreter (in particular, module initialization
1190 functions) have to be declared using ``extern "C"``. It is unnecessary to
1191 enclose the Python header files in ``extern "C" {...}`` --- they use this form
1192 already if the symbol ``__cplusplus`` is defined (all recent C++ compilers
1193 define this symbol).
1194
1195
1196 .. _using-capsules:
1197
1198 Providing a C API for an Extension Module
1199 =========================================
1200
1201 .. sectionauthor:: Konrad Hinsen <[email protected]>
1202
1203
1204 Many extension modules just provide new functions and types to be used from
1205 Python, but sometimes the code in an extension module can be useful for other
1206 extension modules. For example, an extension module could implement a type
1207 "collection" which works like lists without order. Just like the standard Python
1208 list type has a C API which permits extension modules to create and manipulate
1209 lists, this new collection type should have a set of C functions for direct
1210 manipulation from other extension modules.
1211
1212 At first sight this seems easy: just write the functions (without declaring them
1213 ``static``, of course), provide an appropriate header file, and document
1214 the C API. And in fact this would work if all extension modules were always
1215 linked statically with the Python interpreter. When modules are used as shared
1216 libraries, however, the symbols defined in one module may not be visible to
1217 another module. The details of visibility depend on the operating system; some
1218 systems use one global namespace for the Python interpreter and all extension
1219 modules (Windows, for example), whereas others require an explicit list of
1220 imported symbols at module link time (AIX is one example), or offer a choice of
1221 different strategies (most Unices). And even if symbols are globally visible,
1222 the module whose functions one wishes to call might not have been loaded yet!
1223
1224 Portability therefore requires not to make any assumptions about symbol
1225 visibility. This means that all symbols in extension modules should be declared
1226 ``static``, except for the module's initialization function, in order to
1227 avoid name clashes with other extension modules (as discussed in section
1228 :ref:`methodtable`). And it means that symbols that *should* be accessible from
1229 other extension modules must be exported in a different way.
1230
1231 Python provides a special mechanism to pass C-level information (pointers) from
1232 one extension module to another one: Capsules. A Capsule is a Python data type
1233 which stores a pointer (:c:expr:`void \*`). Capsules can only be created and
1234 accessed via their C API, but they can be passed around like any other Python
1235 object. In particular, they can be assigned to a name in an extension module's
1236 namespace. Other extension modules can then import this module, retrieve the
1237 value of this name, and then retrieve the pointer from the Capsule.
1238
1239 There are many ways in which Capsules can be used to export the C API of an
1240 extension module. Each function could get its own Capsule, or all C API pointers
1241 could be stored in an array whose address is published in a Capsule. And the
1242 various tasks of storing and retrieving the pointers can be distributed in
1243 different ways between the module providing the code and the client modules.
1244
1245 Whichever method you choose, it's important to name your Capsules properly.
1246 The function :c:func:`PyCapsule_New` takes a name parameter
1247 (:c:expr:`const char \*`); you're permitted to pass in a ``NULL`` name, but
1248 we strongly encourage you to specify a name. Properly named Capsules provide
1249 a degree of runtime type-safety; there is no feasible way to tell one unnamed
1250 Capsule from another.
1251
1252 In particular, Capsules used to expose C APIs should be given a name following
1253 this convention::
1254
1255 modulename.attributename
1256
1257 The convenience function :c:func:`PyCapsule_Import` makes it easy to
1258 load a C API provided via a Capsule, but only if the Capsule's name
1259 matches this convention. This behavior gives C API users a high degree
1260 of certainty that the Capsule they load contains the correct C API.
1261
1262 The following example demonstrates an approach that puts most of the burden on
1263 the writer of the exporting module, which is appropriate for commonly used
1264 library modules. It stores all C API pointers (just one in the example!) in an
1265 array of :c:expr:`void` pointers which becomes the value of a Capsule. The header
1266 file corresponding to the module provides a macro that takes care of importing
1267 the module and retrieving its C API pointers; client modules only have to call
1268 this macro before accessing the C API.
1269
1270 The exporting module is a modification of the :mod:`!spam` module from section
1271 :ref:`extending-simpleexample`. The function :func:`!spam.system` does not call
1272 the C library function :c:func:`system` directly, but a function
1273 :c:func:`!PySpam_System`, which would of course do something more complicated in
1274 reality (such as adding "spam" to every command). This function
1275 :c:func:`!PySpam_System` is also exported to other extension modules.
1276
1277 The function :c:func:`!PySpam_System` is a plain C function, declared
1278 ``static`` like everything else::
1279
1280 static int
1281 PySpam_System(const char *command)
1282 {
1283 return system(command);
1284 }
1285
1286 The function :c:func:`!spam_system` is modified in a trivial way::
1287
1288 static PyObject *
1289 spam_system(PyObject *self, PyObject *args)
1290 {
1291 const char *command;
1292 int sts;
1293
1294 if (!PyArg_ParseTuple(args, "s", &command))
1295 return NULL;
1296 sts = PySpam_System(command);
1297 return PyLong_FromLong(sts);
1298 }
1299
1300 In the beginning of the module, right after the line ::
1301
1302 #include <Python.h>
1303
1304 two more lines must be added::
1305
1306 #define SPAM_MODULE
1307 #include "spammodule.h"
1308
1309 The ``#define`` is used to tell the header file that it is being included in the
1310 exporting module, not a client module. Finally, the module's :c:data:`mod_exec
1311 <Py_mod_exec>` function must take care of initializing the C API pointer array::
1312
1313 static int
1314 spam_module_exec(PyObject *m)
1315 {
1316 static void *PySpam_API[PySpam_API_pointers];
1317 PyObject *c_api_object;
1318
1319 /* Initialize the C API pointer array */
1320 PySpam_API[PySpam_System_NUM] = (void *)PySpam_System;
1321
1322 /* Create a Capsule containing the API pointer array's address */
1323 c_api_object = PyCapsule_New((void *)PySpam_API, "spam._C_API", NULL);
1324
1325 if (PyModule_Add(m, "_C_API", c_api_object) < 0) {
1326 return -1;
1327 }
1328
1329 return 0;
1330 }
1331
1332 Note that ``PySpam_API`` is declared ``static``; otherwise the pointer
1333 array would disappear when :c:func:`!PyInit_spam` terminates!
1334
1335 The bulk of the work is in the header file :file:`spammodule.h`, which looks
1336 like this::
1337
1338 #ifndef Py_SPAMMODULE_H
1339 #define Py_SPAMMODULE_H
1340 #ifdef __cplusplus
1341 extern "C" {
1342 #endif
1343
1344 /* Header file for spammodule */
1345
1346 /* C API functions */
1347 #define PySpam_System_NUM 0
1348 #define PySpam_System_RETURN int
1349 #define PySpam_System_PROTO (const char *command)
1350
1351 /* Total number of C API pointers */
1352 #define PySpam_API_pointers 1
1353
1354
1355 #ifdef SPAM_MODULE
1356 /* This section is used when compiling spammodule.c */
1357
1358 static PySpam_System_RETURN PySpam_System PySpam_System_PROTO;
1359
1360 #else
1361 /* This section is used in modules that use spammodule's API */
1362
1363 static void **PySpam_API;
1364
1365 #define PySpam_System \
1366 (*(PySpam_System_RETURN (*)PySpam_System_PROTO) PySpam_API[PySpam_System_NUM])
1367
1368 /* Return -1 on error, 0 on success.
1369 * PyCapsule_Import will set an exception if there's an error.
1370 */
1371 static int
1372 import_spam(void)
1373 {
1374 PySpam_API = (void **)PyCapsule_Import("spam._C_API", 0);
1375 return (PySpam_API != NULL) ? 0 : -1;
1376 }
1377
1378 #endif
1379
1380 #ifdef __cplusplus
1381 }
1382 #endif
1383
1384 #endif /* !defined(Py_SPAMMODULE_H) */
1385
1386 All that a client module must do in order to have access to the function
1387 :c:func:`!PySpam_System` is to call the function (or rather macro)
1388 :c:func:`!import_spam` in its :c:data:`mod_exec <Py_mod_exec>` function::
1389
1390 static int
1391 client_module_exec(PyObject *m)
1392 {
1393 if (import_spam() < 0) {
1394 return -1;
1395 }
1396 /* additional initialization can happen here */
1397 return 0;
1398 }
1399
1400 The main disadvantage of this approach is that the file :file:`spammodule.h` is
1401 rather complicated. However, the basic structure is the same for each function
1402 that is exported, so it has to be learned only once.
1403
1404 Finally it should be mentioned that Capsules offer additional functionality,
1405 which is especially useful for memory allocation and deallocation of the pointer
1406 stored in a Capsule. The details are described in the Python/C API Reference
1407 Manual in the section :ref:`capsules` and in the implementation of Capsules (files
1408 :file:`Include/pycapsule.h` and :file:`Objects/pycapsule.c` in the Python source
1409 code distribution).
1410
1411 .. rubric:: Footnotes
1412
1413 .. [#] An interface for this function already exists in the standard module :mod:`os`
1414 --- it was chosen as a simple and straightforward example.
1415
1416 .. [#] The metaphor of "borrowing" a reference is not completely correct: the owner
1417 still has a copy of the reference.
1418
1419 .. [#] Checking that the reference count is at least 1 **does not work** --- the
1420 reference count itself could be in freed memory and may thus be reused for
1421 another object!
1422
1423 .. [#] These guarantees don't hold when you use the "old" style calling convention ---
1424 this is still found in much existing code.