Debian Codegraph @

Doc/c-api/intro.rst

raw
1 .. highlight:: c
2
3
4 .. _api-intro:
5
6 ************
7 Introduction
8 ************
9
10 The Application Programmer's Interface to Python gives C and C++ programmers
11 access to the Python interpreter at a variety of levels. The API is equally
12 usable from C++, but for brevity it is generally referred to as the Python/C
13 API. There are two fundamentally different reasons for using the Python/C API.
14 The first reason is to write *extension modules* for specific purposes; these
15 are C modules that extend the Python interpreter. This is probably the most
16 common use. The second reason is to use Python as a component in a larger
17 application; this technique is generally referred to as :dfn:`embedding` Python
18 in an application.
19
20 Writing an extension module is a relatively well-understood process, where a
21 "cookbook" approach works well. There are several tools that automate the
22 process to some extent. While people have embedded Python in other
23 applications since its early existence, the process of embedding Python is
24 less straightforward than writing an extension.
25
26 Many API functions are useful independent of whether you're embedding or
27 extending Python; moreover, most applications that embed Python will need to
28 provide a custom extension as well, so it's probably a good idea to become
29 familiar with writing an extension before attempting to embed Python in a real
30 application.
31
32
33 Language version compatibility
34 ==============================
35
36 Python's C API is compatible with C11 and C++11 versions of C and C++.
37
38 This is a lower limit: the C API does not require features from later
39 C/C++ versions.
40 You do *not* need to enable your compiler's "c11 mode".
41
42
43 Coding standards
44 ================
45
46 If you're writing C code for inclusion in CPython, you **must** follow the
47 guidelines and standards defined in :PEP:`7`. These guidelines apply
48 regardless of the version of Python you are contributing to. Following these
49 conventions is not necessary for your own third party extension modules,
50 unless you eventually expect to contribute them to Python.
51
52
53 .. _api-includes:
54
55 Include Files
56 =============
57
58 All function, type and macro definitions needed to use the Python/C API are
59 included in your code by the following line::
60
61 #define PY_SSIZE_T_CLEAN
62 #include <Python.h>
63
64 This implies inclusion of the following standard headers: ``<stdio.h>``,
65 ``<string.h>``, ``<errno.h>``, ``<limits.h>``, ``<assert.h>`` and ``<stdlib.h>``
66 (if available).
67
68 .. note::
69
70 Since Python may define some pre-processor definitions which affect the standard
71 headers on some systems, you *must* include :file:`Python.h` before any standard
72 headers are included.
73
74 It is recommended to always define ``PY_SSIZE_T_CLEAN`` before including
75 ``Python.h``. See :ref:`arg-parsing` for a description of this macro.
76
77 All user visible names defined by Python.h (except those defined by the included
78 standard headers) have one of the prefixes ``Py`` or ``_Py``. Names beginning
79 with ``_Py`` are for internal use by the Python implementation and should not be
80 used by extension writers. Structure member names do not have a reserved prefix.
81
82 .. note::
83
84 User code should never define names that begin with ``Py`` or ``_Py``. This
85 confuses the reader, and jeopardizes the portability of the user code to
86 future Python versions, which may define additional names beginning with one
87 of these prefixes.
88
89 The header files are typically installed with Python. On Unix, these are
90 located in the directories :file:`{prefix}/include/pythonversion/` and
91 :file:`{exec_prefix}/include/pythonversion/`, where :option:`prefix <--prefix>` and
92 :option:`exec_prefix <--exec-prefix>` are defined by the corresponding parameters to Python's
93 :program:`configure` script and *version* is
94 ``'%d.%d' % sys.version_info[:2]``. On Windows, the headers are installed
95 in :file:`{prefix}/include`, where ``prefix`` is the installation
96 directory specified to the installer.
97
98 To include the headers, place both directories (if different) on your compiler's
99 search path for includes. Do *not* place the parent directories on the search
100 path and then use ``#include <pythonX.Y/Python.h>``; this will break on
101 multi-platform builds since the platform independent headers under
102 :option:`prefix <--prefix>` include the platform specific headers from
103 :option:`exec_prefix <--exec-prefix>`.
104
105 C++ users should note that although the API is defined entirely using C, the
106 header files properly declare the entry points to be ``extern "C"``. As a result,
107 there is no need to do anything special to use the API from C++.
108
109
110 Useful macros
111 =============
112
113 Several useful macros are defined in the Python header files. Many are
114 defined closer to where they are useful (for example, :c:macro:`Py_RETURN_NONE`,
115 :c:macro:`PyMODINIT_FUNC`).
116 Others of a more general utility are defined here. This is not necessarily a
117 complete listing.
118
119 .. c:macro:: Py_CAN_START_THREADS
120
121 If this macro is defined, then the current system is able to start threads.
122
123 Currently, all systems supported by CPython (per :pep:`11`), with the
124 exception of some WebAssembly platforms, support starting threads.
125
126 .. versionadded:: 3.13
127
128 .. c:macro:: Py_GETENV(s)
129
130 Like :samp:`getenv({s})`, but returns ``NULL`` if :option:`-E` was passed
131 on the command line (see :c:member:`PyConfig.use_environment`).
132
133
134 Docstring macros
135 ----------------
136
137 .. c:macro:: PyDoc_STRVAR(name, str)
138
139 Creates a variable with name *name* that can be used in docstrings.
140 If Python is built without docstrings (:option:`--without-doc-strings`),
141 the value will be an empty string.
142
143 Example::
144
145 PyDoc_STRVAR(pop_doc, "Remove and return the rightmost element.");
146
147 static PyMethodDef deque_methods[] = {
148 // ...
149 {"pop", (PyCFunction)deque_pop, METH_NOARGS, pop_doc},
150 // ...
151 }
152
153 Expands to :samp:`PyDoc_VAR({name}) = PyDoc_STR({str})`.
154
155 .. c:macro:: PyDoc_STR(str)
156
157 Expands to the given input string, or an empty string
158 if docstrings are disabled (:option:`--without-doc-strings`).
159
160 Example::
161
162 static PyMethodDef pysqlite_row_methods[] = {
163 {"keys", (PyCFunction)pysqlite_row_keys, METH_NOARGS,
164 PyDoc_STR("Returns the keys of the row.")},
165 {NULL, NULL}
166 };
167
168 .. c:macro:: PyDoc_VAR(name)
169
170 Declares a static character array variable with the given *name*.
171 Expands to :samp:`static const char {name}[]`
172
173 For example::
174
175 PyDoc_VAR(python_doc) = PyDoc_STR(
176 "A genus of constricting snakes in the Pythonidae family native "
177 "to the tropics and subtropics of the Eastern Hemisphere.");
178
179
180 General utility macros
181 ----------------------
182
183 The following macros are for common tasks not specific to Python.
184
185 .. c:macro:: Py_UNUSED(arg)
186
187 Use this for unused arguments in a function definition to silence compiler
188 warnings. Example: ``int func(int a, int Py_UNUSED(b)) { return a; }``.
189
190 .. versionadded:: 3.4
191
192 .. c:macro:: Py_GCC_ATTRIBUTE(name)
193
194 Use a GCC attribute *name*, hiding it from compilers that don't support GCC
195 attributes (such as MSVC).
196
197 This expands to :samp:`__attribute__(({name)})` on a GCC compiler,
198 and expands to nothing on compilers that don't support GCC attributes.
199
200
201 Numeric utilities
202 ^^^^^^^^^^^^^^^^^
203
204 .. c:macro:: Py_ABS(x)
205
206 Return the absolute value of ``x``.
207
208 The argument may be evaluated more than once.
209 Consequently, do not pass an expression with side-effects directly
210 to this macro.
211
212 If the result cannot be represented (for example, if ``x`` has
213 :c:macro:`!INT_MIN` value for :c:expr:`int` type), the behavior is
214 undefined.
215
216 Corresponds roughly to :samp:`(({x}) < 0 ? -({x}) : ({x}))`
217
218 .. versionadded:: 3.3
219
220 .. c:macro:: Py_MAX(x, y)
221 Py_MIN(x, y)
222
223 Return the larger or smaller of the arguments, respectively.
224
225 Any arguments may be evaluated more than once.
226 Consequently, do not pass an expression with side-effects directly
227 to this macro.
228
229 :c:macro:`!Py_MAX` corresponds roughly to
230 :samp:`((({x}) > ({y})) ? ({x}) : ({y}))`.
231
232 .. versionadded:: 3.3
233
234 .. c:macro:: Py_ARITHMETIC_RIGHT_SHIFT(type, integer, positions)
235
236 Similar to :samp:`{integer} >> {positions}`, but forces sign extension,
237 as the C standard does not define whether a right-shift of a signed
238 integer will perform sign extension or a zero-fill.
239
240 *integer* should be any signed integer type.
241 *positions* is the number of positions to shift to the right.
242
243 Both *integer* and *positions* can be evaluated more than once;
244 consequently, avoid directly passing a function call or some other
245 operation with side-effects to this macro. Instead, store the result as a
246 variable and then pass it.
247
248 *type* is unused and only kept for backwards compatibility. Historically,
249 *type* was used to cast *integer*.
250
251 .. versionchanged:: 3.1
252
253 This macro is now valid for all signed integer types, not just those for
254 which ``unsigned type`` is legal. As a result, *type* is no longer
255 used.
256
257 .. c:macro:: Py_CHARMASK(c)
258
259 Argument must be a character or an integer in the range [-128, 127] or [0,
260 255]. This macro returns ``c`` cast to an ``unsigned char``.
261
262
263 Assertion utilities
264 ^^^^^^^^^^^^^^^^^^^
265
266 .. c:macro:: Py_UNREACHABLE()
267
268 Use this when you have a code path that cannot be reached by design.
269 For example, in the ``default:`` clause in a ``switch`` statement for which
270 all possible values are covered in ``case`` statements. Use this in places
271 where you might be tempted to put an ``assert(0)`` or ``abort()`` call.
272
273 In release mode, the macro helps the compiler to optimize the code, and
274 avoids a warning about unreachable code. For example, the macro is
275 implemented with ``__builtin_unreachable()`` on GCC in release mode.
276
277 In debug mode, and on unsupported compilers, the macro expands to a call to
278 :c:func:`Py_FatalError`.
279
280 A use for ``Py_UNREACHABLE()`` is following a call to a function that
281 never returns but that is not declared ``_Noreturn``.
282
283 If a code path is very unlikely code but can be reached under exceptional
284 case, this macro must not be used. For example, under low memory condition
285 or if a system call returns a value out of the expected range. In this
286 case, it's better to report the error to the caller. If the error cannot
287 be reported to caller, :c:func:`Py_FatalError` can be used.
288
289 .. versionadded:: 3.7
290
291 .. c:macro:: Py_SAFE_DOWNCAST(value, larger, smaller)
292
293 Cast *value* to type *smaller* from type *larger*, validating that no
294 information was lost.
295
296 On release builds of Python, this is roughly equivalent to
297 :samp:`(({smaller}) {value})`
298 (in C++, :samp:`static_cast<{smaller}>({value})` will be used instead).
299
300 On debug builds (implying that :c:macro:`Py_DEBUG` is defined), this asserts
301 that no information was lost with the cast from *larger* to *smaller*.
302
303 *value*, *larger*, and *smaller* may all be evaluated more than once in the
304 expression; consequently, do not pass an expression with side-effects
305 directly to this macro.
306
307 .. c:macro:: Py_BUILD_ASSERT(cond)
308
309 Asserts a compile-time condition *cond*, as a statement.
310 The build will fail if the condition is false or cannot be evaluated at compile time.
311
312 Corresponds roughly to :samp:`static_assert({cond})` on C23 and above.
313
314 For example::
315
316 Py_BUILD_ASSERT(sizeof(PyTime_t) == sizeof(int64_t));
317
318 .. versionadded:: 3.3
319
320 .. c:macro:: Py_BUILD_ASSERT_EXPR(cond)
321
322 Asserts a compile-time condition *cond*, as an expression that evaluates to ``0``.
323 The build will fail if the condition is false or cannot be evaluated at compile time.
324
325 For example::
326
327 #define foo_to_char(foo) \
328 ((char *)(foo) + Py_BUILD_ASSERT_EXPR(offsetof(struct foo, string) == 0))
329
330 .. versionadded:: 3.3
331
332
333 Type size utilities
334 ^^^^^^^^^^^^^^^^^^^
335
336 .. c:macro:: Py_ARRAY_LENGTH(array)
337
338 Compute the length of a statically allocated C array at compile time.
339
340 The *array* argument must be a C array with a size known at compile time.
341 Passing an array with an unknown size, such as a heap-allocated array,
342 will result in a compilation error on some compilers, or otherwise produce
343 incorrect results.
344
345 This is roughly equivalent to::
346
347 sizeof(array) / sizeof((array)[0])
348
349 .. c:macro:: Py_MEMBER_SIZE(type, member)
350
351 Return the size of a structure (*type*) *member* in bytes.
352
353 Corresponds roughly to :samp:`sizeof((({type} *)NULL)->{member})`.
354
355 .. versionadded:: 3.6
356
357
358 Macro definition utilities
359 ^^^^^^^^^^^^^^^^^^^^^^^^^^
360
361 .. c:macro:: Py_FORCE_EXPANSION(X)
362
363 This is equivalent to :samp:`{X}`, which is useful for token-pasting in
364 macros, as macro expansions in *X* are forcefully evaluated by the
365 preprocessor.
366
367 .. c:macro:: Py_STRINGIFY(x)
368
369 Convert ``x`` to a C string. For example, ``Py_STRINGIFY(123)`` returns
370 ``"123"``.
371
372 .. versionadded:: 3.4
373
374
375 Declaration utilities
376 ---------------------
377
378 The following macros can be used in declarations.
379 They are most useful for defining the C API itself, and have limited use
380 for extension authors.
381 Most of them expand to compiler-specific spellings of common extensions
382 to the C language.
383
384 .. c:macro:: Py_ALWAYS_INLINE
385
386 Ask the compiler to always inline a static inline function. The compiler can
387 ignore it and decide to not inline the function.
388
389 Corresponds to ``always_inline`` attribute in GCC and ``__forceinline``
390 in MSVC.
391
392 It can be used to inline performance critical static inline functions when
393 building Python in debug mode with function inlining disabled. For example,
394 MSC disables function inlining when building in debug mode.
395
396 Marking blindly a static inline function with Py_ALWAYS_INLINE can result in
397 worse performances (due to increased code size for example). The compiler is
398 usually smarter than the developer for the cost/benefit analysis.
399
400 If Python is :ref:`built in debug mode <debug-build>` (if the :c:macro:`Py_DEBUG`
401 macro is defined), the :c:macro:`Py_ALWAYS_INLINE` macro does nothing.
402
403 It must be specified before the function return type. Usage::
404
405 static inline Py_ALWAYS_INLINE int random(void) { return 4; }
406
407 .. versionadded:: 3.11
408
409 .. c:macro:: Py_NO_INLINE
410
411 Disable inlining on a function. For example, it reduces the C stack
412 consumption: useful on LTO+PGO builds which heavily inline code (see
413 :issue:`33720`).
414
415 Corresponds to the ``noinline`` attribute/specification on GCC and MSVC.
416
417 Usage::
418
419 Py_NO_INLINE static int random(void) { return 4; }
420
421 .. versionadded:: 3.11
422
423 .. c:macro:: Py_DEPRECATED(version)
424
425 Use this to declare APIs that were deprecated in a specific CPython version.
426 The macro must be placed before the symbol name.
427
428 Example::
429
430 Py_DEPRECATED(3.8) PyAPI_FUNC(int) Py_OldFunction(void);
431
432 .. versionchanged:: 3.8
433 MSVC support was added.
434
435 .. c:macro:: Py_LOCAL(type)
436
437 Declare a function returning the specified *type* using a fast-calling
438 qualifier for functions that are local to the current file.
439 Semantically, this is equivalent to :samp:`static {type}`.
440
441 .. c:macro:: Py_LOCAL_INLINE(type)
442
443 Equivalent to :c:macro:`Py_LOCAL` but additionally requests the function
444 be inlined.
445
446 .. c:macro:: Py_LOCAL_SYMBOL
447
448 Macro used to declare a symbol as local to the shared library (hidden).
449 On supported platforms, it ensures the symbol is not exported.
450
451 On compatible versions of GCC/Clang, it
452 expands to ``__attribute__((visibility("hidden")))``.
453
454 .. c:macro:: Py_EXPORTED_SYMBOL
455
456 Macro used to declare a symbol (function or data) as exported.
457 On Windows, this expands to ``__declspec(dllexport)``.
458 On compatible versions of GCC/Clang, it
459 expands to ``__attribute__((visibility("default")))``.
460 This macro is for defining the C API itself; extension modules should not use it.
461
462
463 .. c:macro:: Py_IMPORTED_SYMBOL
464
465 Macro used to declare a symbol as imported.
466 On Windows, this expands to ``__declspec(dllimport)``.
467 This macro is for defining the C API itself; extension modules should not use it.
468
469
470 .. c:macro:: PyAPI_FUNC(type)
471
472 Macro used by CPython to declare a function as part of the C API.
473 Its expansion depends on the platform and build configuration.
474 This macro is intended for defining CPython's C API itself;
475 extension modules should not use it for their own symbols.
476
477
478 .. c:macro:: PyAPI_DATA(type)
479
480 Macro used by CPython to declare a public global variable as part of the C API.
481 Its expansion depends on the platform and build configuration.
482 This macro is intended for defining CPython's C API itself;
483 extension modules should not use it for their own symbols.
484
485
486 Outdated macros
487 ---------------
488
489 The following macros have been used to features that have been standardized
490 in C11.
491
492 .. c:macro:: Py_ALIGNED(num)
493
494 Specify alignment to *num* bytes on compilers that support it.
495
496 Consider using the C11 standard ``_Alignas`` specifier over this macro.
497
498 .. c:macro:: Py_LL(number)
499 Py_ULL(number)
500
501 Use *number* as a ``long long`` or ``unsigned long long`` integer literal,
502 respectively.
503
504 Expands to *number* followed by ``LL`` or ``LLU``, respectively, but will
505 expand to some compiler-specific suffixes on some older compilers.
506
507 Consider using the C99 standard suffixes ``LL`` and ``LLU`` directly.
508
509 .. c:macro:: Py_MEMCPY(dest, src, n)
510
511 This is a :term:`soft deprecated` alias to :c:func:`!memcpy`.
512 Use :c:func:`!memcpy` directly instead.
513
514 .. deprecated:: 3.14
515 The macro is :term:`soft deprecated`.
516
517 .. c:macro:: Py_VA_COPY
518
519 This is a :term:`soft deprecated` alias to the C99-standard ``va_copy``
520 function.
521
522 Historically, this would use a compiler-specific method to copy a ``va_list``.
523
524 .. versionchanged:: 3.6
525 This is now an alias to ``va_copy``.
526
527
528 .. _api-objects:
529
530 Objects, Types and Reference Counts
531 ===================================
532
533 .. index:: pair: object; type
534
535 Most Python/C API functions have one or more arguments as well as a return value
536 of type :c:expr:`PyObject*`. This type is a pointer to an opaque data type
537 representing an arbitrary Python object. Since all Python object types are
538 treated the same way by the Python language in most situations (e.g.,
539 assignments, scope rules, and argument passing), it is only fitting that they
540 should be represented by a single C type. Almost all Python objects live on the
541 heap: you never declare an automatic or static variable of type
542 :c:type:`PyObject`, only pointer variables of type :c:expr:`PyObject*` can be
543 declared. The sole exception are the type objects; since these must never be
544 deallocated, they are typically static :c:type:`PyTypeObject` objects.
545
546 All Python objects (even Python integers) have a :dfn:`type` and a
547 :dfn:`reference count`. An object's type determines what kind of object it is
548 (e.g., an integer, a list, or a user-defined function; there are many more as
549 explained in :ref:`types`). For each of the well-known types there is a macro
550 to check whether an object is of that type; for instance, ``PyList_Check(a)`` is
551 true if (and only if) the object pointed to by *a* is a Python list.
552
553
554 .. _api-refcounts:
555
556 Reference Counts
557 ----------------
558
559 The reference count is important because today's computers have a finite
560 (and often severely limited) memory size; it counts how many different
561 places there are that have a :term:`strong reference` to an object.
562 Such a place could be another object, or a global (or static) C variable,
563 or a local variable in some C function.
564 When the last :term:`strong reference` to an object is released
565 (i.e. its reference count becomes zero), the object is deallocated.
566 If it contains references to other objects, those references are released.
567 Those other objects may be deallocated in turn, if there are no more
568 references to them, and so on. (There's an obvious problem with
569 objects that reference each other here; for now, the solution
570 is "don't do that.")
571
572 .. index::
573 single: Py_INCREF (C function)
574 single: Py_DECREF (C function)
575
576 Reference counts are always manipulated explicitly. The normal way is
577 to use the macro :c:func:`Py_INCREF` to take a new reference to an
578 object (i.e. increment its reference count by one),
579 and :c:func:`Py_DECREF` to release that reference (i.e. decrement the
580 reference count by one). The :c:func:`Py_DECREF` macro
581 is considerably more complex than the incref one, since it must check whether
582 the reference count becomes zero and then cause the object's deallocator to be
583 called. The deallocator is a function pointer contained in the object's type
584 structure. The type-specific deallocator takes care of releasing references
585 for other objects contained in the object if this is a compound
586 object type, such as a list, as well as performing any additional finalization
587 that's needed. There's no chance that the reference count can overflow; at
588 least as many bits are used to hold the reference count as there are distinct
589 memory locations in virtual memory (assuming ``sizeof(Py_ssize_t) >= sizeof(void*)``).
590 Thus, the reference count increment is a simple operation.
591
592 It is not necessary to hold a :term:`strong reference` (i.e. increment
593 the reference count) for every local variable that contains a pointer
594 to an object. In theory, the object's
595 reference count goes up by one when the variable is made to point to it and it
596 goes down by one when the variable goes out of scope. However, these two
597 cancel each other out, so at the end the reference count hasn't changed. The
598 only real reason to use the reference count is to prevent the object from being
599 deallocated as long as our variable is pointing to it. If we know that there
600 is at least one other reference to the object that lives at least as long as
601 our variable, there is no need to take a new :term:`strong reference`
602 (i.e. increment the reference count) temporarily.
603 An important situation where this arises is in objects that are passed as
604 arguments to C functions in an extension module that are called from Python;
605 the call mechanism guarantees to hold a reference to every argument for the
606 duration of the call.
607
608 However, a common pitfall is to extract an object from a list and hold on to it
609 for a while without taking a new reference. Some other operation might
610 conceivably remove the object from the list, releasing that reference,
611 and possibly deallocating it. The real danger is that innocent-looking
612 operations may invoke arbitrary Python code which could do this; there is a code
613 path which allows control to flow back to the user from a :c:func:`Py_DECREF`, so
614 almost any operation is potentially dangerous.
615
616 A safe approach is to always use the generic operations (functions whose name
617 begins with ``PyObject_``, ``PyNumber_``, ``PySequence_`` or ``PyMapping_``).
618 These operations always create a new :term:`strong reference`
619 (i.e. increment the reference count) of the object they return.
620 This leaves the caller with the responsibility to call :c:func:`Py_DECREF` when
621 they are done with the result; this soon becomes second nature.
622
623
624 .. _api-refcountdetails:
625
626 Reference Count Details
627 ^^^^^^^^^^^^^^^^^^^^^^^
628
629 The reference count behavior of functions in the Python/C API is best explained
630 in terms of *ownership of references*. Ownership pertains to references, never
631 to objects (objects are not owned: they are always shared). "Owning a
632 reference" means being responsible for calling Py_DECREF on it when the
633 reference is no longer needed. Ownership can also be transferred, meaning that
634 the code that receives ownership of the reference then becomes responsible for
635 eventually releasing it by calling :c:func:`Py_DECREF` or :c:func:`Py_XDECREF`
636 when it's no longer needed---or passing on this responsibility (usually to its
637 caller). When a function passes ownership of a reference on to its caller, the
638 caller is said to receive a *new* reference. When no ownership is transferred,
639 the caller is said to *borrow* the reference. Nothing needs to be done for a
640 :term:`borrowed reference`.
641
642 Conversely, when a calling function passes in a reference to an object, there
643 are two possibilities: the function *steals* a reference to the object, or it
644 does not. *Stealing a reference* means that when you pass a reference to a
645 function, that function assumes that it now owns that reference, and you are not
646 responsible for it any longer.
647
648 .. index::
649 single: PyList_SetItem (C function)
650 single: PyTuple_SetItem (C function)
651
652 Few functions steal references; the two notable exceptions are
653 :c:func:`PyList_SetItem` and :c:func:`PyTuple_SetItem`, which steal a reference
654 to the item (but not to the tuple or list into which the item is put!). These
655 functions were designed to steal a reference because of a common idiom for
656 populating a tuple or list with newly created objects; for example, the code to
657 create the tuple ``(1, 2, "three")`` could look like this (forgetting about
658 error handling for the moment; a better way to code this is shown below)::
659
660 PyObject *t;
661
662 t = PyTuple_New(3);
663 PyTuple_SetItem(t, 0, PyLong_FromLong(1L));
664 PyTuple_SetItem(t, 1, PyLong_FromLong(2L));
665 PyTuple_SetItem(t, 2, PyUnicode_FromString("three"));
666
667 Here, :c:func:`PyLong_FromLong` returns a new reference which is immediately
668 stolen by :c:func:`PyTuple_SetItem`. When you want to keep using an object
669 although the reference to it will be stolen, use :c:func:`Py_INCREF` to grab
670 another reference before calling the reference-stealing function.
671
672 Incidentally, :c:func:`PyTuple_SetItem` is the *only* way to set tuple items;
673 :c:func:`PySequence_SetItem` and :c:func:`PyObject_SetItem` refuse to do this
674 since tuples are an immutable data type. You should only use
675 :c:func:`PyTuple_SetItem` for tuples that you are creating yourself.
676
677 Equivalent code for populating a list can be written using :c:func:`PyList_New`
678 and :c:func:`PyList_SetItem`.
679
680 However, in practice, you will rarely use these ways of creating and populating
681 a tuple or list. There's a generic function, :c:func:`Py_BuildValue`, that can
682 create most common objects from C values, directed by a :dfn:`format string`.
683 For example, the above two blocks of code could be replaced by the following
684 (which also takes care of the error checking)::
685
686 PyObject *tuple, *list;
687
688 tuple = Py_BuildValue("(iis)", 1, 2, "three");
689 list = Py_BuildValue("[iis]", 1, 2, "three");
690
691 It is much more common to use :c:func:`PyObject_SetItem` and friends with items
692 whose references you are only borrowing, like arguments that were passed in to
693 the function you are writing. In that case, their behaviour regarding references
694 is much saner, since you don't have to take a new reference just so you
695 can give that reference away ("have it be stolen"). For example, this function
696 sets all items of a list (actually, any mutable sequence) to a given item::
697
698 int
699 set_all(PyObject *target, PyObject *item)
700 {
701 Py_ssize_t i, n;
702
703 n = PyObject_Length(target);
704 if (n < 0)
705 return -1;
706 for (i = 0; i < n; i++) {
707 PyObject *index = PyLong_FromSsize_t(i);
708 if (!index)
709 return -1;
710 if (PyObject_SetItem(target, index, item) < 0) {
711 Py_DECREF(index);
712 return -1;
713 }
714 Py_DECREF(index);
715 }
716 return 0;
717 }
718
719 .. index:: single: set_all()
720
721 The situation is slightly different for function return values. While passing
722 a reference to most functions does not change your ownership responsibilities
723 for that reference, many functions that return a reference to an object give
724 you ownership of the reference. The reason is simple: in many cases, the
725 returned object is created on the fly, and the reference you get is the only
726 reference to the object. Therefore, the generic functions that return object
727 references, like :c:func:`PyObject_GetItem` and :c:func:`PySequence_GetItem`,
728 always return a new reference (the caller becomes the owner of the reference).
729
730 It is important to realize that whether you own a reference returned by a
731 function depends on which function you call only --- *the plumage* (the type of
732 the object passed as an argument to the function) *doesn't enter into it!*
733 Thus, if you extract an item from a list using :c:func:`PyList_GetItem`, you
734 don't own the reference --- but if you obtain the same item from the same list
735 using :c:func:`PySequence_GetItem` (which happens to take exactly the same
736 arguments), you do own a reference to the returned object.
737
738 .. index::
739 single: PyList_GetItem (C function)
740 single: PySequence_GetItem (C function)
741
742 Here is an example of how you could write a function that computes the sum of
743 the items in a list of integers; once using :c:func:`PyList_GetItem`, and once
744 using :c:func:`PySequence_GetItem`. ::
745
746 long
747 sum_list(PyObject *list)
748 {
749 Py_ssize_t i, n;
750 long total = 0, value;
751 PyObject *item;
752
753 n = PyList_Size(list);
754 if (n < 0)
755 return -1; /* Not a list */
756 for (i = 0; i < n; i++) {
757 item = PyList_GetItem(list, i); /* Can't fail */
758 if (!PyLong_Check(item)) continue; /* Skip non-integers */
759 value = PyLong_AsLong(item);
760 if (value == -1 && PyErr_Occurred())
761 /* Integer too big to fit in a C long, bail out */
762 return -1;
763 total += value;
764 }
765 return total;
766 }
767
768 .. index:: single: sum_list()
769
770 ::
771
772 long
773 sum_sequence(PyObject *sequence)
774 {
775 Py_ssize_t i, n;
776 long total = 0, value;
777 PyObject *item;
778 n = PySequence_Length(sequence);
779 if (n < 0)
780 return -1; /* Has no length */
781 for (i = 0; i < n; i++) {
782 item = PySequence_GetItem(sequence, i);
783 if (item == NULL)
784 return -1; /* Not a sequence, or other failure */
785 if (PyLong_Check(item)) {
786 value = PyLong_AsLong(item);
787 Py_DECREF(item);
788 if (value == -1 && PyErr_Occurred())
789 /* Integer too big to fit in a C long, bail out */
790 return -1;
791 total += value;
792 }
793 else {
794 Py_DECREF(item); /* Discard reference ownership */
795 }
796 }
797 return total;
798 }
799
800 .. index:: single: sum_sequence()
801
802
803 .. _api-types:
804
805 Types
806 -----
807
808 There are few other data types that play a significant role in the Python/C
809 API; most are simple C types such as :c:expr:`int`, :c:expr:`long`,
810 :c:expr:`double` and :c:expr:`char*`. A few structure types are used to
811 describe static tables used to list the functions exported by a module or the
812 data attributes of a new object type, and another is used to describe the value
813 of a complex number. These will be discussed together with the functions that
814 use them.
815
816 .. c:type:: Py_ssize_t
817
818 A signed integral type such that ``sizeof(Py_ssize_t) == sizeof(size_t)``.
819 C99 doesn't define such a thing directly (size_t is an unsigned integral type).
820 See :pep:`353` for details. ``PY_SSIZE_T_MAX`` is the largest positive value
821 of type :c:type:`Py_ssize_t`.
822
823
824 .. _api-exceptions:
825
826 Exceptions
827 ==========
828
829 The Python programmer only needs to deal with exceptions if specific error
830 handling is required; unhandled exceptions are automatically propagated to the
831 caller, then to the caller's caller, and so on, until they reach the top-level
832 interpreter, where they are reported to the user accompanied by a stack
833 traceback.
834
835 .. index:: single: PyErr_Occurred (C function)
836
837 For C programmers, however, error checking always has to be explicit. All
838 functions in the Python/C API can raise exceptions, unless an explicit claim is
839 made otherwise in a function's documentation. In general, when a function
840 encounters an error, it sets an exception, discards any object references that
841 it owns, and returns an error indicator. If not documented otherwise, this
842 indicator is either ``NULL`` or ``-1``, depending on the function's return type.
843 A few functions return a Boolean true/false result, with false indicating an
844 error. Very few functions return no explicit error indicator or have an
845 ambiguous return value, and require explicit testing for errors with
846 :c:func:`PyErr_Occurred`. These exceptions are always explicitly documented.
847
848 .. index::
849 single: PyErr_SetString (C function)
850 single: PyErr_Clear (C function)
851
852 Exception state is maintained in per-thread storage (this is equivalent to
853 using global storage in an unthreaded application). A thread can be in one of
854 two states: an exception has occurred, or not. The function
855 :c:func:`PyErr_Occurred` can be used to check for this: it returns a borrowed
856 reference to the exception type object when an exception has occurred, and
857 ``NULL`` otherwise. There are a number of functions to set the exception state:
858 :c:func:`PyErr_SetString` is the most common (though not the most general)
859 function to set the exception state, and :c:func:`PyErr_Clear` clears the
860 exception state.
861
862 The full exception state consists of three objects (all of which can be
863 ``NULL``): the exception type, the corresponding exception value, and the
864 traceback. These have the same meanings as the Python result of
865 ``sys.exc_info()``; however, they are not the same: the Python objects represent
866 the last exception being handled by a Python :keyword:`try` ...
867 :keyword:`except` statement, while the C level exception state only exists while
868 an exception is being passed on between C functions until it reaches the Python
869 bytecode interpreter's main loop, which takes care of transferring it to
870 ``sys.exc_info()`` and friends.
871
872 .. index:: single: exc_info (in module sys)
873
874 Note that starting with Python 1.5, the preferred, thread-safe way to access the
875 exception state from Python code is to call the function :func:`sys.exc_info`,
876 which returns the per-thread exception state for Python code. Also, the
877 semantics of both ways to access the exception state have changed so that a
878 function which catches an exception will save and restore its thread's exception
879 state so as to preserve the exception state of its caller. This prevents common
880 bugs in exception handling code caused by an innocent-looking function
881 overwriting the exception being handled; it also reduces the often unwanted
882 lifetime extension for objects that are referenced by the stack frames in the
883 traceback.
884
885 As a general principle, a function that calls another function to perform some
886 task should check whether the called function raised an exception, and if so,
887 pass the exception state on to its caller. It should discard any object
888 references that it owns, and return an error indicator, but it should *not* set
889 another exception --- that would overwrite the exception that was just raised,
890 and lose important information about the exact cause of the error.
891
892 .. index:: single: sum_sequence()
893
894 A simple example of detecting exceptions and passing them on is shown in the
895 :c:func:`!sum_sequence` example above. It so happens that this example doesn't
896 need to clean up any owned references when it detects an error. The following
897 example function shows some error cleanup. First, to remind you why you like
898 Python, we show the equivalent Python code::
899
900 def incr_item(dict, key):
901 try:
902 item = dict[key]
903 except KeyError:
904 item = 0
905 dict[key] = item + 1
906
907 .. index:: single: incr_item()
908
909 Here is the corresponding C code, in all its glory::
910
911 int
912 incr_item(PyObject *dict, PyObject *key)
913 {
914 /* Objects all initialized to NULL for Py_XDECREF */
915 PyObject *item = NULL, *const_one = NULL, *incremented_item = NULL;
916 int rv = -1; /* Return value initialized to -1 (failure) */
917
918 item = PyObject_GetItem(dict, key);
919 if (item == NULL) {
920 /* Handle KeyError only: */
921 if (!PyErr_ExceptionMatches(PyExc_KeyError))
922 goto error;
923
924 /* Clear the error and use zero: */
925 PyErr_Clear();
926 item = PyLong_FromLong(0L);
927 if (item == NULL)
928 goto error;
929 }
930 const_one = PyLong_FromLong(1L);
931 if (const_one == NULL)
932 goto error;
933
934 incremented_item = PyNumber_Add(item, const_one);
935 if (incremented_item == NULL)
936 goto error;
937
938 if (PyObject_SetItem(dict, key, incremented_item) < 0)
939 goto error;
940 rv = 0; /* Success */
941 /* Continue with cleanup code */
942
943 error:
944 /* Cleanup code, shared by success and failure path */
945
946 /* Use Py_XDECREF() to ignore NULL references */
947 Py_XDECREF(item);
948 Py_XDECREF(const_one);
949 Py_XDECREF(incremented_item);
950
951 return rv; /* -1 for error, 0 for success */
952 }
953
954 .. index:: single: incr_item()
955
956 .. index::
957 single: PyErr_ExceptionMatches (C function)
958 single: PyErr_Clear (C function)
959 single: Py_XDECREF (C function)
960
961 This example represents an endorsed use of the ``goto`` statement in C!
962 It illustrates the use of :c:func:`PyErr_ExceptionMatches` and
963 :c:func:`PyErr_Clear` to handle specific exceptions, and the use of
964 :c:func:`Py_XDECREF` to dispose of owned references that may be ``NULL`` (note the
965 ``'X'`` in the name; :c:func:`Py_DECREF` would crash when confronted with a
966 ``NULL`` reference). It is important that the variables used to hold owned
967 references are initialized to ``NULL`` for this to work; likewise, the proposed
968 return value is initialized to ``-1`` (failure) and only set to success after
969 the final call made is successful.
970
971
972 .. _api-embedding:
973
974 Embedding Python
975 ================
976
977 The one important task that only embedders (as opposed to extension writers) of
978 the Python interpreter have to worry about is the initialization, and possibly
979 the finalization, of the Python interpreter. Most functionality of the
980 interpreter can only be used after the interpreter has been initialized.
981
982 .. index::
983 single: Py_Initialize (C function)
984 pair: module; builtins
985 pair: module; __main__
986 pair: module; sys
987 triple: module; search; path
988 single: path (in module sys)
989
990 The basic initialization function is :c:func:`Py_Initialize`. This initializes
991 the table of loaded modules, and creates the fundamental modules
992 :mod:`builtins`, :mod:`__main__`, and :mod:`sys`. It also
993 initializes the module search path (``sys.path``).
994
995 :c:func:`Py_Initialize` does not set the "script argument list" (``sys.argv``).
996 If this variable is needed by Python code that will be executed later, setting
997 :c:member:`PyConfig.argv` and :c:member:`PyConfig.parse_argv` must be set: see
998 :ref:`Python Initialization Configuration <init-config>`.
999
1000 On most systems (in particular, on Unix and Windows, although the details are
1001 slightly different), :c:func:`Py_Initialize` calculates the module search path
1002 based upon its best guess for the location of the standard Python interpreter
1003 executable, assuming that the Python library is found in a fixed location
1004 relative to the Python interpreter executable. In particular, it looks for a
1005 directory named :file:`lib/python{X.Y}` relative to the parent directory
1006 where the executable named :file:`python` is found on the shell command search
1007 path (the environment variable :envvar:`PATH`).
1008
1009 For instance, if the Python executable is found in
1010 :file:`/usr/local/bin/python`, it will assume that the libraries are in
1011 :file:`/usr/local/lib/python{X.Y}`. (In fact, this particular path is also
1012 the "fallback" location, used when no executable file named :file:`python` is
1013 found along :envvar:`PATH`.) The user can override this behavior by setting the
1014 environment variable :envvar:`PYTHONHOME`, or insert additional directories in
1015 front of the standard path by setting :envvar:`PYTHONPATH`.
1016
1017 .. index::
1018 single: Py_GetPath (C function)
1019 single: Py_GetPrefix (C function)
1020 single: Py_GetExecPrefix (C function)
1021 single: Py_GetProgramFullPath (C function)
1022
1023 The embedding application can steer the search by setting
1024 :c:member:`PyConfig.program_name` *before* calling
1025 :c:func:`Py_InitializeFromConfig`. Note that
1026 :envvar:`PYTHONHOME` still overrides this and :envvar:`PYTHONPATH` is still
1027 inserted in front of the standard path. An application that requires total
1028 control has to provide its own implementation of :c:func:`Py_GetPath`,
1029 :c:func:`Py_GetPrefix`, :c:func:`Py_GetExecPrefix`, and
1030 :c:func:`Py_GetProgramFullPath` (all defined in :file:`Modules/getpath.c`).
1031
1032 .. index:: single: Py_IsInitialized (C function)
1033
1034 Sometimes, it is desirable to "uninitialize" Python. For instance, the
1035 application may want to start over (make another call to
1036 :c:func:`Py_Initialize`) or the application is simply done with its use of
1037 Python and wants to free memory allocated by Python. This can be accomplished
1038 by calling :c:func:`Py_FinalizeEx`. The function :c:func:`Py_IsInitialized` returns
1039 true if Python is currently in the initialized state. More information about
1040 these functions is given in a later chapter. Notice that :c:func:`Py_FinalizeEx`
1041 does *not* free all memory allocated by the Python interpreter, e.g. memory
1042 allocated by extension modules currently cannot be released.
1043
1044
1045 .. _api-debugging:
1046
1047 Debugging Builds
1048 ================
1049
1050 Python can be built with several macros to enable extra checks of the
1051 interpreter and extension modules. These checks tend to add a large amount of
1052 overhead to the runtime so they are not enabled by default.
1053
1054 A full list of the various types of debugging builds is in the file
1055 :file:`Misc/SpecialBuilds.txt` in the Python source distribution. Builds are
1056 available that support tracing of reference counts, debugging the memory
1057 allocator, or low-level profiling of the main interpreter loop. Only the most
1058 frequently used builds will be described in the remainder of this section.
1059
1060 .. c:macro:: Py_DEBUG
1061
1062 Compiling the interpreter with the :c:macro:`!Py_DEBUG` macro defined produces
1063 what is generally meant by :ref:`a debug build of Python <debug-build>`.
1064 :c:macro:`!Py_DEBUG` is enabled in the Unix build by adding
1065 :option:`--with-pydebug` to the :file:`./configure` command.
1066 It is also implied by the presence of the
1067 not-Python-specific :c:macro:`!_DEBUG` macro. When :c:macro:`!Py_DEBUG` is enabled
1068 in the Unix build, compiler optimization is disabled.
1069
1070 In addition to the reference count debugging described below, extra checks are
1071 performed, see :ref:`Python Debug Build <debug-build>`.
1072
1073 Defining :c:macro:`Py_TRACE_REFS` enables reference tracing
1074 (see the :option:`configure --with-trace-refs option <--with-trace-refs>`).
1075 When defined, a circular doubly linked list of active objects is maintained by adding two extra
1076 fields to every :c:type:`PyObject`. Total allocations are tracked as well. Upon
1077 exit, all existing references are printed. (In interactive mode this happens
1078 after every statement run by the interpreter.)
1079
1080 Please refer to :file:`Misc/SpecialBuilds.txt` in the Python source distribution
1081 for more detailed information.
1082
1083
1084 .. _c-api-tools:
1085
1086 Recommended third party tools
1087 =============================
1088
1089 The following third party tools offer both simpler and more sophisticated
1090 approaches to creating C, C++ and Rust extensions for Python:
1091
1092 * `Cython <https://cython.org/>`_
1093 * `cffi <https://cffi.readthedocs.io>`_
1094 * `HPy <https://hpyproject.org/>`_
1095 * `nanobind <https://github.com/wjakob/nanobind>`_ (C++)
1096 * `Numba <https://numba.pydata.org/>`_
1097 * `pybind11 <https://pybind11.readthedocs.io/>`_ (C++)
1098 * `PyO3 <https://pyo3.rs/>`_ (Rust)
1099 * `SWIG <https://www.swig.org>`_
1100
1101 Using tools such as these can help avoid writing code that is tightly bound to
1102 a particular version of CPython, avoid reference counting errors, and focus
1103 more on your own code than on using the CPython API. In general, new versions
1104 of Python can be supported by updating the tool, and your code will often use
1105 newer and more efficient APIs automatically. Some tools also support compiling
1106 for other implementations of Python from a single set of sources.
1107
1108 These projects are not supported by the same people who maintain Python, and
1109 issues need to be raised with the projects directly. Remember to check that the
1110 project is still maintained and supported, as the list above may become
1111 outdated.
1112
1113 .. seealso::
1114
1115 `Python Packaging User Guide: Binary Extensions <https://packaging.python.org/guides/packaging-binary-extensions/>`_
1116 The Python Packaging User Guide not only covers several available
1117 tools that simplify the creation of binary extensions, but also
1118 discusses the various reasons why creating an extension module may be
1119 desirable in the first place.