Debian Codegraph @

Doc/extending/newtypes_tutorial.rst

raw
1 .. highlight:: c
2
3 .. _defining-new-types:
4
5 **********************************
6 Defining Extension Types: Tutorial
7 **********************************
8
9 .. sectionauthor:: Michael Hudson <[email protected]>
10 .. sectionauthor:: Dave Kuhlman <[email protected]>
11 .. sectionauthor:: Jim Fulton <[email protected]>
12
13
14 Python allows the writer of a C extension module to define new types that
15 can be manipulated from Python code, much like the built-in :class:`str`
16 and :class:`list` types. The code for all extension types follows a
17 pattern, but there are some details that you need to understand before you
18 can get started. This document is a gentle introduction to the topic.
19
20
21 .. _dnt-basics:
22
23 The Basics
24 ==========
25
26 The :term:`CPython` runtime sees all Python objects as variables of type
27 :c:expr:`PyObject*`, which serves as a "base type" for all Python objects.
28 The :c:type:`PyObject` structure itself only contains the object's
29 :term:`reference count` and a pointer to the object's "type object".
30 This is where the action is; the type object determines which (C) functions
31 get called by the interpreter when, for instance, an attribute gets looked up
32 on an object, a method called, or it is multiplied by another object. These
33 C functions are called "type methods".
34
35 So, if you want to define a new extension type, you need to create a new type
36 object.
37
38 This sort of thing can only be explained by example, so here's a minimal, but
39 complete, module that defines a new type named :class:`!Custom` inside a C
40 extension module :mod:`!custom`:
41
42 .. note::
43 What we're showing here is the traditional way of defining *static*
44 extension types. It should be adequate for most uses. The C API also
45 allows defining heap-allocated extension types using the
46 :c:func:`PyType_FromSpec` function, which isn't covered in this tutorial.
47
48 .. literalinclude:: ../includes/newtypes/custom.c
49
50 Now that's quite a bit to take in at once, but hopefully bits will seem familiar
51 from the previous chapter. This file defines three things:
52
53 #. What a :class:`!Custom` **object** contains: this is the ``CustomObject``
54 struct, which is allocated once for each :class:`!Custom` instance.
55 #. How the :class:`!Custom` **type** behaves: this is the ``CustomType`` struct,
56 which defines a set of flags and function pointers that the interpreter
57 inspects when specific operations are requested.
58 #. How to define and execute the :mod:`!custom` module: this is the
59 ``PyInit_custom`` function and the associated ``custom_module`` struct for
60 defining the module, and the ``custom_module_exec`` function to set up
61 a fresh module object.
62
63 The first bit is::
64
65 typedef struct {
66 PyObject_HEAD
67 } CustomObject;
68
69 This is what a Custom object will contain. ``PyObject_HEAD`` is mandatory
70 at the start of each object struct and defines a field called ``ob_base``
71 of type :c:type:`PyObject`, containing a pointer to a type object and a
72 reference count (these can be accessed using the macros :c:macro:`Py_TYPE`
73 and :c:macro:`Py_REFCNT` respectively). The reason for the macro is to
74 abstract away the layout and to enable additional fields in :ref:`debug builds
75 <debug-build>`.
76
77 .. note::
78 There is no semicolon above after the :c:macro:`PyObject_HEAD` macro.
79 Be wary of adding one by accident: some compilers will complain.
80
81 Of course, objects generally store additional data besides the standard
82 ``PyObject_HEAD`` boilerplate; for example, here is the definition for
83 standard Python floats::
84
85 typedef struct {
86 PyObject_HEAD
87 double ob_fval;
88 } PyFloatObject;
89
90 The second bit is the definition of the type object. ::
91
92 static PyTypeObject CustomType = {
93 .ob_base = PyVarObject_HEAD_INIT(NULL, 0)
94 .tp_name = "custom.Custom",
95 .tp_doc = PyDoc_STR("Custom objects"),
96 .tp_basicsize = sizeof(CustomObject),
97 .tp_itemsize = 0,
98 .tp_flags = Py_TPFLAGS_DEFAULT,
99 .tp_new = PyType_GenericNew,
100 };
101
102 .. note::
103 We recommend using C99-style designated initializers as above, to
104 avoid listing all the :c:type:`PyTypeObject` fields that you don't care
105 about and also to avoid caring about the fields' declaration order.
106
107 The actual definition of :c:type:`PyTypeObject` in :file:`object.h` has
108 many more :ref:`fields <type-structs>` than the definition above. The
109 remaining fields will be filled with zeros by the C compiler, and it's
110 common practice to not specify them explicitly unless you need them.
111
112 We're going to pick it apart, one field at a time::
113
114 .ob_base = PyVarObject_HEAD_INIT(NULL, 0)
115
116 This line is mandatory boilerplate to initialize the ``ob_base``
117 field mentioned above. ::
118
119 .tp_name = "custom.Custom",
120
121 The name of our type. This will appear in the default textual representation of
122 our objects and in some error messages, for example:
123
124 .. code-block:: pycon
125
126 >>> "" + custom.Custom()
127 Traceback (most recent call last):
128 File "<stdin>", line 1, in <module>
129 TypeError: can only concatenate str (not "custom.Custom") to str
130
131 Note that the name is a dotted name that includes both the module name and the
132 name of the type within the module. The module in this case is :mod:`!custom` and
133 the type is :class:`!Custom`, so we set the type name to :class:`!custom.Custom`.
134 Using the real dotted import path is important to make your type compatible
135 with the :mod:`pydoc` and :mod:`pickle` modules. ::
136
137 .tp_basicsize = sizeof(CustomObject),
138 .tp_itemsize = 0,
139
140 This is so that Python knows how much memory to allocate when creating
141 new :class:`!Custom` instances. :c:member:`~PyTypeObject.tp_itemsize` is
142 only used for variable-sized objects and should otherwise be zero.
143
144 .. note::
145
146 If you want your type to be subclassable from Python, and your type has the same
147 :c:member:`~PyTypeObject.tp_basicsize` as its base type, you may have problems with multiple
148 inheritance. A Python subclass of your type will have to list your type first
149 in its :attr:`~type.__bases__`, or else it will not be able to call your type's
150 :meth:`~object.__new__` method without getting an error. You can avoid this problem by
151 ensuring that your type has a larger value for :c:member:`~PyTypeObject.tp_basicsize` than its
152 base type does. Most of the time, this will be true anyway, because either your
153 base type will be :class:`object`, or else you will be adding data members to
154 your base type, and therefore increasing its size.
155
156 We set the class flags to :c:macro:`Py_TPFLAGS_DEFAULT`. ::
157
158 .tp_flags = Py_TPFLAGS_DEFAULT,
159
160 All types should include this constant in their flags. It enables all of the
161 members defined until at least Python 3.3. If you need further members,
162 you will need to OR the corresponding flags.
163
164 We provide a doc string for the type in :c:member:`~PyTypeObject.tp_doc`. ::
165
166 .tp_doc = PyDoc_STR("Custom objects"),
167
168 To enable object creation, we have to provide a :c:member:`~PyTypeObject.tp_new`
169 handler. This is the equivalent of the Python method :meth:`~object.__new__`, but
170 has to be specified explicitly. In this case, we can just use the default
171 implementation provided by the API function :c:func:`PyType_GenericNew`. ::
172
173 .tp_new = PyType_GenericNew,
174
175 Everything else in the file should be familiar, except for some code in
176 :c:func:`!custom_module_exec`::
177
178 if (PyType_Ready(&CustomType) < 0) {
179 return -1;
180 }
181
182 This initializes the :class:`!Custom` type, filling in a number of members
183 to the appropriate default values, including :c:member:`~PyObject.ob_type` that we initially
184 set to ``NULL``. ::
185
186 if (PyModule_AddObjectRef(m, "Custom", (PyObject *) &CustomType) < 0) {
187 return -1;
188 }
189
190 This adds the type to the module dictionary. This allows us to create
191 :class:`!Custom` instances by calling the :class:`!Custom` class:
192
193 .. code-block:: pycon
194
195 >>> import custom
196 >>> mycustom = custom.Custom()
197
198 That's it! All that remains is to build it; put the above code in a file called
199 :file:`custom.c`,
200
201 .. literalinclude:: ../includes/newtypes/pyproject.toml
202
203 in a file called :file:`pyproject.toml`, and
204
205 .. code-block:: python
206
207 from setuptools import Extension, setup
208 setup(ext_modules=[Extension("custom", ["custom.c"])])
209
210 in a file called :file:`setup.py`; then typing
211
212 .. code-block:: shell-session
213
214 $ python -m pip install .
215
216 in a shell should produce a file :file:`custom.so` in a subdirectory
217 and install it; now fire up Python --- you should be able to ``import custom``
218 and play around with ``Custom`` objects.
219
220 That wasn't so hard, was it?
221
222 Of course, the current Custom type is pretty uninteresting. It has no data and
223 doesn't do anything. It can't even be subclassed.
224
225
226 Adding data and methods to the Basic example
227 ============================================
228
229 Let's extend the basic example to add some data and methods. Let's also make
230 the type usable as a base class. We'll create a new module, :mod:`!custom2` that
231 adds these capabilities:
232
233 .. literalinclude:: ../includes/newtypes/custom2.c
234
235
236 This version of the module has a number of changes.
237
238 The :class:`!Custom` type now has three data attributes in its C struct,
239 *first*, *last*, and *number*. The *first* and *last* variables are Python
240 strings containing first and last names. The *number* attribute is a C integer.
241
242 The object structure is updated accordingly::
243
244 typedef struct {
245 PyObject_HEAD
246 PyObject *first; /* first name */
247 PyObject *last; /* last name */
248 int number;
249 } CustomObject;
250
251 Because we now have data to manage, we have to be more careful about object
252 allocation and deallocation. At a minimum, we need a deallocation method::
253
254 static void
255 Custom_dealloc(CustomObject *self)
256 {
257 Py_XDECREF(self->first);
258 Py_XDECREF(self->last);
259 Py_TYPE(self)->tp_free((PyObject *) self);
260 }
261
262 which is assigned to the :c:member:`~PyTypeObject.tp_dealloc` member::
263
264 .tp_dealloc = (destructor) Custom_dealloc,
265
266 This method first clears the reference counts of the two Python attributes.
267 :c:func:`Py_XDECREF` correctly handles the case where its argument is
268 ``NULL`` (which might happen here if ``tp_new`` failed midway). It then
269 calls the :c:member:`~PyTypeObject.tp_free` member of the object's type
270 (computed by ``Py_TYPE(self)``) to free the object's memory. Note that
271 the object's type might not be :class:`!CustomType`, because the object may
272 be an instance of a subclass.
273
274 .. note::
275 The explicit cast to ``destructor`` above is needed because we defined
276 ``Custom_dealloc`` to take a ``CustomObject *`` argument, but the ``tp_dealloc``
277 function pointer expects to receive a ``PyObject *`` argument. Otherwise,
278 the compiler will emit a warning. This is object-oriented polymorphism,
279 in C!
280
281 We want to make sure that the first and last names are initialized to empty
282 strings, so we provide a ``tp_new`` implementation::
283
284 static PyObject *
285 Custom_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
286 {
287 CustomObject *self;
288 self = (CustomObject *) type->tp_alloc(type, 0);
289 if (self != NULL) {
290 self->first = PyUnicode_FromString("");
291 if (self->first == NULL) {
292 Py_DECREF(self);
293 return NULL;
294 }
295 self->last = PyUnicode_FromString("");
296 if (self->last == NULL) {
297 Py_DECREF(self);
298 return NULL;
299 }
300 self->number = 0;
301 }
302 return (PyObject *) self;
303 }
304
305 and install it in the :c:member:`~PyTypeObject.tp_new` member::
306
307 .tp_new = Custom_new,
308
309 The ``tp_new`` handler is responsible for creating (as opposed to initializing)
310 objects of the type. It is exposed in Python as the :meth:`~object.__new__` method.
311 It is not required to define a ``tp_new`` member, and indeed many extension
312 types will simply reuse :c:func:`PyType_GenericNew` as done in the first
313 version of the :class:`!Custom` type above. In this case, we use the ``tp_new``
314 handler to initialize the ``first`` and ``last`` attributes to non-``NULL``
315 default values.
316
317 ``tp_new`` is passed the type being instantiated (not necessarily ``CustomType``,
318 if a subclass is instantiated) and any arguments passed when the type was
319 called, and is expected to return the instance created. ``tp_new`` handlers
320 always accept positional and keyword arguments, but they often ignore the
321 arguments, leaving the argument handling to initializer (a.k.a. ``tp_init``
322 in C or ``__init__`` in Python) methods.
323
324 .. note::
325 ``tp_new`` shouldn't call ``tp_init`` explicitly, as the interpreter
326 will do it itself.
327
328 The ``tp_new`` implementation calls the :c:member:`~PyTypeObject.tp_alloc`
329 slot to allocate memory::
330
331 self = (CustomObject *) type->tp_alloc(type, 0);
332
333 Since memory allocation may fail, we must check the :c:member:`~PyTypeObject.tp_alloc`
334 result against ``NULL`` before proceeding.
335
336 .. note::
337 We didn't fill the :c:member:`~PyTypeObject.tp_alloc` slot ourselves. Rather
338 :c:func:`PyType_Ready` fills it for us by inheriting it from our base class,
339 which is :class:`object` by default. Most types use the default allocation
340 strategy.
341
342 .. note::
343 If you are creating a co-operative :c:member:`~PyTypeObject.tp_new` (one
344 that calls a base type's :c:member:`~PyTypeObject.tp_new` or :meth:`~object.__new__`),
345 you must *not* try to determine what method to call using method resolution
346 order at runtime. Always statically determine what type you are going to
347 call, and call its :c:member:`~PyTypeObject.tp_new` directly, or via
348 ``type->tp_base->tp_new``. If you do not do this, Python subclasses of your
349 type that also inherit from other Python-defined classes may not work correctly.
350 (Specifically, you may not be able to create instances of such subclasses
351 without getting a :exc:`TypeError`.)
352
353 We also define an initialization function which accepts arguments to provide
354 initial values for our instance::
355
356 static int
357 Custom_init(CustomObject *self, PyObject *args, PyObject *kwds)
358 {
359 static char *kwlist[] = {"first", "last", "number", NULL};
360 PyObject *first = NULL, *last = NULL, *tmp;
361
362 if (!PyArg_ParseTupleAndKeywords(args, kwds, "|OOi", kwlist,
363 &first, &last,
364 &self->number))
365 return -1;
366
367 if (first) {
368 tmp = self->first;
369 Py_INCREF(first);
370 self->first = first;
371 Py_XDECREF(tmp);
372 }
373 if (last) {
374 tmp = self->last;
375 Py_INCREF(last);
376 self->last = last;
377 Py_XDECREF(tmp);
378 }
379 return 0;
380 }
381
382 by filling the :c:member:`~PyTypeObject.tp_init` slot. ::
383
384 .tp_init = (initproc) Custom_init,
385
386 The :c:member:`~PyTypeObject.tp_init` slot is exposed in Python as the
387 :meth:`~object.__init__` method. It is used to initialize an object after it's
388 created. Initializers always accept positional and keyword arguments,
389 and they should return either ``0`` on success or ``-1`` on error.
390
391 Unlike the ``tp_new`` handler, there is no guarantee that ``tp_init``
392 is called at all (for example, the :mod:`pickle` module by default
393 doesn't call :meth:`~object.__init__` on unpickled instances). It can also be
394 called multiple times. Anyone can call the :meth:`!__init__` method on
395 our objects. For this reason, we have to be extra careful when assigning
396 the new attribute values. We might be tempted, for example to assign the
397 ``first`` member like this::
398
399 if (first) {
400 Py_XDECREF(self->first);
401 Py_INCREF(first);
402 self->first = first;
403 }
404
405 But this would be risky. Our type doesn't restrict the type of the
406 ``first`` member, so it could be any kind of object. It could have a
407 destructor that causes code to be executed that tries to access the
408 ``first`` member; or that destructor could release the
409 :term:`Global interpreter Lock <GIL>` and let arbitrary code run in other
410 threads that accesses and modifies our object.
411
412 To be paranoid and protect ourselves against this possibility, we almost
413 always reassign members before decrementing their reference counts. When
414 don't we have to do this?
415
416 * when we absolutely know that the reference count is greater than 1;
417
418 * when we know that deallocation of the object [#]_ will neither release
419 the :term:`GIL` nor cause any calls back into our type's code;
420
421 * when decrementing a reference count in a :c:member:`~PyTypeObject.tp_dealloc`
422 handler on a type which doesn't support cyclic garbage collection [#]_.
423
424 We want to expose our instance variables as attributes. There are a
425 number of ways to do that. The simplest way is to define member definitions::
426
427 static PyMemberDef Custom_members[] = {
428 {"first", Py_T_OBJECT_EX, offsetof(CustomObject, first), 0,
429 "first name"},
430 {"last", Py_T_OBJECT_EX, offsetof(CustomObject, last), 0,
431 "last name"},
432 {"number", Py_T_INT, offsetof(CustomObject, number), 0,
433 "custom number"},
434 {NULL} /* Sentinel */
435 };
436
437 and put the definitions in the :c:member:`~PyTypeObject.tp_members` slot::
438
439 .tp_members = Custom_members,
440
441 Each member definition has a member name, type, offset, access flags and
442 documentation string. See the :ref:`Generic-Attribute-Management` section
443 below for details.
444
445 A disadvantage of this approach is that it doesn't provide a way to restrict the
446 types of objects that can be assigned to the Python attributes. We expect the
447 first and last names to be strings, but any Python objects can be assigned.
448 Further, the attributes can be deleted, setting the C pointers to ``NULL``. Even
449 though we can make sure the members are initialized to non-``NULL`` values, the
450 members can be set to ``NULL`` if the attributes are deleted.
451
452 We define a single method, :meth:`!Custom.name`, that outputs the objects name as the
453 concatenation of the first and last names. ::
454
455 static PyObject *
456 Custom_name(CustomObject *self, PyObject *Py_UNUSED(ignored))
457 {
458 if (self->first == NULL) {
459 PyErr_SetString(PyExc_AttributeError, "first");
460 return NULL;
461 }
462 if (self->last == NULL) {
463 PyErr_SetString(PyExc_AttributeError, "last");
464 return NULL;
465 }
466 return PyUnicode_FromFormat("%S %S", self->first, self->last);
467 }
468
469 The method is implemented as a C function that takes a :class:`!Custom` (or
470 :class:`!Custom` subclass) instance as the first argument. Methods always take an
471 instance as the first argument. Methods often take positional and keyword
472 arguments as well, but in this case we don't take any and don't need to accept
473 a positional argument tuple or keyword argument dictionary. This method is
474 equivalent to the Python method:
475
476 .. code-block:: python
477
478 def name(self):
479 return "%s %s" % (self.first, self.last)
480
481 Note that we have to check for the possibility that our :attr:`!first` and
482 :attr:`!last` members are ``NULL``. This is because they can be deleted, in which
483 case they are set to ``NULL``. It would be better to prevent deletion of these
484 attributes and to restrict the attribute values to be strings. We'll see how to
485 do that in the next section.
486
487 Now that we've defined the method, we need to create an array of method
488 definitions::
489
490 static PyMethodDef Custom_methods[] = {
491 {"name", (PyCFunction) Custom_name, METH_NOARGS,
492 "Return the name, combining the first and last name"
493 },
494 {NULL} /* Sentinel */
495 };
496
497 (note that we used the :c:macro:`METH_NOARGS` flag to indicate that the method
498 is expecting no arguments other than *self*)
499
500 and assign it to the :c:member:`~PyTypeObject.tp_methods` slot::
501
502 .tp_methods = Custom_methods,
503
504 Finally, we'll make our type usable as a base class for subclassing. We've
505 written our methods carefully so far so that they don't make any assumptions
506 about the type of the object being created or used, so all we need to do is
507 to add the :c:macro:`Py_TPFLAGS_BASETYPE` to our class flag definition::
508
509 .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,
510
511 We rename :c:func:`!PyInit_custom` to :c:func:`!PyInit_custom2`, update the
512 module name in the :c:type:`PyModuleDef` struct, and update the full class
513 name in the :c:type:`PyTypeObject` struct.
514
515 Finally, we update our :file:`setup.py` file to include the new module,
516
517 .. code-block:: python
518
519 from setuptools import Extension, setup
520 setup(ext_modules=[
521 Extension("custom", ["custom.c"]),
522 Extension("custom2", ["custom2.c"]),
523 ])
524
525 and then we re-install so that we can ``import custom2``:
526
527 .. code-block:: shell-session
528
529 $ python -m pip install .
530
531 Providing finer control over data attributes
532 ============================================
533
534 In this section, we'll provide finer control over how the :attr:`!first` and
535 :attr:`!last` attributes are set in the :class:`!Custom` example. In the previous
536 version of our module, the instance variables :attr:`!first` and :attr:`!last`
537 could be set to non-string values or even deleted. We want to make sure that
538 these attributes always contain strings.
539
540 .. literalinclude:: ../includes/newtypes/custom3.c
541
542
543 To provide greater control, over the :attr:`!first` and :attr:`!last` attributes,
544 we'll use custom getter and setter functions. Here are the functions for
545 getting and setting the :attr:`!first` attribute::
546
547 static PyObject *
548 Custom_getfirst(CustomObject *self, void *closure)
549 {
550 Py_INCREF(self->first);
551 return self->first;
552 }
553
554 static int
555 Custom_setfirst(CustomObject *self, PyObject *value, void *closure)
556 {
557 PyObject *tmp;
558 if (value == NULL) {
559 PyErr_SetString(PyExc_TypeError, "Cannot delete the first attribute");
560 return -1;
561 }
562 if (!PyUnicode_Check(value)) {
563 PyErr_SetString(PyExc_TypeError,
564 "The first attribute value must be a string");
565 return -1;
566 }
567 tmp = self->first;
568 Py_INCREF(value);
569 self->first = value;
570 Py_DECREF(tmp);
571 return 0;
572 }
573
574 The getter function is passed a :class:`!Custom` object and a "closure", which is
575 a void pointer. In this case, the closure is ignored. (The closure supports an
576 advanced usage in which definition data is passed to the getter and setter. This
577 could, for example, be used to allow a single set of getter and setter functions
578 that decide the attribute to get or set based on data in the closure.)
579
580 The setter function is passed the :class:`!Custom` object, the new value, and the
581 closure. The new value may be ``NULL``, in which case the attribute is being
582 deleted. In our setter, we raise an error if the attribute is deleted or if its
583 new value is not a string.
584
585 We create an array of :c:type:`PyGetSetDef` structures::
586
587 static PyGetSetDef Custom_getsetters[] = {
588 {"first", (getter) Custom_getfirst, (setter) Custom_setfirst,
589 "first name", NULL},
590 {"last", (getter) Custom_getlast, (setter) Custom_setlast,
591 "last name", NULL},
592 {NULL} /* Sentinel */
593 };
594
595 and register it in the :c:member:`~PyTypeObject.tp_getset` slot::
596
597 .tp_getset = Custom_getsetters,
598
599 The last item in a :c:type:`PyGetSetDef` structure is the "closure" mentioned
600 above. In this case, we aren't using a closure, so we just pass ``NULL``.
601
602 We also remove the member definitions for these attributes::
603
604 static PyMemberDef Custom_members[] = {
605 {"number", Py_T_INT, offsetof(CustomObject, number), 0,
606 "custom number"},
607 {NULL} /* Sentinel */
608 };
609
610 We also need to update the :c:member:`~PyTypeObject.tp_init` handler to only
611 allow strings [#]_ to be passed::
612
613 static int
614 Custom_init(CustomObject *self, PyObject *args, PyObject *kwds)
615 {
616 static char *kwlist[] = {"first", "last", "number", NULL};
617 PyObject *first = NULL, *last = NULL, *tmp;
618
619 if (!PyArg_ParseTupleAndKeywords(args, kwds, "|UUi", kwlist,
620 &first, &last,
621 &self->number))
622 return -1;
623
624 if (first) {
625 tmp = self->first;
626 Py_INCREF(first);
627 self->first = first;
628 Py_DECREF(tmp);
629 }
630 if (last) {
631 tmp = self->last;
632 Py_INCREF(last);
633 self->last = last;
634 Py_DECREF(tmp);
635 }
636 return 0;
637 }
638
639 With these changes, we can assure that the ``first`` and ``last`` members are
640 never ``NULL`` so we can remove checks for ``NULL`` values in almost all cases.
641 This means that most of the :c:func:`Py_XDECREF` calls can be converted to
642 :c:func:`Py_DECREF` calls. The only place we can't change these calls is in
643 the ``tp_dealloc`` implementation, where there is the possibility that the
644 initialization of these members failed in ``tp_new``.
645
646 We also rename the module initialization function and module name in the
647 initialization function, as we did before, and we add an extra definition to the
648 :file:`setup.py` file.
649
650
651 Supporting cyclic garbage collection
652 ====================================
653
654 Python has a :term:`cyclic garbage collector (GC) <garbage collection>` that
655 can identify unneeded objects even when their reference counts are not zero.
656 This can happen when objects are involved in cycles. For example, consider:
657
658 .. code-block:: pycon
659
660 >>> l = []
661 >>> l.append(l)
662 >>> del l
663
664 In this example, we create a list that contains itself. When we delete it, it
665 still has a reference from itself. Its reference count doesn't drop to zero.
666 Fortunately, Python's cyclic garbage collector will eventually figure out that
667 the list is garbage and free it.
668
669 In the second version of the :class:`!Custom` example, we allowed any kind of
670 object to be stored in the :attr:`!first` or :attr:`!last` attributes [#]_.
671 Besides, in the second and third versions, we allowed subclassing
672 :class:`!Custom`, and subclasses may add arbitrary attributes. For any of
673 those two reasons, :class:`!Custom` objects can participate in cycles:
674
675 .. code-block:: pycon
676
677 >>> import custom3
678 >>> class Derived(custom3.Custom): pass
679 ...
680 >>> n = Derived()
681 >>> n.some_attribute = n
682
683 To allow a :class:`!Custom` instance participating in a reference cycle to
684 be properly detected and collected by the cyclic GC, our :class:`!Custom` type
685 needs to fill two additional slots and to enable a flag that enables these slots:
686
687 .. literalinclude:: ../includes/newtypes/custom4.c
688
689
690 First, the traversal method lets the cyclic GC know about subobjects that could
691 participate in cycles::
692
693 static int
694 Custom_traverse(CustomObject *self, visitproc visit, void *arg)
695 {
696 int vret;
697 if (self->first) {
698 vret = visit(self->first, arg);
699 if (vret != 0)
700 return vret;
701 }
702 if (self->last) {
703 vret = visit(self->last, arg);
704 if (vret != 0)
705 return vret;
706 }
707 return 0;
708 }
709
710 For each subobject that can participate in cycles, we need to call the
711 :c:func:`!visit` function, which is passed to the traversal method. The
712 :c:func:`!visit` function takes as arguments the subobject and the extra argument
713 *arg* passed to the traversal method. It returns an integer value that must be
714 returned if it is non-zero.
715
716 Python provides a :c:func:`Py_VISIT` macro that automates calling visit
717 functions. With :c:func:`Py_VISIT`, we can minimize the amount of boilerplate
718 in ``Custom_traverse``::
719
720 static int
721 Custom_traverse(CustomObject *self, visitproc visit, void *arg)
722 {
723 Py_VISIT(self->first);
724 Py_VISIT(self->last);
725 return 0;
726 }
727
728 .. note::
729 The :c:member:`~PyTypeObject.tp_traverse` implementation must name its
730 arguments exactly *visit* and *arg* in order to use :c:func:`Py_VISIT`.
731
732 Second, we need to provide a method for clearing any subobjects that can
733 participate in cycles::
734
735 static int
736 Custom_clear(CustomObject *self)
737 {
738 Py_CLEAR(self->first);
739 Py_CLEAR(self->last);
740 return 0;
741 }
742
743 Notice the use of the :c:func:`Py_CLEAR` macro. It is the recommended and safe
744 way to clear data attributes of arbitrary types while decrementing
745 their reference counts. If you were to call :c:func:`Py_XDECREF` instead
746 on the attribute before setting it to ``NULL``, there is a possibility
747 that the attribute's destructor would call back into code that reads the
748 attribute again (*especially* if there is a reference cycle).
749
750 .. note::
751 You could emulate :c:func:`Py_CLEAR` by writing::
752
753 PyObject *tmp;
754 tmp = self->first;
755 self->first = NULL;
756 Py_XDECREF(tmp);
757
758 Nevertheless, it is much easier and less error-prone to always
759 use :c:func:`Py_CLEAR` when deleting an attribute. Don't
760 try to micro-optimize at the expense of robustness!
761
762 The deallocator ``Custom_dealloc`` may call arbitrary code when clearing
763 attributes. It means the circular GC can be triggered inside the function.
764 Since the GC assumes reference count is not zero, we need to untrack the object
765 from the GC by calling :c:func:`PyObject_GC_UnTrack` before clearing members.
766 Here is our reimplemented deallocator using :c:func:`PyObject_GC_UnTrack`
767 and ``Custom_clear``::
768
769 static void
770 Custom_dealloc(CustomObject *self)
771 {
772 PyObject_GC_UnTrack(self);
773 Custom_clear(self);
774 Py_TYPE(self)->tp_free((PyObject *) self);
775 }
776
777 Finally, we add the :c:macro:`Py_TPFLAGS_HAVE_GC` flag to the class flags::
778
779 .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC,
780
781 That's pretty much it. If we had written custom :c:member:`~PyTypeObject.tp_alloc` or
782 :c:member:`~PyTypeObject.tp_free` handlers, we'd need to modify them for cyclic
783 garbage collection. Most extensions will use the versions automatically provided.
784
785
786 Subclassing other types
787 =======================
788
789 It is possible to create new extension types that are derived from existing
790 types. It is easiest to inherit from the built in types, since an extension can
791 easily use the :c:type:`PyTypeObject` it needs. It can be difficult to share
792 these :c:type:`PyTypeObject` structures between extension modules.
793
794 In this example we will create a :class:`!SubList` type that inherits from the
795 built-in :class:`list` type. The new type will be completely compatible with
796 regular lists, but will have an additional :meth:`!increment` method that
797 increases an internal counter:
798
799 .. code-block:: pycon
800
801 >>> import sublist
802 >>> s = sublist.SubList(range(3))
803 >>> s.extend(s)
804 >>> print(len(s))
805 6
806 >>> print(s.increment())
807 1
808 >>> print(s.increment())
809 2
810
811 .. literalinclude:: ../includes/newtypes/sublist.c
812
813
814 As you can see, the source code closely resembles the :class:`!Custom` examples in
815 previous sections. We will break down the main differences between them. ::
816
817 typedef struct {
818 PyListObject list;
819 int state;
820 } SubListObject;
821
822 The primary difference for derived type objects is that the base type's
823 object structure must be the first value. The base type will already include
824 the :c:func:`PyObject_HEAD` at the beginning of its structure.
825
826 When a Python object is a :class:`!SubList` instance, its ``PyObject *`` pointer
827 can be safely cast to both ``PyListObject *`` and ``SubListObject *``::
828
829 static int
830 SubList_init(SubListObject *self, PyObject *args, PyObject *kwds)
831 {
832 if (PyList_Type.tp_init((PyObject *) self, args, kwds) < 0)
833 return -1;
834 self->state = 0;
835 return 0;
836 }
837
838 We see above how to call through to the :meth:`~object.__init__` method of the base
839 type.
840
841 This pattern is important when writing a type with custom
842 :c:member:`~PyTypeObject.tp_new` and :c:member:`~PyTypeObject.tp_dealloc`
843 members. The :c:member:`~PyTypeObject.tp_new` handler should not actually
844 create the memory for the object with its :c:member:`~PyTypeObject.tp_alloc`,
845 but let the base class handle it by calling its own :c:member:`~PyTypeObject.tp_new`.
846
847 The :c:type:`PyTypeObject` struct supports a :c:member:`~PyTypeObject.tp_base`
848 specifying the type's concrete base class. Due to cross-platform compiler
849 issues, you can't fill that field directly with a reference to
850 :c:type:`PyList_Type`; it should be done in the :c:data:`Py_mod_exec`
851 function::
852
853 static int
854 sublist_module_exec(PyObject *m)
855 {
856 SubListType.tp_base = &PyList_Type;
857 if (PyType_Ready(&SubListType) < 0) {
858 return -1;
859 }
860
861 if (PyModule_AddObjectRef(m, "SubList", (PyObject *) &SubListType) < 0) {
862 return -1;
863 }
864
865 return 0;
866 }
867
868 Before calling :c:func:`PyType_Ready`, the type structure must have the
869 :c:member:`~PyTypeObject.tp_base` slot filled in. When we are deriving an
870 existing type, it is not necessary to fill out the :c:member:`~PyTypeObject.tp_alloc`
871 slot with :c:func:`PyType_GenericNew` -- the allocation function from the base
872 type will be inherited.
873
874 After that, calling :c:func:`PyType_Ready` and adding the type object to the
875 module is the same as with the basic :class:`!Custom` examples.
876
877
878 .. rubric:: Footnotes
879
880 .. [#] This is true when we know that the object is a basic type, like a string or a
881 float.
882
883 .. [#] We relied on this in the :c:member:`~PyTypeObject.tp_dealloc` handler
884 in this example, because our type doesn't support garbage collection.
885
886 .. [#] We now know that the first and last members are strings, so perhaps we
887 could be less careful about decrementing their reference counts, however,
888 we accept instances of string subclasses. Even though deallocating normal
889 strings won't call back into our objects, we can't guarantee that deallocating
890 an instance of a string subclass won't call back into our objects.
891
892 .. [#] Also, even with our attributes restricted to strings instances, the user
893 could pass arbitrary :class:`str` subclasses and therefore still create
894 reference cycles.