Debian Codegraph @

Doc/extending/newtypes_tutorial.rst

raw
1 .. highlight:: c
2
3 .. _defining-new-types:
4
5 **********************************
6 Defining Extension Types: Tutorial
7 **********************************
8
9 .. sectionauthor:: Michael Hudson <[email protected]>
10 .. sectionauthor:: Dave Kuhlman <[email protected]>
11 .. sectionauthor:: Jim Fulton <[email protected]>
12
13
14 Python allows the writer of a C extension module to define new types that
15 can be manipulated from Python code, much like the built-in :class:`str`
16 and :class:`list` types. The code for all extension types follows a
17 pattern, but there are some details that you need to understand before you
18 can get started. This document is a gentle introduction to the topic.
19
20
21 .. _dnt-basics:
22
23 The Basics
24 ==========
25
26 The :term:`CPython` runtime sees all Python objects as variables of type
27 :c:expr:`PyObject*`, which serves as a "base type" for all Python objects.
28 The :c:type:`PyObject` structure itself only contains the object's
29 :term:`reference count` and a pointer to the object's "type object".
30 This is where the action is; the type object determines which (C) functions
31 get called by the interpreter when, for instance, an attribute gets looked up
32 on an object, a method called, or it is multiplied by another object. These
33 C functions are called "type methods".
34
35 So, if you want to define a new extension type, you need to create a new type
36 object.
37
38 This sort of thing can only be explained by example, so here's a minimal, but
39 complete, module that defines a new type named :class:`!Custom` inside a C
40 extension module :mod:`!custom`:
41
42 .. note::
43 What we're showing here is the traditional way of defining *static*
44 extension types. It should be adequate for most uses. The C API also
45 allows defining heap-allocated extension types using the
46 :c:func:`PyType_FromSpec` function, which isn't covered in this tutorial.
47
48 .. literalinclude:: ../includes/newtypes/custom.c
49
50 Now that's quite a bit to take in at once, but hopefully bits will seem familiar
51 from the previous chapter. This file defines three things:
52
53 #. What a :class:`!Custom` **object** contains: this is the ``CustomObject``
54 struct, which is allocated once for each :class:`!Custom` instance.
55 #. How the :class:`!Custom` **type** behaves: this is the ``CustomType`` struct,
56 which defines a set of flags and function pointers that the interpreter
57 inspects when specific operations are requested.
58 #. How to define and execute the :mod:`!custom` module: this is the
59 ``PyInit_custom`` function and the associated ``custom_module`` struct for
60 defining the module, and the ``custom_module_exec`` function to set up
61 a fresh module object.
62
63 The first bit is::
64
65 typedef struct {
66 PyObject_HEAD
67 } CustomObject;
68
69 This is what a Custom object will contain. ``PyObject_HEAD`` is mandatory
70 at the start of each object struct and defines a field called ``ob_base``
71 of type :c:type:`PyObject`, containing a pointer to a type object and a
72 reference count (these can be accessed using the macros :c:macro:`Py_TYPE`
73 and :c:macro:`Py_REFCNT` respectively). The reason for the macro is to
74 abstract away the layout and to enable additional fields in :ref:`debug builds
75 <debug-build>`.
76
77 .. note::
78 There is no semicolon above after the :c:macro:`PyObject_HEAD` macro.
79 Be wary of adding one by accident: some compilers will complain.
80
81 Of course, objects generally store additional data besides the standard
82 ``PyObject_HEAD`` boilerplate; for example, here is the definition for
83 standard Python floats::
84
85 typedef struct {
86 PyObject_HEAD
87 double ob_fval;
88 } PyFloatObject;
89
90 The second bit is the definition of the type object. ::
91
92 static PyTypeObject CustomType = {
93 .ob_base = PyVarObject_HEAD_INIT(NULL, 0)
94 .tp_name = "custom.Custom",
95 .tp_doc = PyDoc_STR("Custom objects"),
96 .tp_basicsize = sizeof(CustomObject),
97 .tp_itemsize = 0,
98 .tp_flags = Py_TPFLAGS_DEFAULT,
99 .tp_new = PyType_GenericNew,
100 };
101
102 .. note::
103 We recommend using C99-style designated initializers as above, to
104 avoid listing all the :c:type:`PyTypeObject` fields that you don't care
105 about and also to avoid caring about the fields' declaration order.
106
107 The actual definition of :c:type:`PyTypeObject` in :file:`object.h` has
108 many more :ref:`fields <type-structs>` than the definition above. The
109 remaining fields will be filled with zeros by the C compiler, and it's
110 common practice to not specify them explicitly unless you need them.
111
112 We're going to pick it apart, one field at a time::
113
114 .ob_base = PyVarObject_HEAD_INIT(NULL, 0)
115
116 This line is mandatory boilerplate to initialize the ``ob_base``
117 field mentioned above. ::
118
119 .tp_name = "custom.Custom",
120
121 The name of our type. This will appear in the default textual representation of
122 our objects and in some error messages, for example:
123
124 .. code-block:: pycon
125
126 >>> "" + custom.Custom()
127 Traceback (most recent call last):
128 File "<stdin>", line 1, in <module>
129 TypeError: can only concatenate str (not "custom.Custom") to str
130
131 Note that the name is a dotted name that includes both the module name and the
132 name of the type within the module. The module in this case is :mod:`!custom` and
133 the type is :class:`!Custom`, so we set the type name to :class:`!custom.Custom`.
134 Using the real dotted import path is important to make your type compatible
135 with the :mod:`pydoc` and :mod:`pickle` modules. ::
136
137 .tp_basicsize = sizeof(CustomObject),
138 .tp_itemsize = 0,
139
140 This is so that Python knows how much memory to allocate when creating
141 new :class:`!Custom` instances. :c:member:`~PyTypeObject.tp_itemsize` is
142 only used for variable-sized objects and should otherwise be zero.
143
144 .. note::
145
146 If you want your type to be subclassable from Python, and your type has the same
147 :c:member:`~PyTypeObject.tp_basicsize` as its base type, you may have problems with multiple
148 inheritance. A Python subclass of your type will have to list your type first
149 in its :attr:`~type.__bases__`, or else it will not be able to call your type's
150 :meth:`~object.__new__` method without getting an error. You can avoid this problem by
151 ensuring that your type has a larger value for :c:member:`~PyTypeObject.tp_basicsize` than its
152 base type does. Most of the time, this will be true anyway, because either your
153 base type will be :class:`object`, or else you will be adding data members to
154 your base type, and therefore increasing its size.
155
156 We set the class flags to :c:macro:`Py_TPFLAGS_DEFAULT`. ::
157
158 .tp_flags = Py_TPFLAGS_DEFAULT,
159
160 All types should include this constant in their flags. It enables all of the
161 members defined until at least Python 3.3. If you need further members,
162 you will need to OR the corresponding flags.
163
164 We provide a doc string for the type in :c:member:`~PyTypeObject.tp_doc`. ::
165
166 .tp_doc = PyDoc_STR("Custom objects"),
167
168 To enable object creation, we have to provide a :c:member:`~PyTypeObject.tp_new`
169 handler. This is the equivalent of the Python method :meth:`~object.__new__`, but
170 has to be specified explicitly. In this case, we can just use the default
171 implementation provided by the API function :c:func:`PyType_GenericNew`. ::
172
173 .tp_new = PyType_GenericNew,
174
175 Everything else in the file should be familiar, except for some code in
176 :c:func:`!custom_module_exec`::
177
178 if (PyType_Ready(&CustomType) < 0) {
179 return -1;
180 }
181
182 This initializes the :class:`!Custom` type, filling in a number of members
183 to the appropriate default values, including :c:member:`~PyObject.ob_type` that we initially
184 set to ``NULL``. ::
185
186 if (PyModule_AddObjectRef(m, "Custom", (PyObject *) &CustomType) < 0) {
187 return -1;
188 }
189
190 This adds the type to the module dictionary. This allows us to create
191 :class:`!Custom` instances by calling the :class:`!Custom` class:
192
193 .. code-block:: pycon
194
195 >>> import custom
196 >>> mycustom = custom.Custom()
197
198 That's it! All that remains is to build it; put the above code in a file called
199 :file:`custom.c`,
200
201 .. literalinclude:: ../includes/newtypes/pyproject.toml
202
203 in a file called :file:`pyproject.toml`, and
204
205 .. code-block:: python
206
207 from setuptools import Extension, setup
208 setup(ext_modules=[Extension("custom", ["custom.c"])])
209
210 in a file called :file:`setup.py`; then typing
211
212 .. code-block:: shell-session
213
214 $ python -m pip install .
215
216 in a shell should produce a file :file:`custom.so` in a subdirectory
217 and install it; now fire up Python --- you should be able to ``import custom``
218 and play around with ``Custom`` objects.
219
220 That wasn't so hard, was it?
221
222 Of course, the current Custom type is pretty uninteresting. It has no data and
223 doesn't do anything. It can't even be subclassed.
224
225
226 Adding data and methods to the Basic example
227 ============================================
228
229 Let's extend the basic example to add some data and methods. Let's also make
230 the type usable as a base class. We'll create a new module, :mod:`!custom2` that
231 adds these capabilities:
232
233 .. literalinclude:: ../includes/newtypes/custom2.c
234
235
236 This version of the module has a number of changes.
237
238 The :class:`!Custom` type now has three data attributes in its C struct,
239 *first*, *last*, and *number*. The *first* and *last* variables are Python
240 strings containing first and last names. The *number* attribute is a C integer.
241
242 The object structure is updated accordingly::
243
244 typedef struct {
245 PyObject_HEAD
246 PyObject *first; /* first name */
247 PyObject *last; /* last name */
248 int number;
249 } CustomObject;
250
251 Because we now have data to manage, we have to be more careful about object
252 allocation and deallocation. At a minimum, we need a deallocation method::
253
254 static void
255 Custom_dealloc(PyObject *op)
256 {
257 CustomObject *self = (CustomObject *) op;
258 Py_XDECREF(self->first);
259 Py_XDECREF(self->last);
260 Py_TYPE(self)->tp_free(self);
261 }
262
263 which is assigned to the :c:member:`~PyTypeObject.tp_dealloc` member::
264
265 .tp_dealloc = Custom_dealloc,
266
267 This method first clears the reference counts of the two Python attributes.
268 :c:func:`Py_XDECREF` correctly handles the case where its argument is
269 ``NULL`` (which might happen here if ``tp_new`` failed midway). It then
270 calls the :c:member:`~PyTypeObject.tp_free` member of the object's type
271 (computed by ``Py_TYPE(self)``) to free the object's memory. Note that
272 the object's type might not be :class:`!CustomType`, because the object may
273 be an instance of a subclass.
274
275 .. note::
276
277 The explicit cast to ``CustomObject *`` above is needed because we defined
278 ``Custom_dealloc`` to take a ``PyObject *`` argument, as the ``tp_dealloc``
279 function pointer expects to receive a ``PyObject *`` argument.
280 By assigning to the ``tp_dealloc`` slot of a type, we declare
281 that it can only be called with instances of our ``CustomObject``
282 class, so the cast to ``(CustomObject *)`` is safe.
283 This is object-oriented polymorphism, in C!
284
285 In existing code, or in previous versions of this tutorial,
286 you might see similar functions take a pointer to the subtype
287 object structure (``CustomObject*``) directly, like this::
288
289 Custom_dealloc(CustomObject *self)
290 {
291 Py_XDECREF(self->first);
292 Py_XDECREF(self->last);
293 Py_TYPE(self)->tp_free((PyObject *) self);
294 }
295 ...
296 .tp_dealloc = (destructor) Custom_dealloc,
297
298 This does the same thing on all architectures that CPython
299 supports, but according to the C standard, it invokes
300 undefined behavior.
301
302 We want to make sure that the first and last names are initialized to empty
303 strings, so we provide a ``tp_new`` implementation::
304
305 static PyObject *
306 Custom_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
307 {
308 CustomObject *self;
309 self = (CustomObject *) type->tp_alloc(type, 0);
310 if (self != NULL) {
311 self->first = PyUnicode_FromString("");
312 if (self->first == NULL) {
313 Py_DECREF(self);
314 return NULL;
315 }
316 self->last = PyUnicode_FromString("");
317 if (self->last == NULL) {
318 Py_DECREF(self);
319 return NULL;
320 }
321 self->number = 0;
322 }
323 return (PyObject *) self;
324 }
325
326 and install it in the :c:member:`~PyTypeObject.tp_new` member::
327
328 .tp_new = Custom_new,
329
330 The ``tp_new`` handler is responsible for creating (as opposed to initializing)
331 objects of the type. It is exposed in Python as the :meth:`~object.__new__` method.
332 It is not required to define a ``tp_new`` member, and indeed many extension
333 types will simply reuse :c:func:`PyType_GenericNew` as done in the first
334 version of the :class:`!Custom` type above. In this case, we use the ``tp_new``
335 handler to initialize the ``first`` and ``last`` attributes to non-``NULL``
336 default values.
337
338 ``tp_new`` is passed the type being instantiated (not necessarily ``CustomType``,
339 if a subclass is instantiated) and any arguments passed when the type was
340 called, and is expected to return the instance created. ``tp_new`` handlers
341 always accept positional and keyword arguments, but they often ignore the
342 arguments, leaving the argument handling to initializer (a.k.a. ``tp_init``
343 in C or ``__init__`` in Python) methods.
344
345 .. note::
346 ``tp_new`` shouldn't call ``tp_init`` explicitly, as the interpreter
347 will do it itself.
348
349 The ``tp_new`` implementation calls the :c:member:`~PyTypeObject.tp_alloc`
350 slot to allocate memory::
351
352 self = (CustomObject *) type->tp_alloc(type, 0);
353
354 Since memory allocation may fail, we must check the :c:member:`~PyTypeObject.tp_alloc`
355 result against ``NULL`` before proceeding.
356
357 .. note::
358 We didn't fill the :c:member:`~PyTypeObject.tp_alloc` slot ourselves. Rather
359 :c:func:`PyType_Ready` fills it for us by inheriting it from our base class,
360 which is :class:`object` by default. Most types use the default allocation
361 strategy.
362
363 .. note::
364 If you are creating a co-operative :c:member:`~PyTypeObject.tp_new` (one
365 that calls a base type's :c:member:`~PyTypeObject.tp_new` or :meth:`~object.__new__`),
366 you must *not* try to determine what method to call using method resolution
367 order at runtime. Always statically determine what type you are going to
368 call, and call its :c:member:`~PyTypeObject.tp_new` directly, or via
369 ``type->tp_base->tp_new``. If you do not do this, Python subclasses of your
370 type that also inherit from other Python-defined classes may not work correctly.
371 (Specifically, you may not be able to create instances of such subclasses
372 without getting a :exc:`TypeError`.)
373
374 We also define an initialization function which accepts arguments to provide
375 initial values for our instance::
376
377 static int
378 Custom_init(PyObject *op, PyObject *args, PyObject *kwds)
379 {
380 CustomObject *self = (CustomObject *) op;
381 static char *kwlist[] = {"first", "last", "number", NULL};
382 PyObject *first = NULL, *last = NULL, *tmp;
383
384 if (!PyArg_ParseTupleAndKeywords(args, kwds, "|OOi", kwlist,
385 &first, &last,
386 &self->number))
387 return -1;
388
389 if (first) {
390 tmp = self->first;
391 Py_INCREF(first);
392 self->first = first;
393 Py_XDECREF(tmp);
394 }
395 if (last) {
396 tmp = self->last;
397 Py_INCREF(last);
398 self->last = last;
399 Py_XDECREF(tmp);
400 }
401 return 0;
402 }
403
404 by filling the :c:member:`~PyTypeObject.tp_init` slot. ::
405
406 .tp_init = Custom_init,
407
408 The :c:member:`~PyTypeObject.tp_init` slot is exposed in Python as the
409 :meth:`~object.__init__` method. It is used to initialize an object after it's
410 created. Initializers always accept positional and keyword arguments,
411 and they should return either ``0`` on success or ``-1`` on error.
412
413 Unlike the ``tp_new`` handler, there is no guarantee that ``tp_init``
414 is called at all (for example, the :mod:`pickle` module by default
415 doesn't call :meth:`~object.__init__` on unpickled instances). It can also be
416 called multiple times. Anyone can call the :meth:`!__init__` method on
417 our objects. For this reason, we have to be extra careful when assigning
418 the new attribute values. We might be tempted, for example to assign the
419 ``first`` member like this::
420
421 if (first) {
422 Py_XDECREF(self->first);
423 Py_INCREF(first);
424 self->first = first;
425 }
426
427 But this would be risky. Our type doesn't restrict the type of the
428 ``first`` member, so it could be any kind of object. It could have a
429 destructor that causes code to be executed that tries to access the
430 ``first`` member; or that destructor could detach the
431 :term:`thread state <attached thread state>` and let arbitrary code run in other
432 threads that accesses and modifies our object.
433
434 To be paranoid and protect ourselves against this possibility, we almost
435 always reassign members before decrementing their reference counts. When
436 don't we have to do this?
437
438 * when we absolutely know that the reference count is greater than 1;
439
440 * when we know that deallocation of the object [#]_ will neither detach
441 the :term:`thread state <attached thread state>` nor cause any calls back into our type's code;
442
443 * when decrementing a reference count in a :c:member:`~PyTypeObject.tp_dealloc`
444 handler on a type which doesn't support cyclic garbage collection [#]_.
445
446 We want to expose our instance variables as attributes. There are a
447 number of ways to do that. The simplest way is to define member definitions::
448
449 static PyMemberDef Custom_members[] = {
450 {"first", Py_T_OBJECT_EX, offsetof(CustomObject, first), 0,
451 "first name"},
452 {"last", Py_T_OBJECT_EX, offsetof(CustomObject, last), 0,
453 "last name"},
454 {"number", Py_T_INT, offsetof(CustomObject, number), 0,
455 "custom number"},
456 {NULL} /* Sentinel */
457 };
458
459 and put the definitions in the :c:member:`~PyTypeObject.tp_members` slot::
460
461 .tp_members = Custom_members,
462
463 Each member definition has a member name, type, offset, access flags and
464 documentation string. See the :ref:`Generic-Attribute-Management` section
465 below for details.
466
467 A disadvantage of this approach is that it doesn't provide a way to restrict the
468 types of objects that can be assigned to the Python attributes. We expect the
469 first and last names to be strings, but any Python objects can be assigned.
470 Further, the attributes can be deleted, setting the C pointers to ``NULL``. Even
471 though we can make sure the members are initialized to non-``NULL`` values, the
472 members can be set to ``NULL`` if the attributes are deleted.
473
474 We define a single method, :meth:`!Custom.name`, that outputs the objects name as the
475 concatenation of the first and last names. ::
476
477 static PyObject *
478 Custom_name(PyObject *op, PyObject *Py_UNUSED(dummy))
479 {
480 CustomObject *self = (CustomObject *) op;
481 if (self->first == NULL) {
482 PyErr_SetString(PyExc_AttributeError, "first");
483 return NULL;
484 }
485 if (self->last == NULL) {
486 PyErr_SetString(PyExc_AttributeError, "last");
487 return NULL;
488 }
489 return PyUnicode_FromFormat("%S %S", self->first, self->last);
490 }
491
492 The method is implemented as a C function that takes a :class:`!Custom` (or
493 :class:`!Custom` subclass) instance as the first argument. Methods always take an
494 instance as the first argument. Methods often take positional and keyword
495 arguments as well, but in this case we don't take any and don't need to accept
496 a positional argument tuple or keyword argument dictionary. This method is
497 equivalent to the Python method:
498
499 .. code-block:: python
500
501 def name(self):
502 return "%s %s" % (self.first, self.last)
503
504 Note that we have to check for the possibility that our :attr:`!first` and
505 :attr:`!last` members are ``NULL``. This is because they can be deleted, in which
506 case they are set to ``NULL``. It would be better to prevent deletion of these
507 attributes and to restrict the attribute values to be strings. We'll see how to
508 do that in the next section.
509
510 Now that we've defined the method, we need to create an array of method
511 definitions::
512
513 static PyMethodDef Custom_methods[] = {
514 {"name", Custom_name, METH_NOARGS,
515 "Return the name, combining the first and last name"
516 },
517 {NULL} /* Sentinel */
518 };
519
520 (note that we used the :c:macro:`METH_NOARGS` flag to indicate that the method
521 is expecting no arguments other than *self*)
522
523 and assign it to the :c:member:`~PyTypeObject.tp_methods` slot::
524
525 .tp_methods = Custom_methods,
526
527 Finally, we'll make our type usable as a base class for subclassing. We've
528 written our methods carefully so far so that they don't make any assumptions
529 about the type of the object being created or used, so all we need to do is
530 to add the :c:macro:`Py_TPFLAGS_BASETYPE` to our class flag definition::
531
532 .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,
533
534 We rename :c:func:`!PyInit_custom` to :c:func:`!PyInit_custom2`, update the
535 module name in the :c:type:`PyModuleDef` struct, and update the full class
536 name in the :c:type:`PyTypeObject` struct.
537
538 Finally, we update our :file:`setup.py` file to include the new module,
539
540 .. code-block:: python
541
542 from setuptools import Extension, setup
543 setup(ext_modules=[
544 Extension("custom", ["custom.c"]),
545 Extension("custom2", ["custom2.c"]),
546 ])
547
548 and then we re-install so that we can ``import custom2``:
549
550 .. code-block:: shell-session
551
552 $ python -m pip install .
553
554 Providing finer control over data attributes
555 ============================================
556
557 In this section, we'll provide finer control over how the :attr:`!first` and
558 :attr:`!last` attributes are set in the :class:`!Custom` example. In the previous
559 version of our module, the instance variables :attr:`!first` and :attr:`!last`
560 could be set to non-string values or even deleted. We want to make sure that
561 these attributes always contain strings.
562
563 .. literalinclude:: ../includes/newtypes/custom3.c
564
565
566 To provide greater control, over the :attr:`!first` and :attr:`!last` attributes,
567 we'll use custom getter and setter functions. Here are the functions for
568 getting and setting the :attr:`!first` attribute::
569
570 static PyObject *
571 Custom_getfirst(PyObject *op, void *closure)
572 {
573 CustomObject *self = (CustomObject *) op;
574 Py_INCREF(self->first);
575 return self->first;
576 }
577
578 static int
579 Custom_setfirst(PyObject *op, PyObject *value, void *closure)
580 {
581 CustomObject *self = (CustomObject *) op;
582 PyObject *tmp;
583 if (value == NULL) {
584 PyErr_SetString(PyExc_TypeError, "Cannot delete the first attribute");
585 return -1;
586 }
587 if (!PyUnicode_Check(value)) {
588 PyErr_SetString(PyExc_TypeError,
589 "The first attribute value must be a string");
590 return -1;
591 }
592 tmp = self->first;
593 Py_INCREF(value);
594 self->first = value;
595 Py_DECREF(tmp);
596 return 0;
597 }
598
599 The getter function is passed a :class:`!Custom` object and a "closure", which is
600 a void pointer. In this case, the closure is ignored. (The closure supports an
601 advanced usage in which definition data is passed to the getter and setter. This
602 could, for example, be used to allow a single set of getter and setter functions
603 that decide the attribute to get or set based on data in the closure.)
604
605 The setter function is passed the :class:`!Custom` object, the new value, and the
606 closure. The new value may be ``NULL``, in which case the attribute is being
607 deleted. In our setter, we raise an error if the attribute is deleted or if its
608 new value is not a string.
609
610 We create an array of :c:type:`PyGetSetDef` structures::
611
612 static PyGetSetDef Custom_getsetters[] = {
613 {"first", Custom_getfirst, Custom_setfirst,
614 "first name", NULL},
615 {"last", Custom_getlast, Custom_setlast,
616 "last name", NULL},
617 {NULL} /* Sentinel */
618 };
619
620 and register it in the :c:member:`~PyTypeObject.tp_getset` slot::
621
622 .tp_getset = Custom_getsetters,
623
624 The last item in a :c:type:`PyGetSetDef` structure is the "closure" mentioned
625 above. In this case, we aren't using a closure, so we just pass ``NULL``.
626
627 We also remove the member definitions for these attributes::
628
629 static PyMemberDef Custom_members[] = {
630 {"number", Py_T_INT, offsetof(CustomObject, number), 0,
631 "custom number"},
632 {NULL} /* Sentinel */
633 };
634
635 We also need to update the :c:member:`~PyTypeObject.tp_init` handler to only
636 allow strings [#]_ to be passed::
637
638 static int
639 Custom_init(PyObject *op, PyObject *args, PyObject *kwds)
640 {
641 CustomObject *self = (CustomObject *) op;
642 static char *kwlist[] = {"first", "last", "number", NULL};
643 PyObject *first = NULL, *last = NULL, *tmp;
644
645 if (!PyArg_ParseTupleAndKeywords(args, kwds, "|UUi", kwlist,
646 &first, &last,
647 &self->number))
648 return -1;
649
650 if (first) {
651 tmp = self->first;
652 Py_INCREF(first);
653 self->first = first;
654 Py_DECREF(tmp);
655 }
656 if (last) {
657 tmp = self->last;
658 Py_INCREF(last);
659 self->last = last;
660 Py_DECREF(tmp);
661 }
662 return 0;
663 }
664
665 With these changes, we can assure that the ``first`` and ``last`` members are
666 never ``NULL`` so we can remove checks for ``NULL`` values in almost all cases.
667 This means that most of the :c:func:`Py_XDECREF` calls can be converted to
668 :c:func:`Py_DECREF` calls. The only place we can't change these calls is in
669 the ``tp_dealloc`` implementation, where there is the possibility that the
670 initialization of these members failed in ``tp_new``.
671
672 We also rename the module initialization function and module name in the
673 initialization function, as we did before, and we add an extra definition to the
674 :file:`setup.py` file.
675
676
677 Supporting cyclic garbage collection
678 ====================================
679
680 Python has a :term:`cyclic garbage collector (GC) <garbage collection>` that
681 can identify unneeded objects even when their reference counts are not zero.
682 This can happen when objects are involved in cycles. For example, consider:
683
684 .. code-block:: pycon
685
686 >>> l = []
687 >>> l.append(l)
688 >>> del l
689
690 In this example, we create a list that contains itself. When we delete it, it
691 still has a reference from itself. Its reference count doesn't drop to zero.
692 Fortunately, Python's cyclic garbage collector will eventually figure out that
693 the list is garbage and free it.
694
695 In the second version of the :class:`!Custom` example, we allowed any kind of
696 object to be stored in the :attr:`!first` or :attr:`!last` attributes [#]_.
697 Besides, in the second and third versions, we allowed subclassing
698 :class:`!Custom`, and subclasses may add arbitrary attributes. For any of
699 those two reasons, :class:`!Custom` objects can participate in cycles:
700
701 .. code-block:: pycon
702
703 >>> import custom3
704 >>> class Derived(custom3.Custom): pass
705 ...
706 >>> n = Derived()
707 >>> n.some_attribute = n
708
709 To allow a :class:`!Custom` instance participating in a reference cycle to
710 be properly detected and collected by the cyclic GC, our :class:`!Custom` type
711 needs to fill two additional slots and to enable a flag that enables these slots:
712
713 .. literalinclude:: ../includes/newtypes/custom4.c
714
715
716 First, the traversal method lets the cyclic GC know about subobjects that could
717 participate in cycles::
718
719 static int
720 Custom_traverse(PyObject *op, visitproc visit, void *arg)
721 {
722 CustomObject *self = (CustomObject *) op;
723 int vret;
724 if (self->first) {
725 vret = visit(self->first, arg);
726 if (vret != 0)
727 return vret;
728 }
729 if (self->last) {
730 vret = visit(self->last, arg);
731 if (vret != 0)
732 return vret;
733 }
734 return 0;
735 }
736
737 For each subobject that can participate in cycles, we need to call the
738 :c:func:`!visit` function, which is passed to the traversal method. The
739 :c:func:`!visit` function takes as arguments the subobject and the extra argument
740 *arg* passed to the traversal method. It returns an integer value that must be
741 returned if it is non-zero.
742
743 Python provides a :c:func:`Py_VISIT` macro that automates calling visit
744 functions. With :c:func:`Py_VISIT`, we can minimize the amount of boilerplate
745 in ``Custom_traverse``::
746
747 static int
748 Custom_traverse(PyObject *op, visitproc visit, void *arg)
749 {
750 CustomObject *self = (CustomObject *) op;
751 Py_VISIT(self->first);
752 Py_VISIT(self->last);
753 return 0;
754 }
755
756 .. note::
757 The :c:member:`~PyTypeObject.tp_traverse` implementation must name its
758 arguments exactly *visit* and *arg* in order to use :c:func:`Py_VISIT`.
759
760 Second, we need to provide a method for clearing any subobjects that can
761 participate in cycles::
762
763 static int
764 Custom_clear(PyObject *op)
765 {
766 CustomObject *self = (CustomObject *) op;
767 Py_CLEAR(self->first);
768 Py_CLEAR(self->last);
769 return 0;
770 }
771
772 Notice the use of the :c:func:`Py_CLEAR` macro. It is the recommended and safe
773 way to clear data attributes of arbitrary types while decrementing
774 their reference counts. If you were to call :c:func:`Py_XDECREF` instead
775 on the attribute before setting it to ``NULL``, there is a possibility
776 that the attribute's destructor would call back into code that reads the
777 attribute again (*especially* if there is a reference cycle).
778
779 .. note::
780 You could emulate :c:func:`Py_CLEAR` by writing::
781
782 PyObject *tmp;
783 tmp = self->first;
784 self->first = NULL;
785 Py_XDECREF(tmp);
786
787 Nevertheless, it is much easier and less error-prone to always
788 use :c:func:`Py_CLEAR` when deleting an attribute. Don't
789 try to micro-optimize at the expense of robustness!
790
791 The deallocator ``Custom_dealloc`` may call arbitrary code when clearing
792 attributes. It means the circular GC can be triggered inside the function.
793 Since the GC assumes reference count is not zero, we need to untrack the object
794 from the GC by calling :c:func:`PyObject_GC_UnTrack` before clearing members.
795 Here is our reimplemented deallocator using :c:func:`PyObject_GC_UnTrack`
796 and ``Custom_clear``::
797
798 static void
799 Custom_dealloc(PyObject *op)
800 {
801 PyObject_GC_UnTrack(op);
802 (void)Custom_clear(op);
803 Py_TYPE(op)->tp_free(op);
804 }
805
806 Finally, we add the :c:macro:`Py_TPFLAGS_HAVE_GC` flag to the class flags::
807
808 .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC,
809
810 That's pretty much it. If we had written custom :c:member:`~PyTypeObject.tp_alloc` or
811 :c:member:`~PyTypeObject.tp_free` handlers, we'd need to modify them for cyclic
812 garbage collection. Most extensions will use the versions automatically provided.
813
814
815 Subclassing other types
816 =======================
817
818 It is possible to create new extension types that are derived from existing
819 types. It is easiest to inherit from the built in types, since an extension can
820 easily use the :c:type:`PyTypeObject` it needs. It can be difficult to share
821 these :c:type:`PyTypeObject` structures between extension modules.
822
823 In this example we will create a :class:`!SubList` type that inherits from the
824 built-in :class:`list` type. The new type will be completely compatible with
825 regular lists, but will have an additional :meth:`!increment` method that
826 increases an internal counter:
827
828 .. code-block:: pycon
829
830 >>> import sublist
831 >>> s = sublist.SubList(range(3))
832 >>> s.extend(s)
833 >>> print(len(s))
834 6
835 >>> print(s.increment())
836 1
837 >>> print(s.increment())
838 2
839
840 .. literalinclude:: ../includes/newtypes/sublist.c
841
842
843 As you can see, the source code closely resembles the :class:`!Custom` examples in
844 previous sections. We will break down the main differences between them. ::
845
846 typedef struct {
847 PyListObject list;
848 int state;
849 } SubListObject;
850
851 The primary difference for derived type objects is that the base type's
852 object structure must be the first value. The base type will already include
853 the :c:func:`PyObject_HEAD` at the beginning of its structure.
854
855 When a Python object is a :class:`!SubList` instance, its ``PyObject *`` pointer
856 can be safely cast to both ``PyListObject *`` and ``SubListObject *``::
857
858 static int
859 SubList_init(PyObject *op, PyObject *args, PyObject *kwds)
860 {
861 SubListObject *self = (SubListObject *) op;
862 if (PyList_Type.tp_init(op, args, kwds) < 0)
863 return -1;
864 self->state = 0;
865 return 0;
866 }
867
868 We see above how to call through to the :meth:`~object.__init__` method of the base
869 type.
870
871 This pattern is important when writing a type with custom
872 :c:member:`~PyTypeObject.tp_new` and :c:member:`~PyTypeObject.tp_dealloc`
873 members. The :c:member:`~PyTypeObject.tp_new` handler should not actually
874 create the memory for the object with its :c:member:`~PyTypeObject.tp_alloc`,
875 but let the base class handle it by calling its own :c:member:`~PyTypeObject.tp_new`.
876
877 The :c:type:`PyTypeObject` struct supports a :c:member:`~PyTypeObject.tp_base`
878 specifying the type's concrete base class. Due to cross-platform compiler
879 issues, you can't fill that field directly with a reference to
880 :c:type:`PyList_Type`; it should be done in the :c:data:`Py_mod_exec`
881 function::
882
883 static int
884 sublist_module_exec(PyObject *m)
885 {
886 SubListType.tp_base = &PyList_Type;
887 if (PyType_Ready(&SubListType) < 0) {
888 return -1;
889 }
890
891 if (PyModule_AddObjectRef(m, "SubList", (PyObject *) &SubListType) < 0) {
892 return -1;
893 }
894
895 return 0;
896 }
897
898 Before calling :c:func:`PyType_Ready`, the type structure must have the
899 :c:member:`~PyTypeObject.tp_base` slot filled in. When we are deriving an
900 existing type, it is not necessary to fill out the :c:member:`~PyTypeObject.tp_alloc`
901 slot with :c:func:`PyType_GenericNew` -- the allocation function from the base
902 type will be inherited.
903
904 After that, calling :c:func:`PyType_Ready` and adding the type object to the
905 module is the same as with the basic :class:`!Custom` examples.
906
907
908 .. rubric:: Footnotes
909
910 .. [#] This is true when we know that the object is a basic type, like a string or a
911 float.
912
913 .. [#] We relied on this in the :c:member:`~PyTypeObject.tp_dealloc` handler
914 in this example, because our type doesn't support garbage collection.
915
916 .. [#] We now know that the first and last members are strings, so perhaps we
917 could be less careful about decrementing their reference counts, however,
918 we accept instances of string subclasses. Even though deallocating normal
919 strings won't call back into our objects, we can't guarantee that deallocating
920 an instance of a string subclass won't call back into our objects.
921
922 .. [#] Also, even with our attributes restricted to strings instances, the user
923 could pass arbitrary :class:`str` subclasses and therefore still create
924 reference cycles.