| 1 |
.. highlight:: c |
| 2 |
|
| 3 |
.. _defining-new-types: |
| 4 |
|
| 5 |
********************************** |
| 6 |
Defining Extension Types: Tutorial |
| 7 |
********************************** |
| 8 |
|
| 9 |
.. sectionauthor:: Michael Hudson <[email protected]> |
| 10 |
.. sectionauthor:: Dave Kuhlman <[email protected]> |
| 11 |
.. sectionauthor:: Jim Fulton <[email protected]> |
| 12 |
|
| 13 |
|
| 14 |
Python allows the writer of a C extension module to define new types that |
| 15 |
can be manipulated from Python code, much like the built-in :class:`str` |
| 16 |
and :class:`list` types. The code for all extension types follows a |
| 17 |
pattern, but there are some details that you need to understand before you |
| 18 |
can get started. This document is a gentle introduction to the topic. |
| 19 |
|
| 20 |
|
| 21 |
.. _dnt-basics: |
| 22 |
|
| 23 |
The Basics |
| 24 |
========== |
| 25 |
|
| 26 |
The :term:`CPython` runtime sees all Python objects as variables of type |
| 27 |
:c:expr:`PyObject*`, which serves as a "base type" for all Python objects. |
| 28 |
The :c:type:`PyObject` structure itself only contains the object's |
| 29 |
:term:`reference count` and a pointer to the object's "type object". |
| 30 |
This is where the action is; the type object determines which (C) functions |
| 31 |
get called by the interpreter when, for instance, an attribute gets looked up |
| 32 |
on an object, a method called, or it is multiplied by another object. These |
| 33 |
C functions are called "type methods". |
| 34 |
|
| 35 |
So, if you want to define a new extension type, you need to create a new type |
| 36 |
object. |
| 37 |
|
| 38 |
This sort of thing can only be explained by example, so here's a minimal, but |
| 39 |
complete, module that defines a new type named :class:`!Custom` inside a C |
| 40 |
extension module :mod:`!custom`: |
| 41 |
|
| 42 |
.. note:: |
| 43 |
What we're showing here is the traditional way of defining *static* |
| 44 |
extension types. It should be adequate for most uses. The C API also |
| 45 |
allows defining heap-allocated extension types using the |
| 46 |
:c:func:`PyType_FromSpec` function, which isn't covered in this tutorial. |
| 47 |
|
| 48 |
.. literalinclude:: ../includes/newtypes/custom.c |
| 49 |
|
| 50 |
Now that's quite a bit to take in at once, but hopefully bits will seem familiar |
| 51 |
from the previous chapter. This file defines three things: |
| 52 |
|
| 53 |
#. What a :class:`!Custom` **object** contains: this is the ``CustomObject`` |
| 54 |
struct, which is allocated once for each :class:`!Custom` instance. |
| 55 |
#. How the :class:`!Custom` **type** behaves: this is the ``CustomType`` struct, |
| 56 |
which defines a set of flags and function pointers that the interpreter |
| 57 |
inspects when specific operations are requested. |
| 58 |
#. How to define and execute the :mod:`!custom` module: this is the |
| 59 |
``PyInit_custom`` function and the associated ``custom_module`` struct for |
| 60 |
defining the module, and the ``custom_module_exec`` function to set up |
| 61 |
a fresh module object. |
| 62 |
|
| 63 |
The first bit is:: |
| 64 |
|
| 65 |
typedef struct { |
| 66 |
PyObject_HEAD |
| 67 |
} CustomObject; |
| 68 |
|
| 69 |
This is what a Custom object will contain. ``PyObject_HEAD`` is mandatory |
| 70 |
at the start of each object struct and defines a field called ``ob_base`` |
| 71 |
of type :c:type:`PyObject`, containing a pointer to a type object and a |
| 72 |
reference count (these can be accessed using the macros :c:macro:`Py_TYPE` |
| 73 |
and :c:macro:`Py_REFCNT` respectively). The reason for the macro is to |
| 74 |
abstract away the layout and to enable additional fields in :ref:`debug builds |
| 75 |
<debug-build>`. |
| 76 |
|
| 77 |
.. note:: |
| 78 |
There is no semicolon above after the :c:macro:`PyObject_HEAD` macro. |
| 79 |
Be wary of adding one by accident: some compilers will complain. |
| 80 |
|
| 81 |
Of course, objects generally store additional data besides the standard |
| 82 |
``PyObject_HEAD`` boilerplate; for example, here is the definition for |
| 83 |
standard Python floats:: |
| 84 |
|
| 85 |
typedef struct { |
| 86 |
PyObject_HEAD |
| 87 |
double ob_fval; |
| 88 |
} PyFloatObject; |
| 89 |
|
| 90 |
The second bit is the definition of the type object. :: |
| 91 |
|
| 92 |
static PyTypeObject CustomType = { |
| 93 |
.ob_base = PyVarObject_HEAD_INIT(NULL, 0) |
| 94 |
.tp_name = "custom.Custom", |
| 95 |
.tp_doc = PyDoc_STR("Custom objects"), |
| 96 |
.tp_basicsize = sizeof(CustomObject), |
| 97 |
.tp_itemsize = 0, |
| 98 |
.tp_flags = Py_TPFLAGS_DEFAULT, |
| 99 |
.tp_new = PyType_GenericNew, |
| 100 |
}; |
| 101 |
|
| 102 |
.. note:: |
| 103 |
We recommend using C99-style designated initializers as above, to |
| 104 |
avoid listing all the :c:type:`PyTypeObject` fields that you don't care |
| 105 |
about and also to avoid caring about the fields' declaration order. |
| 106 |
|
| 107 |
The actual definition of :c:type:`PyTypeObject` in :file:`object.h` has |
| 108 |
many more :ref:`fields <type-structs>` than the definition above. The |
| 109 |
remaining fields will be filled with zeros by the C compiler, and it's |
| 110 |
common practice to not specify them explicitly unless you need them. |
| 111 |
|
| 112 |
We're going to pick it apart, one field at a time:: |
| 113 |
|
| 114 |
.ob_base = PyVarObject_HEAD_INIT(NULL, 0) |
| 115 |
|
| 116 |
This line is mandatory boilerplate to initialize the ``ob_base`` |
| 117 |
field mentioned above. :: |
| 118 |
|
| 119 |
.tp_name = "custom.Custom", |
| 120 |
|
| 121 |
The name of our type. This will appear in the default textual representation of |
| 122 |
our objects and in some error messages, for example: |
| 123 |
|
| 124 |
.. code-block:: pycon |
| 125 |
|
| 126 |
>>> "" + custom.Custom() |
| 127 |
Traceback (most recent call last): |
| 128 |
File "<stdin>", line 1, in <module> |
| 129 |
TypeError: can only concatenate str (not "custom.Custom") to str |
| 130 |
|
| 131 |
Note that the name is a dotted name that includes both the module name and the |
| 132 |
name of the type within the module. The module in this case is :mod:`!custom` and |
| 133 |
the type is :class:`!Custom`, so we set the type name to :class:`!custom.Custom`. |
| 134 |
Using the real dotted import path is important to make your type compatible |
| 135 |
with the :mod:`pydoc` and :mod:`pickle` modules. :: |
| 136 |
|
| 137 |
.tp_basicsize = sizeof(CustomObject), |
| 138 |
.tp_itemsize = 0, |
| 139 |
|
| 140 |
This is so that Python knows how much memory to allocate when creating |
| 141 |
new :class:`!Custom` instances. :c:member:`~PyTypeObject.tp_itemsize` is |
| 142 |
only used for variable-sized objects and should otherwise be zero. |
| 143 |
|
| 144 |
.. note:: |
| 145 |
|
| 146 |
If you want your type to be subclassable from Python, and your type has the same |
| 147 |
:c:member:`~PyTypeObject.tp_basicsize` as its base type, you may have problems with multiple |
| 148 |
inheritance. A Python subclass of your type will have to list your type first |
| 149 |
in its :attr:`~type.__bases__`, or else it will not be able to call your type's |
| 150 |
:meth:`~object.__new__` method without getting an error. You can avoid this problem by |
| 151 |
ensuring that your type has a larger value for :c:member:`~PyTypeObject.tp_basicsize` than its |
| 152 |
base type does. Most of the time, this will be true anyway, because either your |
| 153 |
base type will be :class:`object`, or else you will be adding data members to |
| 154 |
your base type, and therefore increasing its size. |
| 155 |
|
| 156 |
We set the class flags to :c:macro:`Py_TPFLAGS_DEFAULT`. :: |
| 157 |
|
| 158 |
.tp_flags = Py_TPFLAGS_DEFAULT, |
| 159 |
|
| 160 |
All types should include this constant in their flags. It enables all of the |
| 161 |
members defined until at least Python 3.3. If you need further members, |
| 162 |
you will need to OR the corresponding flags. |
| 163 |
|
| 164 |
We provide a doc string for the type in :c:member:`~PyTypeObject.tp_doc`. :: |
| 165 |
|
| 166 |
.tp_doc = PyDoc_STR("Custom objects"), |
| 167 |
|
| 168 |
To enable object creation, we have to provide a :c:member:`~PyTypeObject.tp_new` |
| 169 |
handler. This is the equivalent of the Python method :meth:`~object.__new__`, but |
| 170 |
has to be specified explicitly. In this case, we can just use the default |
| 171 |
implementation provided by the API function :c:func:`PyType_GenericNew`. :: |
| 172 |
|
| 173 |
.tp_new = PyType_GenericNew, |
| 174 |
|
| 175 |
Everything else in the file should be familiar, except for some code in |
| 176 |
:c:func:`!custom_module_exec`:: |
| 177 |
|
| 178 |
if (PyType_Ready(&CustomType) < 0) { |
| 179 |
return -1; |
| 180 |
} |
| 181 |
|
| 182 |
This initializes the :class:`!Custom` type, filling in a number of members |
| 183 |
to the appropriate default values, including :c:member:`~PyObject.ob_type` that we initially |
| 184 |
set to ``NULL``. :: |
| 185 |
|
| 186 |
if (PyModule_AddObjectRef(m, "Custom", (PyObject *) &CustomType) < 0) { |
| 187 |
return -1; |
| 188 |
} |
| 189 |
|
| 190 |
This adds the type to the module dictionary. This allows us to create |
| 191 |
:class:`!Custom` instances by calling the :class:`!Custom` class: |
| 192 |
|
| 193 |
.. code-block:: pycon |
| 194 |
|
| 195 |
>>> import custom |
| 196 |
>>> mycustom = custom.Custom() |
| 197 |
|
| 198 |
That's it! All that remains is to build it; put the above code in a file called |
| 199 |
:file:`custom.c`, |
| 200 |
|
| 201 |
.. literalinclude:: ../includes/newtypes/pyproject.toml |
| 202 |
|
| 203 |
in a file called :file:`pyproject.toml`, and |
| 204 |
|
| 205 |
.. code-block:: python |
| 206 |
|
| 207 |
from setuptools import Extension, setup |
| 208 |
setup(ext_modules=[Extension("custom", ["custom.c"])]) |
| 209 |
|
| 210 |
in a file called :file:`setup.py`; then typing |
| 211 |
|
| 212 |
.. code-block:: shell-session |
| 213 |
|
| 214 |
$ python -m pip install . |
| 215 |
|
| 216 |
in a shell should produce a file :file:`custom.so` in a subdirectory |
| 217 |
and install it; now fire up Python --- you should be able to ``import custom`` |
| 218 |
and play around with ``Custom`` objects. |
| 219 |
|
| 220 |
That wasn't so hard, was it? |
| 221 |
|
| 222 |
Of course, the current Custom type is pretty uninteresting. It has no data and |
| 223 |
doesn't do anything. It can't even be subclassed. |
| 224 |
|
| 225 |
|
| 226 |
Adding data and methods to the Basic example |
| 227 |
============================================ |
| 228 |
|
| 229 |
Let's extend the basic example to add some data and methods. Let's also make |
| 230 |
the type usable as a base class. We'll create a new module, :mod:`!custom2` that |
| 231 |
adds these capabilities: |
| 232 |
|
| 233 |
.. literalinclude:: ../includes/newtypes/custom2.c |
| 234 |
|
| 235 |
|
| 236 |
This version of the module has a number of changes. |
| 237 |
|
| 238 |
The :class:`!Custom` type now has three data attributes in its C struct, |
| 239 |
*first*, *last*, and *number*. The *first* and *last* variables are Python |
| 240 |
strings containing first and last names. The *number* attribute is a C integer. |
| 241 |
|
| 242 |
The object structure is updated accordingly:: |
| 243 |
|
| 244 |
typedef struct { |
| 245 |
PyObject_HEAD |
| 246 |
PyObject *first; /* first name */ |
| 247 |
PyObject *last; /* last name */ |
| 248 |
int number; |
| 249 |
} CustomObject; |
| 250 |
|
| 251 |
Because we now have data to manage, we have to be more careful about object |
| 252 |
allocation and deallocation. At a minimum, we need a deallocation method:: |
| 253 |
|
| 254 |
static void |
| 255 |
Custom_dealloc(PyObject *op) |
| 256 |
{ |
| 257 |
CustomObject *self = (CustomObject *) op; |
| 258 |
Py_XDECREF(self->first); |
| 259 |
Py_XDECREF(self->last); |
| 260 |
Py_TYPE(self)->tp_free(self); |
| 261 |
} |
| 262 |
|
| 263 |
which is assigned to the :c:member:`~PyTypeObject.tp_dealloc` member:: |
| 264 |
|
| 265 |
.tp_dealloc = Custom_dealloc, |
| 266 |
|
| 267 |
This method first clears the reference counts of the two Python attributes. |
| 268 |
:c:func:`Py_XDECREF` correctly handles the case where its argument is |
| 269 |
``NULL`` (which might happen here if ``tp_new`` failed midway). It then |
| 270 |
calls the :c:member:`~PyTypeObject.tp_free` member of the object's type |
| 271 |
(computed by ``Py_TYPE(self)``) to free the object's memory. Note that |
| 272 |
the object's type might not be :class:`!CustomType`, because the object may |
| 273 |
be an instance of a subclass. |
| 274 |
|
| 275 |
.. note:: |
| 276 |
|
| 277 |
The explicit cast to ``CustomObject *`` above is needed because we defined |
| 278 |
``Custom_dealloc`` to take a ``PyObject *`` argument, as the ``tp_dealloc`` |
| 279 |
function pointer expects to receive a ``PyObject *`` argument. |
| 280 |
By assigning to the ``tp_dealloc`` slot of a type, we declare |
| 281 |
that it can only be called with instances of our ``CustomObject`` |
| 282 |
class, so the cast to ``(CustomObject *)`` is safe. |
| 283 |
This is object-oriented polymorphism, in C! |
| 284 |
|
| 285 |
In existing code, or in previous versions of this tutorial, |
| 286 |
you might see similar functions take a pointer to the subtype |
| 287 |
object structure (``CustomObject*``) directly, like this:: |
| 288 |
|
| 289 |
Custom_dealloc(CustomObject *self) |
| 290 |
{ |
| 291 |
Py_XDECREF(self->first); |
| 292 |
Py_XDECREF(self->last); |
| 293 |
Py_TYPE(self)->tp_free((PyObject *) self); |
| 294 |
} |
| 295 |
... |
| 296 |
.tp_dealloc = (destructor) Custom_dealloc, |
| 297 |
|
| 298 |
This does the same thing on all architectures that CPython |
| 299 |
supports, but according to the C standard, it invokes |
| 300 |
undefined behavior. |
| 301 |
|
| 302 |
We want to make sure that the first and last names are initialized to empty |
| 303 |
strings, so we provide a ``tp_new`` implementation:: |
| 304 |
|
| 305 |
static PyObject * |
| 306 |
Custom_new(PyTypeObject *type, PyObject *args, PyObject *kwds) |
| 307 |
{ |
| 308 |
CustomObject *self; |
| 309 |
self = (CustomObject *) type->tp_alloc(type, 0); |
| 310 |
if (self != NULL) { |
| 311 |
self->first = PyUnicode_FromString(""); |
| 312 |
if (self->first == NULL) { |
| 313 |
Py_DECREF(self); |
| 314 |
return NULL; |
| 315 |
} |
| 316 |
self->last = PyUnicode_FromString(""); |
| 317 |
if (self->last == NULL) { |
| 318 |
Py_DECREF(self); |
| 319 |
return NULL; |
| 320 |
} |
| 321 |
self->number = 0; |
| 322 |
} |
| 323 |
return (PyObject *) self; |
| 324 |
} |
| 325 |
|
| 326 |
and install it in the :c:member:`~PyTypeObject.tp_new` member:: |
| 327 |
|
| 328 |
.tp_new = Custom_new, |
| 329 |
|
| 330 |
The ``tp_new`` handler is responsible for creating (as opposed to initializing) |
| 331 |
objects of the type. It is exposed in Python as the :meth:`~object.__new__` method. |
| 332 |
It is not required to define a ``tp_new`` member, and indeed many extension |
| 333 |
types will simply reuse :c:func:`PyType_GenericNew` as done in the first |
| 334 |
version of the :class:`!Custom` type above. In this case, we use the ``tp_new`` |
| 335 |
handler to initialize the ``first`` and ``last`` attributes to non-``NULL`` |
| 336 |
default values. |
| 337 |
|
| 338 |
``tp_new`` is passed the type being instantiated (not necessarily ``CustomType``, |
| 339 |
if a subclass is instantiated) and any arguments passed when the type was |
| 340 |
called, and is expected to return the instance created. ``tp_new`` handlers |
| 341 |
always accept positional and keyword arguments, but they often ignore the |
| 342 |
arguments, leaving the argument handling to initializer (a.k.a. ``tp_init`` |
| 343 |
in C or ``__init__`` in Python) methods. |
| 344 |
|
| 345 |
.. note:: |
| 346 |
``tp_new`` shouldn't call ``tp_init`` explicitly, as the interpreter |
| 347 |
will do it itself. |
| 348 |
|
| 349 |
The ``tp_new`` implementation calls the :c:member:`~PyTypeObject.tp_alloc` |
| 350 |
slot to allocate memory:: |
| 351 |
|
| 352 |
self = (CustomObject *) type->tp_alloc(type, 0); |
| 353 |
|
| 354 |
Since memory allocation may fail, we must check the :c:member:`~PyTypeObject.tp_alloc` |
| 355 |
result against ``NULL`` before proceeding. |
| 356 |
|
| 357 |
.. note:: |
| 358 |
We didn't fill the :c:member:`~PyTypeObject.tp_alloc` slot ourselves. Rather |
| 359 |
:c:func:`PyType_Ready` fills it for us by inheriting it from our base class, |
| 360 |
which is :class:`object` by default. Most types use the default allocation |
| 361 |
strategy. |
| 362 |
|
| 363 |
.. note:: |
| 364 |
If you are creating a co-operative :c:member:`~PyTypeObject.tp_new` (one |
| 365 |
that calls a base type's :c:member:`~PyTypeObject.tp_new` or :meth:`~object.__new__`), |
| 366 |
you must *not* try to determine what method to call using method resolution |
| 367 |
order at runtime. Always statically determine what type you are going to |
| 368 |
call, and call its :c:member:`~PyTypeObject.tp_new` directly, or via |
| 369 |
``type->tp_base->tp_new``. If you do not do this, Python subclasses of your |
| 370 |
type that also inherit from other Python-defined classes may not work correctly. |
| 371 |
(Specifically, you may not be able to create instances of such subclasses |
| 372 |
without getting a :exc:`TypeError`.) |
| 373 |
|
| 374 |
We also define an initialization function which accepts arguments to provide |
| 375 |
initial values for our instance:: |
| 376 |
|
| 377 |
static int |
| 378 |
Custom_init(PyObject *op, PyObject *args, PyObject *kwds) |
| 379 |
{ |
| 380 |
CustomObject *self = (CustomObject *) op; |
| 381 |
static char *kwlist[] = {"first", "last", "number", NULL}; |
| 382 |
PyObject *first = NULL, *last = NULL, *tmp; |
| 383 |
|
| 384 |
if (!PyArg_ParseTupleAndKeywords(args, kwds, "|OOi", kwlist, |
| 385 |
&first, &last, |
| 386 |
&self->number)) |
| 387 |
return -1; |
| 388 |
|
| 389 |
if (first) { |
| 390 |
tmp = self->first; |
| 391 |
Py_INCREF(first); |
| 392 |
self->first = first; |
| 393 |
Py_XDECREF(tmp); |
| 394 |
} |
| 395 |
if (last) { |
| 396 |
tmp = self->last; |
| 397 |
Py_INCREF(last); |
| 398 |
self->last = last; |
| 399 |
Py_XDECREF(tmp); |
| 400 |
} |
| 401 |
return 0; |
| 402 |
} |
| 403 |
|
| 404 |
by filling the :c:member:`~PyTypeObject.tp_init` slot. :: |
| 405 |
|
| 406 |
.tp_init = Custom_init, |
| 407 |
|
| 408 |
The :c:member:`~PyTypeObject.tp_init` slot is exposed in Python as the |
| 409 |
:meth:`~object.__init__` method. It is used to initialize an object after it's |
| 410 |
created. Initializers always accept positional and keyword arguments, |
| 411 |
and they should return either ``0`` on success or ``-1`` on error. |
| 412 |
|
| 413 |
Unlike the ``tp_new`` handler, there is no guarantee that ``tp_init`` |
| 414 |
is called at all (for example, the :mod:`pickle` module by default |
| 415 |
doesn't call :meth:`~object.__init__` on unpickled instances). It can also be |
| 416 |
called multiple times. Anyone can call the :meth:`!__init__` method on |
| 417 |
our objects. For this reason, we have to be extra careful when assigning |
| 418 |
the new attribute values. We might be tempted, for example to assign the |
| 419 |
``first`` member like this:: |
| 420 |
|
| 421 |
if (first) { |
| 422 |
Py_XDECREF(self->first); |
| 423 |
Py_INCREF(first); |
| 424 |
self->first = first; |
| 425 |
} |
| 426 |
|
| 427 |
But this would be risky. Our type doesn't restrict the type of the |
| 428 |
``first`` member, so it could be any kind of object. It could have a |
| 429 |
destructor that causes code to be executed that tries to access the |
| 430 |
``first`` member; or that destructor could detach the |
| 431 |
:term:`thread state <attached thread state>` and let arbitrary code run in other |
| 432 |
threads that accesses and modifies our object. |
| 433 |
|
| 434 |
To be paranoid and protect ourselves against this possibility, we almost |
| 435 |
always reassign members before decrementing their reference counts. When |
| 436 |
don't we have to do this? |
| 437 |
|
| 438 |
* when we absolutely know that the reference count is greater than 1; |
| 439 |
|
| 440 |
* when we know that deallocation of the object [#]_ will neither detach |
| 441 |
the :term:`thread state <attached thread state>` nor cause any calls back into our type's code; |
| 442 |
|
| 443 |
* when decrementing a reference count in a :c:member:`~PyTypeObject.tp_dealloc` |
| 444 |
handler on a type which doesn't support cyclic garbage collection [#]_. |
| 445 |
|
| 446 |
We want to expose our instance variables as attributes. There are a |
| 447 |
number of ways to do that. The simplest way is to define member definitions:: |
| 448 |
|
| 449 |
static PyMemberDef Custom_members[] = { |
| 450 |
{"first", Py_T_OBJECT_EX, offsetof(CustomObject, first), 0, |
| 451 |
"first name"}, |
| 452 |
{"last", Py_T_OBJECT_EX, offsetof(CustomObject, last), 0, |
| 453 |
"last name"}, |
| 454 |
{"number", Py_T_INT, offsetof(CustomObject, number), 0, |
| 455 |
"custom number"}, |
| 456 |
{NULL} /* Sentinel */ |
| 457 |
}; |
| 458 |
|
| 459 |
and put the definitions in the :c:member:`~PyTypeObject.tp_members` slot:: |
| 460 |
|
| 461 |
.tp_members = Custom_members, |
| 462 |
|
| 463 |
Each member definition has a member name, type, offset, access flags and |
| 464 |
documentation string. See the :ref:`Generic-Attribute-Management` section |
| 465 |
below for details. |
| 466 |
|
| 467 |
A disadvantage of this approach is that it doesn't provide a way to restrict the |
| 468 |
types of objects that can be assigned to the Python attributes. We expect the |
| 469 |
first and last names to be strings, but any Python objects can be assigned. |
| 470 |
Further, the attributes can be deleted, setting the C pointers to ``NULL``. Even |
| 471 |
though we can make sure the members are initialized to non-``NULL`` values, the |
| 472 |
members can be set to ``NULL`` if the attributes are deleted. |
| 473 |
|
| 474 |
We define a single method, :meth:`!Custom.name`, that outputs the objects name as the |
| 475 |
concatenation of the first and last names. :: |
| 476 |
|
| 477 |
static PyObject * |
| 478 |
Custom_name(PyObject *op, PyObject *Py_UNUSED(dummy)) |
| 479 |
{ |
| 480 |
CustomObject *self = (CustomObject *) op; |
| 481 |
if (self->first == NULL) { |
| 482 |
PyErr_SetString(PyExc_AttributeError, "first"); |
| 483 |
return NULL; |
| 484 |
} |
| 485 |
if (self->last == NULL) { |
| 486 |
PyErr_SetString(PyExc_AttributeError, "last"); |
| 487 |
return NULL; |
| 488 |
} |
| 489 |
return PyUnicode_FromFormat("%S %S", self->first, self->last); |
| 490 |
} |
| 491 |
|
| 492 |
The method is implemented as a C function that takes a :class:`!Custom` (or |
| 493 |
:class:`!Custom` subclass) instance as the first argument. Methods always take an |
| 494 |
instance as the first argument. Methods often take positional and keyword |
| 495 |
arguments as well, but in this case we don't take any and don't need to accept |
| 496 |
a positional argument tuple or keyword argument dictionary. This method is |
| 497 |
equivalent to the Python method: |
| 498 |
|
| 499 |
.. code-block:: python |
| 500 |
|
| 501 |
def name(self): |
| 502 |
return "%s %s" % (self.first, self.last) |
| 503 |
|
| 504 |
Note that we have to check for the possibility that our :attr:`!first` and |
| 505 |
:attr:`!last` members are ``NULL``. This is because they can be deleted, in which |
| 506 |
case they are set to ``NULL``. It would be better to prevent deletion of these |
| 507 |
attributes and to restrict the attribute values to be strings. We'll see how to |
| 508 |
do that in the next section. |
| 509 |
|
| 510 |
Now that we've defined the method, we need to create an array of method |
| 511 |
definitions:: |
| 512 |
|
| 513 |
static PyMethodDef Custom_methods[] = { |
| 514 |
{"name", Custom_name, METH_NOARGS, |
| 515 |
"Return the name, combining the first and last name" |
| 516 |
}, |
| 517 |
{NULL} /* Sentinel */ |
| 518 |
}; |
| 519 |
|
| 520 |
(note that we used the :c:macro:`METH_NOARGS` flag to indicate that the method |
| 521 |
is expecting no arguments other than *self*) |
| 522 |
|
| 523 |
and assign it to the :c:member:`~PyTypeObject.tp_methods` slot:: |
| 524 |
|
| 525 |
.tp_methods = Custom_methods, |
| 526 |
|
| 527 |
Finally, we'll make our type usable as a base class for subclassing. We've |
| 528 |
written our methods carefully so far so that they don't make any assumptions |
| 529 |
about the type of the object being created or used, so all we need to do is |
| 530 |
to add the :c:macro:`Py_TPFLAGS_BASETYPE` to our class flag definition:: |
| 531 |
|
| 532 |
.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, |
| 533 |
|
| 534 |
We rename :c:func:`!PyInit_custom` to :c:func:`!PyInit_custom2`, update the |
| 535 |
module name in the :c:type:`PyModuleDef` struct, and update the full class |
| 536 |
name in the :c:type:`PyTypeObject` struct. |
| 537 |
|
| 538 |
Finally, we update our :file:`setup.py` file to include the new module, |
| 539 |
|
| 540 |
.. code-block:: python |
| 541 |
|
| 542 |
from setuptools import Extension, setup |
| 543 |
setup(ext_modules=[ |
| 544 |
Extension("custom", ["custom.c"]), |
| 545 |
Extension("custom2", ["custom2.c"]), |
| 546 |
]) |
| 547 |
|
| 548 |
and then we re-install so that we can ``import custom2``: |
| 549 |
|
| 550 |
.. code-block:: shell-session |
| 551 |
|
| 552 |
$ python -m pip install . |
| 553 |
|
| 554 |
Providing finer control over data attributes |
| 555 |
============================================ |
| 556 |
|
| 557 |
In this section, we'll provide finer control over how the :attr:`!first` and |
| 558 |
:attr:`!last` attributes are set in the :class:`!Custom` example. In the previous |
| 559 |
version of our module, the instance variables :attr:`!first` and :attr:`!last` |
| 560 |
could be set to non-string values or even deleted. We want to make sure that |
| 561 |
these attributes always contain strings. |
| 562 |
|
| 563 |
.. literalinclude:: ../includes/newtypes/custom3.c |
| 564 |
|
| 565 |
|
| 566 |
To provide greater control, over the :attr:`!first` and :attr:`!last` attributes, |
| 567 |
we'll use custom getter and setter functions. Here are the functions for |
| 568 |
getting and setting the :attr:`!first` attribute:: |
| 569 |
|
| 570 |
static PyObject * |
| 571 |
Custom_getfirst(PyObject *op, void *closure) |
| 572 |
{ |
| 573 |
CustomObject *self = (CustomObject *) op; |
| 574 |
Py_INCREF(self->first); |
| 575 |
return self->first; |
| 576 |
} |
| 577 |
|
| 578 |
static int |
| 579 |
Custom_setfirst(PyObject *op, PyObject *value, void *closure) |
| 580 |
{ |
| 581 |
CustomObject *self = (CustomObject *) op; |
| 582 |
PyObject *tmp; |
| 583 |
if (value == NULL) { |
| 584 |
PyErr_SetString(PyExc_TypeError, "Cannot delete the first attribute"); |
| 585 |
return -1; |
| 586 |
} |
| 587 |
if (!PyUnicode_Check(value)) { |
| 588 |
PyErr_SetString(PyExc_TypeError, |
| 589 |
"The first attribute value must be a string"); |
| 590 |
return -1; |
| 591 |
} |
| 592 |
tmp = self->first; |
| 593 |
Py_INCREF(value); |
| 594 |
self->first = value; |
| 595 |
Py_DECREF(tmp); |
| 596 |
return 0; |
| 597 |
} |
| 598 |
|
| 599 |
The getter function is passed a :class:`!Custom` object and a "closure", which is |
| 600 |
a void pointer. In this case, the closure is ignored. (The closure supports an |
| 601 |
advanced usage in which definition data is passed to the getter and setter. This |
| 602 |
could, for example, be used to allow a single set of getter and setter functions |
| 603 |
that decide the attribute to get or set based on data in the closure.) |
| 604 |
|
| 605 |
The setter function is passed the :class:`!Custom` object, the new value, and the |
| 606 |
closure. The new value may be ``NULL``, in which case the attribute is being |
| 607 |
deleted. In our setter, we raise an error if the attribute is deleted or if its |
| 608 |
new value is not a string. |
| 609 |
|
| 610 |
We create an array of :c:type:`PyGetSetDef` structures:: |
| 611 |
|
| 612 |
static PyGetSetDef Custom_getsetters[] = { |
| 613 |
{"first", Custom_getfirst, Custom_setfirst, |
| 614 |
"first name", NULL}, |
| 615 |
{"last", Custom_getlast, Custom_setlast, |
| 616 |
"last name", NULL}, |
| 617 |
{NULL} /* Sentinel */ |
| 618 |
}; |
| 619 |
|
| 620 |
and register it in the :c:member:`~PyTypeObject.tp_getset` slot:: |
| 621 |
|
| 622 |
.tp_getset = Custom_getsetters, |
| 623 |
|
| 624 |
The last item in a :c:type:`PyGetSetDef` structure is the "closure" mentioned |
| 625 |
above. In this case, we aren't using a closure, so we just pass ``NULL``. |
| 626 |
|
| 627 |
We also remove the member definitions for these attributes:: |
| 628 |
|
| 629 |
static PyMemberDef Custom_members[] = { |
| 630 |
{"number", Py_T_INT, offsetof(CustomObject, number), 0, |
| 631 |
"custom number"}, |
| 632 |
{NULL} /* Sentinel */ |
| 633 |
}; |
| 634 |
|
| 635 |
We also need to update the :c:member:`~PyTypeObject.tp_init` handler to only |
| 636 |
allow strings [#]_ to be passed:: |
| 637 |
|
| 638 |
static int |
| 639 |
Custom_init(PyObject *op, PyObject *args, PyObject *kwds) |
| 640 |
{ |
| 641 |
CustomObject *self = (CustomObject *) op; |
| 642 |
static char *kwlist[] = {"first", "last", "number", NULL}; |
| 643 |
PyObject *first = NULL, *last = NULL, *tmp; |
| 644 |
|
| 645 |
if (!PyArg_ParseTupleAndKeywords(args, kwds, "|UUi", kwlist, |
| 646 |
&first, &last, |
| 647 |
&self->number)) |
| 648 |
return -1; |
| 649 |
|
| 650 |
if (first) { |
| 651 |
tmp = self->first; |
| 652 |
Py_INCREF(first); |
| 653 |
self->first = first; |
| 654 |
Py_DECREF(tmp); |
| 655 |
} |
| 656 |
if (last) { |
| 657 |
tmp = self->last; |
| 658 |
Py_INCREF(last); |
| 659 |
self->last = last; |
| 660 |
Py_DECREF(tmp); |
| 661 |
} |
| 662 |
return 0; |
| 663 |
} |
| 664 |
|
| 665 |
With these changes, we can assure that the ``first`` and ``last`` members are |
| 666 |
never ``NULL`` so we can remove checks for ``NULL`` values in almost all cases. |
| 667 |
This means that most of the :c:func:`Py_XDECREF` calls can be converted to |
| 668 |
:c:func:`Py_DECREF` calls. The only place we can't change these calls is in |
| 669 |
the ``tp_dealloc`` implementation, where there is the possibility that the |
| 670 |
initialization of these members failed in ``tp_new``. |
| 671 |
|
| 672 |
We also rename the module initialization function and module name in the |
| 673 |
initialization function, as we did before, and we add an extra definition to the |
| 674 |
:file:`setup.py` file. |
| 675 |
|
| 676 |
|
| 677 |
Supporting cyclic garbage collection |
| 678 |
==================================== |
| 679 |
|
| 680 |
Python has a :term:`cyclic garbage collector (GC) <garbage collection>` that |
| 681 |
can identify unneeded objects even when their reference counts are not zero. |
| 682 |
This can happen when objects are involved in cycles. For example, consider: |
| 683 |
|
| 684 |
.. code-block:: pycon |
| 685 |
|
| 686 |
>>> l = [] |
| 687 |
>>> l.append(l) |
| 688 |
>>> del l |
| 689 |
|
| 690 |
In this example, we create a list that contains itself. When we delete it, it |
| 691 |
still has a reference from itself. Its reference count doesn't drop to zero. |
| 692 |
Fortunately, Python's cyclic garbage collector will eventually figure out that |
| 693 |
the list is garbage and free it. |
| 694 |
|
| 695 |
In the second version of the :class:`!Custom` example, we allowed any kind of |
| 696 |
object to be stored in the :attr:`!first` or :attr:`!last` attributes [#]_. |
| 697 |
Besides, in the second and third versions, we allowed subclassing |
| 698 |
:class:`!Custom`, and subclasses may add arbitrary attributes. For any of |
| 699 |
those two reasons, :class:`!Custom` objects can participate in cycles: |
| 700 |
|
| 701 |
.. code-block:: pycon |
| 702 |
|
| 703 |
>>> import custom3 |
| 704 |
>>> class Derived(custom3.Custom): pass |
| 705 |
... |
| 706 |
>>> n = Derived() |
| 707 |
>>> n.some_attribute = n |
| 708 |
|
| 709 |
To allow a :class:`!Custom` instance participating in a reference cycle to |
| 710 |
be properly detected and collected by the cyclic GC, our :class:`!Custom` type |
| 711 |
needs to fill two additional slots and to enable a flag that enables these slots: |
| 712 |
|
| 713 |
.. literalinclude:: ../includes/newtypes/custom4.c |
| 714 |
|
| 715 |
|
| 716 |
First, the traversal method lets the cyclic GC know about subobjects that could |
| 717 |
participate in cycles:: |
| 718 |
|
| 719 |
static int |
| 720 |
Custom_traverse(PyObject *op, visitproc visit, void *arg) |
| 721 |
{ |
| 722 |
CustomObject *self = (CustomObject *) op; |
| 723 |
int vret; |
| 724 |
if (self->first) { |
| 725 |
vret = visit(self->first, arg); |
| 726 |
if (vret != 0) |
| 727 |
return vret; |
| 728 |
} |
| 729 |
if (self->last) { |
| 730 |
vret = visit(self->last, arg); |
| 731 |
if (vret != 0) |
| 732 |
return vret; |
| 733 |
} |
| 734 |
return 0; |
| 735 |
} |
| 736 |
|
| 737 |
For each subobject that can participate in cycles, we need to call the |
| 738 |
:c:func:`!visit` function, which is passed to the traversal method. The |
| 739 |
:c:func:`!visit` function takes as arguments the subobject and the extra argument |
| 740 |
*arg* passed to the traversal method. It returns an integer value that must be |
| 741 |
returned if it is non-zero. |
| 742 |
|
| 743 |
Python provides a :c:func:`Py_VISIT` macro that automates calling visit |
| 744 |
functions. With :c:func:`Py_VISIT`, we can minimize the amount of boilerplate |
| 745 |
in ``Custom_traverse``:: |
| 746 |
|
| 747 |
static int |
| 748 |
Custom_traverse(PyObject *op, visitproc visit, void *arg) |
| 749 |
{ |
| 750 |
CustomObject *self = (CustomObject *) op; |
| 751 |
Py_VISIT(self->first); |
| 752 |
Py_VISIT(self->last); |
| 753 |
return 0; |
| 754 |
} |
| 755 |
|
| 756 |
.. note:: |
| 757 |
The :c:member:`~PyTypeObject.tp_traverse` implementation must name its |
| 758 |
arguments exactly *visit* and *arg* in order to use :c:func:`Py_VISIT`. |
| 759 |
|
| 760 |
Second, we need to provide a method for clearing any subobjects that can |
| 761 |
participate in cycles:: |
| 762 |
|
| 763 |
static int |
| 764 |
Custom_clear(PyObject *op) |
| 765 |
{ |
| 766 |
CustomObject *self = (CustomObject *) op; |
| 767 |
Py_CLEAR(self->first); |
| 768 |
Py_CLEAR(self->last); |
| 769 |
return 0; |
| 770 |
} |
| 771 |
|
| 772 |
Notice the use of the :c:func:`Py_CLEAR` macro. It is the recommended and safe |
| 773 |
way to clear data attributes of arbitrary types while decrementing |
| 774 |
their reference counts. If you were to call :c:func:`Py_XDECREF` instead |
| 775 |
on the attribute before setting it to ``NULL``, there is a possibility |
| 776 |
that the attribute's destructor would call back into code that reads the |
| 777 |
attribute again (*especially* if there is a reference cycle). |
| 778 |
|
| 779 |
.. note:: |
| 780 |
You could emulate :c:func:`Py_CLEAR` by writing:: |
| 781 |
|
| 782 |
PyObject *tmp; |
| 783 |
tmp = self->first; |
| 784 |
self->first = NULL; |
| 785 |
Py_XDECREF(tmp); |
| 786 |
|
| 787 |
Nevertheless, it is much easier and less error-prone to always |
| 788 |
use :c:func:`Py_CLEAR` when deleting an attribute. Don't |
| 789 |
try to micro-optimize at the expense of robustness! |
| 790 |
|
| 791 |
The deallocator ``Custom_dealloc`` may call arbitrary code when clearing |
| 792 |
attributes. It means the circular GC can be triggered inside the function. |
| 793 |
Since the GC assumes reference count is not zero, we need to untrack the object |
| 794 |
from the GC by calling :c:func:`PyObject_GC_UnTrack` before clearing members. |
| 795 |
Here is our reimplemented deallocator using :c:func:`PyObject_GC_UnTrack` |
| 796 |
and ``Custom_clear``:: |
| 797 |
|
| 798 |
static void |
| 799 |
Custom_dealloc(PyObject *op) |
| 800 |
{ |
| 801 |
PyObject_GC_UnTrack(op); |
| 802 |
(void)Custom_clear(op); |
| 803 |
Py_TYPE(op)->tp_free(op); |
| 804 |
} |
| 805 |
|
| 806 |
Finally, we add the :c:macro:`Py_TPFLAGS_HAVE_GC` flag to the class flags:: |
| 807 |
|
| 808 |
.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, |
| 809 |
|
| 810 |
That's pretty much it. If we had written custom :c:member:`~PyTypeObject.tp_alloc` or |
| 811 |
:c:member:`~PyTypeObject.tp_free` handlers, we'd need to modify them for cyclic |
| 812 |
garbage collection. Most extensions will use the versions automatically provided. |
| 813 |
|
| 814 |
|
| 815 |
Subclassing other types |
| 816 |
======================= |
| 817 |
|
| 818 |
It is possible to create new extension types that are derived from existing |
| 819 |
types. It is easiest to inherit from the built in types, since an extension can |
| 820 |
easily use the :c:type:`PyTypeObject` it needs. It can be difficult to share |
| 821 |
these :c:type:`PyTypeObject` structures between extension modules. |
| 822 |
|
| 823 |
In this example we will create a :class:`!SubList` type that inherits from the |
| 824 |
built-in :class:`list` type. The new type will be completely compatible with |
| 825 |
regular lists, but will have an additional :meth:`!increment` method that |
| 826 |
increases an internal counter: |
| 827 |
|
| 828 |
.. code-block:: pycon |
| 829 |
|
| 830 |
>>> import sublist |
| 831 |
>>> s = sublist.SubList(range(3)) |
| 832 |
>>> s.extend(s) |
| 833 |
>>> print(len(s)) |
| 834 |
6 |
| 835 |
>>> print(s.increment()) |
| 836 |
1 |
| 837 |
>>> print(s.increment()) |
| 838 |
2 |
| 839 |
|
| 840 |
.. literalinclude:: ../includes/newtypes/sublist.c |
| 841 |
|
| 842 |
|
| 843 |
As you can see, the source code closely resembles the :class:`!Custom` examples in |
| 844 |
previous sections. We will break down the main differences between them. :: |
| 845 |
|
| 846 |
typedef struct { |
| 847 |
PyListObject list; |
| 848 |
int state; |
| 849 |
} SubListObject; |
| 850 |
|
| 851 |
The primary difference for derived type objects is that the base type's |
| 852 |
object structure must be the first value. The base type will already include |
| 853 |
the :c:func:`PyObject_HEAD` at the beginning of its structure. |
| 854 |
|
| 855 |
When a Python object is a :class:`!SubList` instance, its ``PyObject *`` pointer |
| 856 |
can be safely cast to both ``PyListObject *`` and ``SubListObject *``:: |
| 857 |
|
| 858 |
static int |
| 859 |
SubList_init(PyObject *op, PyObject *args, PyObject *kwds) |
| 860 |
{ |
| 861 |
SubListObject *self = (SubListObject *) op; |
| 862 |
if (PyList_Type.tp_init(op, args, kwds) < 0) |
| 863 |
return -1; |
| 864 |
self->state = 0; |
| 865 |
return 0; |
| 866 |
} |
| 867 |
|
| 868 |
We see above how to call through to the :meth:`~object.__init__` method of the base |
| 869 |
type. |
| 870 |
|
| 871 |
This pattern is important when writing a type with custom |
| 872 |
:c:member:`~PyTypeObject.tp_new` and :c:member:`~PyTypeObject.tp_dealloc` |
| 873 |
members. The :c:member:`~PyTypeObject.tp_new` handler should not actually |
| 874 |
create the memory for the object with its :c:member:`~PyTypeObject.tp_alloc`, |
| 875 |
but let the base class handle it by calling its own :c:member:`~PyTypeObject.tp_new`. |
| 876 |
|
| 877 |
The :c:type:`PyTypeObject` struct supports a :c:member:`~PyTypeObject.tp_base` |
| 878 |
specifying the type's concrete base class. Due to cross-platform compiler |
| 879 |
issues, you can't fill that field directly with a reference to |
| 880 |
:c:type:`PyList_Type`; it should be done in the :c:data:`Py_mod_exec` |
| 881 |
function:: |
| 882 |
|
| 883 |
static int |
| 884 |
sublist_module_exec(PyObject *m) |
| 885 |
{ |
| 886 |
SubListType.tp_base = &PyList_Type; |
| 887 |
if (PyType_Ready(&SubListType) < 0) { |
| 888 |
return -1; |
| 889 |
} |
| 890 |
|
| 891 |
if (PyModule_AddObjectRef(m, "SubList", (PyObject *) &SubListType) < 0) { |
| 892 |
return -1; |
| 893 |
} |
| 894 |
|
| 895 |
return 0; |
| 896 |
} |
| 897 |
|
| 898 |
Before calling :c:func:`PyType_Ready`, the type structure must have the |
| 899 |
:c:member:`~PyTypeObject.tp_base` slot filled in. When we are deriving an |
| 900 |
existing type, it is not necessary to fill out the :c:member:`~PyTypeObject.tp_alloc` |
| 901 |
slot with :c:func:`PyType_GenericNew` -- the allocation function from the base |
| 902 |
type will be inherited. |
| 903 |
|
| 904 |
After that, calling :c:func:`PyType_Ready` and adding the type object to the |
| 905 |
module is the same as with the basic :class:`!Custom` examples. |
| 906 |
|
| 907 |
|
| 908 |
.. rubric:: Footnotes |
| 909 |
|
| 910 |
.. [#] This is true when we know that the object is a basic type, like a string or a |
| 911 |
float. |
| 912 |
|
| 913 |
.. [#] We relied on this in the :c:member:`~PyTypeObject.tp_dealloc` handler |
| 914 |
in this example, because our type doesn't support garbage collection. |
| 915 |
|
| 916 |
.. [#] We now know that the first and last members are strings, so perhaps we |
| 917 |
could be less careful about decrementing their reference counts, however, |
| 918 |
we accept instances of string subclasses. Even though deallocating normal |
| 919 |
strings won't call back into our objects, we can't guarantee that deallocating |
| 920 |
an instance of a string subclass won't call back into our objects. |
| 921 |
|
| 922 |
.. [#] Also, even with our attributes restricted to strings instances, the user |
| 923 |
could pass arbitrary :class:`str` subclasses and therefore still create |
| 924 |
reference cycles. |