Debian Codegraph @

Doc/extending/newtypes.rst

raw
1 .. highlight:: c
2
3 .. _new-types-topics:
4
5 *****************************************
6 Defining Extension Types: Assorted Topics
7 *****************************************
8
9 .. _dnt-type-methods:
10
11 This section aims to give a quick fly-by on the various type methods you can
12 implement and what they do.
13
14 Here is the definition of :c:type:`PyTypeObject`, with some fields only used in
15 :ref:`debug builds <debug-build>` omitted:
16
17 .. literalinclude:: ../includes/typestruct.h
18
19
20 Now that's a *lot* of methods. Don't worry too much though -- if you have
21 a type you want to define, the chances are very good that you will only
22 implement a handful of these.
23
24 As you probably expect by now, we're going to go over this and give more
25 information about the various handlers. We won't go in the order they are
26 defined in the structure, because there is a lot of historical baggage that
27 impacts the ordering of the fields. It's often easiest to find an example
28 that includes the fields you need and then change the values to suit your new
29 type. ::
30
31 const char *tp_name; /* For printing */
32
33 The name of the type -- as mentioned in the previous chapter, this will appear in
34 various places, almost entirely for diagnostic purposes. Try to choose something
35 that will be helpful in such a situation! ::
36
37 Py_ssize_t tp_basicsize, tp_itemsize; /* For allocation */
38
39 These fields tell the runtime how much memory to allocate when new objects of
40 this type are created. Python has some built-in support for variable length
41 structures (think: strings, tuples) which is where the :c:member:`~PyTypeObject.tp_itemsize` field
42 comes in. This will be dealt with later. ::
43
44 const char *tp_doc;
45
46 Here you can put a string (or its address) that you want returned when the
47 Python script references ``obj.__doc__`` to retrieve the doc string.
48
49 Now we come to the basic type methods -- the ones most extension types will
50 implement.
51
52
53 Finalization and De-allocation
54 ------------------------------
55
56 .. index::
57 single: object; deallocation
58 single: deallocation, object
59 single: object; finalization
60 single: finalization, of objects
61
62 ::
63
64 destructor tp_dealloc;
65
66 This function is called when the reference count of the instance of your type is
67 reduced to zero and the Python interpreter wants to reclaim it. If your type
68 has memory to free or other clean-up to perform, you can put it here. The
69 object itself needs to be freed here as well. Here is an example of this
70 function::
71
72 static void
73 newdatatype_dealloc(PyObject *op)
74 {
75 newdatatypeobject *self = (newdatatypeobject *) op;
76 free(self->obj_UnderlyingDatatypePtr);
77 Py_TYPE(self)->tp_free(self);
78 }
79
80 If your type supports garbage collection, the destructor should call
81 :c:func:`PyObject_GC_UnTrack` before clearing any member fields::
82
83 static void
84 newdatatype_dealloc(PyObject *op)
85 {
86 newdatatypeobject *self = (newdatatypeobject *) op;
87 PyObject_GC_UnTrack(op);
88 Py_CLEAR(self->other_obj);
89 ...
90 Py_TYPE(self)->tp_free(self);
91 }
92
93 .. index::
94 single: PyErr_Fetch (C function)
95 single: PyErr_Restore (C function)
96
97 One important requirement of the deallocator function is that it leaves any
98 pending exceptions alone. This is important since deallocators are frequently
99 called as the interpreter unwinds the Python stack; when the stack is unwound
100 due to an exception (rather than normal returns), nothing is done to protect the
101 deallocators from seeing that an exception has already been set. Any actions
102 which a deallocator performs which may cause additional Python code to be
103 executed may detect that an exception has been set. This can lead to misleading
104 errors from the interpreter. The proper way to protect against this is to save
105 a pending exception before performing the unsafe action, and restoring it when
106 done. This can be done using the :c:func:`PyErr_Fetch` and
107 :c:func:`PyErr_Restore` functions::
108
109 static void
110 my_dealloc(PyObject *obj)
111 {
112 MyObject *self = (MyObject *) obj;
113 PyObject *cbresult;
114
115 if (self->my_callback != NULL) {
116 PyObject *err_type, *err_value, *err_traceback;
117
118 /* This saves the current exception state */
119 PyErr_Fetch(&err_type, &err_value, &err_traceback);
120
121 cbresult = PyObject_CallNoArgs(self->my_callback);
122 if (cbresult == NULL) {
123 PyErr_WriteUnraisable(self->my_callback);
124 }
125 else {
126 Py_DECREF(cbresult);
127 }
128
129 /* This restores the saved exception state */
130 PyErr_Restore(err_type, err_value, err_traceback);
131
132 Py_DECREF(self->my_callback);
133 }
134 Py_TYPE(self)->tp_free(self);
135 }
136
137 .. note::
138 There are limitations to what you can safely do in a deallocator function.
139 First, if your type supports garbage collection (using :c:member:`~PyTypeObject.tp_traverse`
140 and/or :c:member:`~PyTypeObject.tp_clear`), some of the object's members can have been
141 cleared or finalized by the time :c:member:`~PyTypeObject.tp_dealloc` is called. Second, in
142 :c:member:`~PyTypeObject.tp_dealloc`, your object is in an unstable state: its reference
143 count is equal to zero. Any call to a non-trivial object or API (as in the
144 example above) might end up calling :c:member:`~PyTypeObject.tp_dealloc` again, causing a
145 double free and a crash.
146
147 Starting with Python 3.4, it is recommended not to put any complex
148 finalization code in :c:member:`~PyTypeObject.tp_dealloc`, and instead use the new
149 :c:member:`~PyTypeObject.tp_finalize` type method.
150
151 .. seealso::
152 :pep:`442` explains the new finalization scheme.
153
154 .. index::
155 single: string; object representation
156 pair: built-in function; repr
157
158 Object Presentation
159 -------------------
160
161 In Python, there are two ways to generate a textual representation of an object:
162 the :func:`repr` function, and the :func:`str` function. (The :func:`print`
163 function just calls :func:`str`.) These handlers are both optional.
164
165 ::
166
167 reprfunc tp_repr;
168 reprfunc tp_str;
169
170 The :c:member:`~PyTypeObject.tp_repr` handler should return a string object containing a
171 representation of the instance for which it is called. Here is a simple
172 example::
173
174 static PyObject *
175 newdatatype_repr(PyObject *op)
176 {
177 newdatatypeobject *self = (newdatatypeobject *) op;
178 return PyUnicode_FromFormat("Repr-ified_newdatatype{{size:%d}}",
179 self->obj_UnderlyingDatatypePtr->size);
180 }
181
182 If no :c:member:`~PyTypeObject.tp_repr` handler is specified, the interpreter will supply a
183 representation that uses the type's :c:member:`~PyTypeObject.tp_name` and a uniquely identifying
184 value for the object.
185
186 The :c:member:`~PyTypeObject.tp_str` handler is to :func:`str` what the :c:member:`~PyTypeObject.tp_repr` handler
187 described above is to :func:`repr`; that is, it is called when Python code calls
188 :func:`str` on an instance of your object. Its implementation is very similar
189 to the :c:member:`~PyTypeObject.tp_repr` function, but the resulting string is intended for human
190 consumption. If :c:member:`~PyTypeObject.tp_str` is not specified, the :c:member:`~PyTypeObject.tp_repr` handler is
191 used instead.
192
193 Here is a simple example::
194
195 static PyObject *
196 newdatatype_str(PyObject *op)
197 {
198 newdatatypeobject *self = (newdatatypeobject *) op;
199 return PyUnicode_FromFormat("Stringified_newdatatype{{size:%d}}",
200 self->obj_UnderlyingDatatypePtr->size);
201 }
202
203
204
205 Attribute Management
206 --------------------
207
208 For every object which can support attributes, the corresponding type must
209 provide the functions that control how the attributes are resolved. There needs
210 to be a function which can retrieve attributes (if any are defined), and another
211 to set attributes (if setting attributes is allowed). Removing an attribute is
212 a special case, for which the new value passed to the handler is ``NULL``.
213
214 Python supports two pairs of attribute handlers; a type that supports attributes
215 only needs to implement the functions for one pair. The difference is that one
216 pair takes the name of the attribute as a :c:expr:`char\*`, while the other
217 accepts a :c:expr:`PyObject*`. Each type can use whichever pair makes more
218 sense for the implementation's convenience. ::
219
220 getattrfunc tp_getattr; /* char * version */
221 setattrfunc tp_setattr;
222 /* ... */
223 getattrofunc tp_getattro; /* PyObject * version */
224 setattrofunc tp_setattro;
225
226 If accessing attributes of an object is always a simple operation (this will be
227 explained shortly), there are generic implementations which can be used to
228 provide the :c:expr:`PyObject*` version of the attribute management functions.
229 The actual need for type-specific attribute handlers almost completely
230 disappeared starting with Python 2.2, though there are many examples which have
231 not been updated to use some of the new generic mechanism that is available.
232
233
234 .. _generic-attribute-management:
235
236 Generic Attribute Management
237 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
238
239 Most extension types only use *simple* attributes. So, what makes the
240 attributes simple? There are only a couple of conditions that must be met:
241
242 #. The name of the attributes must be known when :c:func:`PyType_Ready` is
243 called.
244
245 #. No special processing is needed to record that an attribute was looked up or
246 set, nor do actions need to be taken based on the value.
247
248 Note that this list does not place any restrictions on the values of the
249 attributes, when the values are computed, or how relevant data is stored.
250
251 When :c:func:`PyType_Ready` is called, it uses three tables referenced by the
252 type object to create :term:`descriptor`\s which are placed in the dictionary of the
253 type object. Each descriptor controls access to one attribute of the instance
254 object. Each of the tables is optional; if all three are ``NULL``, instances of
255 the type will only have attributes that are inherited from their base type, and
256 should leave the :c:member:`~PyTypeObject.tp_getattro` and :c:member:`~PyTypeObject.tp_setattro` fields ``NULL`` as
257 well, allowing the base type to handle attributes.
258
259 The tables are declared as three fields of the type object::
260
261 struct PyMethodDef *tp_methods;
262 struct PyMemberDef *tp_members;
263 struct PyGetSetDef *tp_getset;
264
265 If :c:member:`~PyTypeObject.tp_methods` is not ``NULL``, it must refer to an array of
266 :c:type:`PyMethodDef` structures. Each entry in the table is an instance of this
267 structure::
268
269 typedef struct PyMethodDef {
270 const char *ml_name; /* method name */
271 PyCFunction ml_meth; /* implementation function */
272 int ml_flags; /* flags */
273 const char *ml_doc; /* docstring */
274 } PyMethodDef;
275
276 One entry should be defined for each method provided by the type; no entries are
277 needed for methods inherited from a base type. One additional entry is needed
278 at the end; it is a sentinel that marks the end of the array. The
279 :c:member:`~PyMethodDef.ml_name` field of the sentinel must be ``NULL``.
280
281 The second table is used to define attributes which map directly to data stored
282 in the instance. A variety of primitive C types are supported, and access may
283 be read-only or read-write. The structures in the table are defined as::
284
285 typedef struct PyMemberDef {
286 const char *name;
287 int type;
288 int offset;
289 int flags;
290 const char *doc;
291 } PyMemberDef;
292
293 For each entry in the table, a :term:`descriptor` will be constructed and added to the
294 type which will be able to extract a value from the instance structure. The
295 :c:member:`~PyMemberDef.type` field should contain a type code like :c:macro:`Py_T_INT` or
296 :c:macro:`Py_T_DOUBLE`; the value will be used to determine how to
297 convert Python values to and from C values. The :c:member:`~PyMemberDef.flags` field is used to
298 store flags which control how the attribute can be accessed: you can set it to
299 :c:macro:`Py_READONLY` to prevent Python code from setting it.
300
301 An interesting advantage of using the :c:member:`~PyTypeObject.tp_members` table to build
302 descriptors that are used at runtime is that any attribute defined this way can
303 have an associated doc string simply by providing the text in the table. An
304 application can use the introspection API to retrieve the descriptor from the
305 class object, and get the doc string using its :attr:`~type.__doc__` attribute.
306
307 As with the :c:member:`~PyTypeObject.tp_methods` table, a sentinel entry with a :c:member:`~PyMethodDef.ml_name` value
308 of ``NULL`` is required.
309
310 .. XXX Descriptors need to be explained in more detail somewhere, but not here.
311
312 Descriptor objects have two handler functions which correspond to the
313 \member{tp_getattro} and \member{tp_setattro} handlers. The
314 \method{__get__()} handler is a function which is passed the descriptor,
315 instance, and type objects, and returns the value of the attribute, or it
316 returns \NULL{} and sets an exception. The \method{__set__()} handler is
317 passed the descriptor, instance, type, and new value;
318
319
320 Type-specific Attribute Management
321 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
322
323 For simplicity, only the :c:expr:`char\*` version will be demonstrated here; the
324 type of the name parameter is the only difference between the :c:expr:`char\*`
325 and :c:expr:`PyObject*` flavors of the interface. This example effectively does
326 the same thing as the generic example above, but does not use the generic
327 support added in Python 2.2. It explains how the handler functions are
328 called, so that if you do need to extend their functionality, you'll understand
329 what needs to be done.
330
331 The :c:member:`~PyTypeObject.tp_getattr` handler is called when the object requires an attribute
332 look-up. It is called in the same situations where the :meth:`~object.__getattr__`
333 method of a class would be called.
334
335 Here is an example::
336
337 static PyObject *
338 newdatatype_getattr(PyObject *op, char *name)
339 {
340 newdatatypeobject *self = (newdatatypeobject *) op;
341 if (strcmp(name, "data") == 0) {
342 return PyLong_FromLong(self->data);
343 }
344
345 PyErr_Format(PyExc_AttributeError,
346 "'%.100s' object has no attribute '%.400s'",
347 Py_TYPE(self)->tp_name, name);
348 return NULL;
349 }
350
351 The :c:member:`~PyTypeObject.tp_setattr` handler is called when the :meth:`~object.__setattr__` or
352 :meth:`~object.__delattr__` method of a class instance would be called. When an
353 attribute should be deleted, the third parameter will be ``NULL``. Here is an
354 example that simply raises an exception; if this were really all you wanted, the
355 :c:member:`~PyTypeObject.tp_setattr` handler should be set to ``NULL``. ::
356
357 static int
358 newdatatype_setattr(PyObject *op, char *name, PyObject *v)
359 {
360 PyErr_Format(PyExc_RuntimeError, "Read-only attribute: %s", name);
361 return -1;
362 }
363
364 Object Comparison
365 -----------------
366
367 ::
368
369 richcmpfunc tp_richcompare;
370
371 The :c:member:`~PyTypeObject.tp_richcompare` handler is called when comparisons are needed. It is
372 analogous to the :ref:`rich comparison methods <richcmpfuncs>`, like
373 :meth:`!__lt__`, and also called by :c:func:`PyObject_RichCompare` and
374 :c:func:`PyObject_RichCompareBool`.
375
376 This function is called with two Python objects and the operator as arguments,
377 where the operator is one of ``Py_EQ``, ``Py_NE``, ``Py_LE``, ``Py_GE``,
378 ``Py_LT`` or ``Py_GT``. It should compare the two objects with respect to the
379 specified operator and return ``Py_True`` or ``Py_False`` if the comparison is
380 successful, ``Py_NotImplemented`` to indicate that comparison is not
381 implemented and the other object's comparison method should be tried, or ``NULL``
382 if an exception was set.
383
384 Here is a sample implementation, for a datatype that is considered equal if the
385 size of an internal pointer is equal::
386
387 static PyObject *
388 newdatatype_richcmp(PyObject *lhs, PyObject *rhs, int op)
389 {
390 newdatatypeobject *obj1 = (newdatatypeobject *) lhs;
391 newdatatypeobject *obj2 = (newdatatypeobject *) rhs;
392 PyObject *result;
393 int c, size1, size2;
394
395 /* code to make sure that both arguments are of type
396 newdatatype omitted */
397
398 size1 = obj1->obj_UnderlyingDatatypePtr->size;
399 size2 = obj2->obj_UnderlyingDatatypePtr->size;
400
401 switch (op) {
402 case Py_LT: c = size1 < size2; break;
403 case Py_LE: c = size1 <= size2; break;
404 case Py_EQ: c = size1 == size2; break;
405 case Py_NE: c = size1 != size2; break;
406 case Py_GT: c = size1 > size2; break;
407 case Py_GE: c = size1 >= size2; break;
408 }
409 result = c ? Py_True : Py_False;
410 return Py_NewRef(result);
411 }
412
413
414 Abstract Protocol Support
415 -------------------------
416
417 Python supports a variety of *abstract* 'protocols;' the specific interfaces
418 provided to use these interfaces are documented in :ref:`abstract`.
419
420
421 A number of these abstract interfaces were defined early in the development of
422 the Python implementation. In particular, the number, mapping, and sequence
423 protocols have been part of Python since the beginning. Other protocols have
424 been added over time. For protocols which depend on several handler routines
425 from the type implementation, the older protocols have been defined as optional
426 blocks of handlers referenced by the type object. For newer protocols there are
427 additional slots in the main type object, with a flag bit being set to indicate
428 that the slots are present and should be checked by the interpreter. (The flag
429 bit does not indicate that the slot values are non-``NULL``. The flag may be set
430 to indicate the presence of a slot, but a slot may still be unfilled.) ::
431
432 PyNumberMethods *tp_as_number;
433 PySequenceMethods *tp_as_sequence;
434 PyMappingMethods *tp_as_mapping;
435
436 If you wish your object to be able to act like a number, a sequence, or a
437 mapping object, then you place the address of a structure that implements the C
438 type :c:type:`PyNumberMethods`, :c:type:`PySequenceMethods`, or
439 :c:type:`PyMappingMethods`, respectively. It is up to you to fill in this
440 structure with appropriate values. You can find examples of the use of each of
441 these in the :file:`Objects` directory of the Python source distribution. ::
442
443 hashfunc tp_hash;
444
445 This function, if you choose to provide it, should return a hash number for an
446 instance of your data type. Here is a simple example::
447
448 static Py_hash_t
449 newdatatype_hash(PyObject *op)
450 {
451 newdatatypeobject *self = (newdatatypeobject *) op;
452 Py_hash_t result;
453 result = self->some_size + 32767 * self->some_number;
454 if (result == -1) {
455 result = -2;
456 }
457 return result;
458 }
459
460 :c:type:`Py_hash_t` is a signed integer type with a platform-varying width.
461 Returning ``-1`` from :c:member:`~PyTypeObject.tp_hash` indicates an error,
462 which is why you should be careful to avoid returning it when hash computation
463 is successful, as seen above.
464
465 ::
466
467 ternaryfunc tp_call;
468
469 This function is called when an instance of your data type is "called", for
470 example, if ``obj1`` is an instance of your data type and the Python script
471 contains ``obj1('hello')``, the :c:member:`~PyTypeObject.tp_call` handler is invoked.
472
473 This function takes three arguments:
474
475 #. *self* is the instance of the data type which is the subject of the call.
476 If the call is ``obj1('hello')``, then *self* is ``obj1``.
477
478 #. *args* is a tuple containing the arguments to the call. You can use
479 :c:func:`PyArg_ParseTuple` to extract the arguments.
480
481 #. *kwds* is a dictionary of keyword arguments that were passed. If this is
482 non-``NULL`` and you support keyword arguments, use
483 :c:func:`PyArg_ParseTupleAndKeywords` to extract the arguments. If you
484 do not want to support keyword arguments and this is non-``NULL``, raise a
485 :exc:`TypeError` with a message saying that keyword arguments are not supported.
486
487 Here is a toy ``tp_call`` implementation::
488
489 static PyObject *
490 newdatatype_call(PyObject *op, PyObject *args, PyObject *kwds)
491 {
492 newdatatypeobject *self = (newdatatypeobject *) op;
493 PyObject *result;
494 const char *arg1;
495 const char *arg2;
496 const char *arg3;
497
498 if (!PyArg_ParseTuple(args, "sss:call", &arg1, &arg2, &arg3)) {
499 return NULL;
500 }
501 result = PyUnicode_FromFormat(
502 "Returning -- value: [%d] arg1: [%s] arg2: [%s] arg3: [%s]\n",
503 self->obj_UnderlyingDatatypePtr->size,
504 arg1, arg2, arg3);
505 return result;
506 }
507
508 ::
509
510 /* Iterators */
511 getiterfunc tp_iter;
512 iternextfunc tp_iternext;
513
514 These functions provide support for the iterator protocol. Both handlers
515 take exactly one parameter, the instance for which they are being called,
516 and return a new reference. In the case of an error, they should set an
517 exception and return ``NULL``. :c:member:`~PyTypeObject.tp_iter` corresponds
518 to the Python :meth:`~object.__iter__` method, while :c:member:`~PyTypeObject.tp_iternext`
519 corresponds to the Python :meth:`~iterator.__next__` method.
520
521 Any :term:`iterable` object must implement the :c:member:`~PyTypeObject.tp_iter`
522 handler, which must return an :term:`iterator` object. Here the same guidelines
523 apply as for Python classes:
524
525 * For collections (such as lists and tuples) which can support multiple
526 independent iterators, a new iterator should be created and returned by
527 each call to :c:member:`~PyTypeObject.tp_iter`.
528 * Objects which can only be iterated over once (usually due to side effects of
529 iteration, such as file objects) can implement :c:member:`~PyTypeObject.tp_iter`
530 by returning a new reference to themselves -- and should also therefore
531 implement the :c:member:`~PyTypeObject.tp_iternext` handler.
532
533 Any :term:`iterator` object should implement both :c:member:`~PyTypeObject.tp_iter`
534 and :c:member:`~PyTypeObject.tp_iternext`. An iterator's
535 :c:member:`~PyTypeObject.tp_iter` handler should return a new reference
536 to the iterator. Its :c:member:`~PyTypeObject.tp_iternext` handler should
537 return a new reference to the next object in the iteration, if there is one.
538 If the iteration has reached the end, :c:member:`~PyTypeObject.tp_iternext`
539 may return ``NULL`` without setting an exception, or it may set
540 :exc:`StopIteration` *in addition* to returning ``NULL``; avoiding
541 the exception can yield slightly better performance. If an actual error
542 occurs, :c:member:`~PyTypeObject.tp_iternext` should always set an exception
543 and return ``NULL``.
544
545
546 .. _weakref-support:
547
548 Weak Reference Support
549 ----------------------
550
551 One of the goals of Python's weak reference implementation is to allow any type
552 to participate in the weak reference mechanism without incurring the overhead on
553 performance-critical objects (such as numbers).
554
555 .. seealso::
556 Documentation for the :mod:`weakref` module.
557
558 For an object to be weakly referenceable, the extension type must set the
559 ``Py_TPFLAGS_MANAGED_WEAKREF`` bit of the :c:member:`~PyTypeObject.tp_flags`
560 field. The legacy :c:member:`~PyTypeObject.tp_weaklistoffset` field should
561 be left as zero.
562
563 Concretely, here is how the statically declared type object would look::
564
565 static PyTypeObject TrivialType = {
566 PyVarObject_HEAD_INIT(NULL, 0)
567 /* ... other members omitted for brevity ... */
568 .tp_flags = Py_TPFLAGS_MANAGED_WEAKREF | ...,
569 };
570
571
572 The only further addition is that ``tp_dealloc`` needs to clear any weak
573 references (by calling :c:func:`PyObject_ClearWeakRefs`)::
574
575 static void
576 Trivial_dealloc(PyObject *op)
577 {
578 /* Clear weakrefs first before calling any destructors */
579 PyObject_ClearWeakRefs(op);
580 /* ... remainder of destruction code omitted for brevity ... */
581 Py_TYPE(op)->tp_free(op);
582 }
583
584
585 More Suggestions
586 ----------------
587
588 In order to learn how to implement any specific method for your new data type,
589 get the :term:`CPython` source code. Go to the :file:`Objects` directory,
590 then search the C source files for ``tp_`` plus the function you want
591 (for example, ``tp_richcompare``). You will find examples of the function
592 you want to implement.
593
594 When you need to verify that an object is a concrete instance of the type you
595 are implementing, use the :c:func:`PyObject_TypeCheck` function. A sample of
596 its use might be something like the following::
597
598 if (!PyObject_TypeCheck(some_object, &MyType)) {
599 PyErr_SetString(PyExc_TypeError, "arg #1 not a mything");
600 return NULL;
601 }
602
603 .. seealso::
604 Download CPython source releases.
605 https://www.python.org/downloads/source/
606
607 The CPython project on GitHub, where the CPython source code is developed.
608 https://github.com/python/cpython