| 1 |
Cppy smart pointer |
| 2 |
================== |
| 3 |
|
| 4 |
CPython relies on reference counting to manage object lifetime. A large pitfall |
| 5 |
when writing C-extension is to properly handle increfing and decrefing the |
| 6 |
reference count. Cppy aims at simplifying this process by providing a smart |
| 7 |
pointer class. Before diving into the details of it helps lets start a CPython |
| 8 |
reference counting crash-course. |
| 9 |
|
| 10 |
CPython reference counting crash course |
| 11 |
--------------------------------------- |
| 12 |
|
| 13 |
Each object allocated by Python has a reference count, indicating how many |
| 14 |
times this object is 'used'. When the reference count of an object goes to |
| 15 |
zero, it is de-allocated. Outside of C extension, one does not need to manage |
| 16 |
the reference count manually. |
| 17 |
|
| 18 |
When a function part of Python C-API returns a Python object, it returns a |
| 19 |
pointer to it. At the time at which the function returns, the referenced |
| 20 |
object is live and its reference count is above zero. Depending of the |
| 21 |
function, you do not have the same responsibility with respect to that object |
| 22 |
reference count: |
| 23 |
|
| 24 |
- Owned references: |
| 25 |
Most functions return a ``new`` reference which means that you are responsible |
| 26 |
for decrefing the object reference count when you are done with it (basically |
| 27 |
the function increfed the object reference count before returning). |
| 28 |
In this situation you own a reference. |
| 29 |
|
| 30 |
- Borrowed reference: |
| 31 |
Some functions (``PyList_GetItem``, ``PyTuple_GetItem``, ``PyDict_GetItem``, ...) |
| 32 |
do not incref the object count before returning. In that case, you have only |
| 33 |
a borrowed reference, you are not responsible for decrefing the object |
| 34 |
reference count. |
| 35 |
|
| 36 |
Borrowed references allow to avoid the cost of increfing/decrefing which is |
| 37 |
nice. However since you do not own the reference, if the object referenced is |
| 38 |
removed from its owner (list, tuple for the above two mentioned functions) it |
| 39 |
may just disappear and your reference becomes invalid. This can cause issues. |
| 40 |
If the object should outlive the container, or the time it will spend in the |
| 41 |
container you have to incref it manually. Lets now discuss the convention when |
| 42 |
calling a function. |
| 43 |
|
| 44 |
When calling a function, the caller is expected to own a reference to each of |
| 45 |
the arguments passed to the callee. The callee does not own the references, it |
| 46 |
only borrows them. As a consequence, it should not decref the reference and if |
| 47 |
it needs to store the object, in for example a C structure, it should incref |
| 48 |
it. Note that this does not apply in general to Python container since those |
| 49 |
are manipulated using functions that take care of it. There are however some |
| 50 |
exceptions that steals a reference, meaning that you are not the owner of the |
| 51 |
reference after the call. ``PyList_SetItem``, for example, steal references. |
| 52 |
|
| 53 |
An easy way to get reference count wrong is forgetting to decref some |
| 54 |
intermediate object before leaving a function. This is particularly true if the |
| 55 |
function has some early exit point because an exception should be raised. A |
| 56 |
good practice is to have a single exit point, however it is not always |
| 57 |
possible/practical and even like this it is possible to miss references, this |
| 58 |
is typically where cppy can help. |
| 59 |
|
| 60 |
This is a very brief introduction to reference counting. You can read a bit |
| 61 |
more in the official `Python documentation`_ and in the `Python API`_ |
| 62 |
documentation. |
| 63 |
|
| 64 |
.. _Python documentation: https://docs.python.org/3/c-api/intro.html#objects-types-and-reference-counts |
| 65 |
|
| 66 |
.. _Python API: https://docs.python.org/3/c-api/refcounting.html |
| 67 |
|
| 68 |
Cppy smart pointer class |
| 69 |
------------------------ |
| 70 |
|
| 71 |
Cppy smart pointer (``cppy::ptr``) can be initialized with a pointer to a Python |
| 72 |
object as follows: |
| 73 |
|
| 74 |
.. code:: c++ |
| 75 |
|
| 76 |
cppy::ptr obj_ptr( PyUnicode_FromString("test") ) |
| 77 |
|
| 78 |
When created, the class assume that you own the reference, if it is not the |
| 79 |
case you should incref it first: |
| 80 |
|
| 81 |
.. code:: c++ |
| 82 |
|
| 83 |
PyObject* function( PyObject* obj ) |
| 84 |
{ |
| 85 |
cppy::ptr obj_ptr( cppy::incref( obj ) ); |
| 86 |
cppy::ptr obj_ptr2( obj, true ); |
| 87 |
} |
| 88 |
|
| 89 |
.. note:: |
| 90 |
|
| 91 |
Cppy provides convenient inline function for common reference manipulation: |
| 92 |
- ``cppy::incref``, ``cppy::xincref``, ``cppy::decref``, ``cppy::xdecref`` use the |
| 93 |
the similarly named Python macros and return the input value. |
| 94 |
- ``cppy::clear``, ``cppy::replace`` are similar but return void. |
| 95 |
|
| 96 |
You can also initialize a ``cppy::ptr`` from another ``cppy::ptr`` in which case |
| 97 |
the reference count will always be incremented. |
| 98 |
|
| 99 |
The main advantage provided by ``cppy::ptr`` is that it implements a destructor |
| 100 |
that will be invoked automatically by the c++ runtime when the ``cppy::ptr`` |
| 101 |
goes out of scope. The destructor will decref the reference for you. As a |
| 102 |
consequence you can be sure that your reference you always be decremented when |
| 103 |
you leave the function. |
| 104 |
|
| 105 |
Sometimes, however, that is not what you want, because you want to return the |
| 106 |
reference the ``cppy::ptr`` manage. You can request the ``cppy::ptr`` to give back |
| 107 |
the reference using its ``release`` method. Lets illustrate on a tiny example: |
| 108 |
|
| 109 |
.. code:: c++ |
| 110 |
|
| 111 |
PyObject* function( PyObject* obj ) |
| 112 |
{ |
| 113 |
cppy::ptr repr_ptr( PyObject_Repr( obj ) ); |
| 114 |
return repr_ptr.release(); |
| 115 |
} |
| 116 |
|
| 117 |
Function which are part of Python C-API are not aware of of ``cppy::ptr`` and |
| 118 |
when calling them you need to provide the original ``PyObject*``. To access, you |
| 119 |
simply need to call the ``get`` method of the ``cppy::ptr`` object. |
| 120 |
|
| 121 |
.. code:: c++ |
| 122 |
|
| 123 |
PyObject* function( PyObject* obj ) |
| 124 |
{ |
| 125 |
cppy::ptr l_ptr( PyList_New() ); |
| 126 |
if( PyList_Append( l_ptr.get(), obj ) != 0 ) |
| 127 |
return 0; |
| 128 |
return l_ptr.release(); |
| 129 |
} |
| 130 |
|
| 131 |
Here we see that because we use ``cppy::ptr`` to manage the list, we do not have |
| 132 |
to worry about decrefing the reference if an exception occurs, the runtime |
| 133 |
will do it for us. If no exception occurs, we stop managing the reference and |
| 134 |
we are good. |
| 135 |
|
| 136 |
Using cppy does not eliminate all the pitfalls of writing C-extensions. For |
| 137 |
example if you release too early (for example when passing the object to a |
| 138 |
function that may fail), you can still leak references. However it does |
| 139 |
alleviate some of the complexity. |
| 140 |
|
| 141 |
``cppy::ptr`` methods |
| 142 |
--------------------- |
| 143 |
|
| 144 |
All methods that takes a ``PyObject*`` can also accept a ``cppy::ptr``. |
| 145 |
Most names should be self-explanatory, and apart from the is\_ methods most of |
| 146 |
them rely on the PyObject\_ functions similarly named: |
| 147 |
|
| 148 |
.. code:: c++ |
| 149 |
|
| 150 |
bool is_none() const |
| 151 |
bool is_true() const |
| 152 |
bool is_false() const |
| 153 |
bool is_bool() const |
| 154 |
bool is_int() const |
| 155 |
bool is_float() const |
| 156 |
bool is_list() const |
| 157 |
bool is_dict() const |
| 158 |
bool is_set() const |
| 159 |
bool is_bytes() const |
| 160 |
bool is_str() const |
| 161 |
bool is_unicode() const |
| 162 |
bool is_callable() const |
| 163 |
bool is_iter() const |
| 164 |
bool is_type( PyTypeObject* cls ) const |
| 165 |
int is_truthy() const |
| 166 |
int is_instance( PyObject* cls ) const |
| 167 |
int is_subclass( PyObject* cls ) const |
| 168 |
PyObject* iter() const |
| 169 |
PyObject* next() const |
| 170 |
PyObject* repr() const |
| 171 |
PyObject* str() const |
| 172 |
PyObject* bytes() const |
| 173 |
PyObject* unicode() const |
| 174 |
Py_ssize_t length() const |
| 175 |
PyTypeObject* type() const |
| 176 |
int richcmp( PyObject* other, int opid ) const |
| 177 |
long hash() const |
| 178 |
bool hasattr( PyObject* attr ) const |
| 179 |
bool hasattr( const char* attr ) const |
| 180 |
bool hasattr( const std::string& attr ) const |
| 181 |
PyObject* getattr( PyObject* attr ) const |
| 182 |
PyObject* getattr( const char* attr ) const |
| 183 |
PyObject* getattr( const std::string& attr ) const |
| 184 |
bool setattr( PyObject* attr, PyObject* value ) const |
| 185 |
bool setattr( const char* attr, PyObject* value ) const |
| 186 |
bool setattr( const std::string& attr, PyObject* value ) const |
| 187 |
bool delattr( PyObject* attr ) const |
| 188 |
bool delattr( const char* attr ) const |
| 189 |
bool delattr( const std::string& attr ) const |
| 190 |
PyObject* getitem( PyObject* key ) const |
| 191 |
bool setitem( PyObject* key, PyObject* value ) const |
| 192 |
bool delitem( PyObject* key ) |
| 193 |
PyObject* call( PyObject* args, PyObject* kwargs = 0 ) const |