| 1 |
.. highlight:: c |
| 2 |
|
| 3 |
|
| 4 |
.. _embedding: |
| 5 |
|
| 6 |
*************************************** |
| 7 |
Embedding Python in Another Application |
| 8 |
*************************************** |
| 9 |
|
| 10 |
The previous chapters discussed how to extend Python, that is, how to extend the |
| 11 |
functionality of Python by attaching a library of C functions to it. It is also |
| 12 |
possible to do it the other way around: enrich your C/C++ application by |
| 13 |
embedding Python in it. Embedding provides your application with the ability to |
| 14 |
implement some of the functionality of your application in Python rather than C |
| 15 |
or C++. This can be used for many purposes; one example would be to allow users |
| 16 |
to tailor the application to their needs by writing some scripts in Python. You |
| 17 |
can also use it yourself if some of the functionality can be written in Python |
| 18 |
more easily. |
| 19 |
|
| 20 |
Embedding Python is similar to extending it, but not quite. The difference is |
| 21 |
that when you extend Python, the main program of the application is still the |
| 22 |
Python interpreter, while if you embed Python, the main program may have nothing |
| 23 |
to do with Python --- instead, some parts of the application occasionally call |
| 24 |
the Python interpreter to run some Python code. |
| 25 |
|
| 26 |
So if you are embedding Python, you are providing your own main program. One of |
| 27 |
the things this main program has to do is initialize the Python interpreter. At |
| 28 |
the very least, you have to call the function :c:func:`Py_Initialize`. There are |
| 29 |
optional calls to pass command line arguments to Python. Then later you can |
| 30 |
call the interpreter from any part of the application. |
| 31 |
|
| 32 |
There are several different ways to call the interpreter: you can pass a string |
| 33 |
containing Python statements to :c:func:`PyRun_SimpleString`, or you can pass a |
| 34 |
stdio file pointer and a file name (for identification in error messages only) |
| 35 |
to :c:func:`PyRun_SimpleFile`. You can also call the lower-level operations |
| 36 |
described in the previous chapters to construct and use Python objects. |
| 37 |
|
| 38 |
|
| 39 |
.. seealso:: |
| 40 |
|
| 41 |
:ref:`c-api-index` |
| 42 |
The details of Python's C interface are given in this manual. A great deal of |
| 43 |
necessary information can be found here. |
| 44 |
|
| 45 |
|
| 46 |
.. _high-level-embedding: |
| 47 |
|
| 48 |
Very High Level Embedding |
| 49 |
========================= |
| 50 |
|
| 51 |
The simplest form of embedding Python is the use of the very high level |
| 52 |
interface. This interface is intended to execute a Python script without needing |
| 53 |
to interact with the application directly. This can for example be used to |
| 54 |
perform some operation on a file. :: |
| 55 |
|
| 56 |
#define PY_SSIZE_T_CLEAN |
| 57 |
#include <Python.h> |
| 58 |
|
| 59 |
int |
| 60 |
main(int argc, char *argv[]) |
| 61 |
{ |
| 62 |
PyStatus status; |
| 63 |
PyConfig config; |
| 64 |
PyConfig_InitPythonConfig(&config); |
| 65 |
|
| 66 |
/* optional but recommended */ |
| 67 |
status = PyConfig_SetBytesString(&config, &config.program_name, argv[0]); |
| 68 |
if (PyStatus_Exception(status)) { |
| 69 |
goto exception; |
| 70 |
} |
| 71 |
|
| 72 |
status = Py_InitializeFromConfig(&config); |
| 73 |
if (PyStatus_Exception(status)) { |
| 74 |
goto exception; |
| 75 |
} |
| 76 |
PyConfig_Clear(&config); |
| 77 |
|
| 78 |
PyRun_SimpleString("from time import time,ctime\n" |
| 79 |
"print('Today is', ctime(time()))\n"); |
| 80 |
if (Py_FinalizeEx() < 0) { |
| 81 |
exit(120); |
| 82 |
} |
| 83 |
return 0; |
| 84 |
|
| 85 |
exception: |
| 86 |
PyConfig_Clear(&config); |
| 87 |
Py_ExitStatusException(status); |
| 88 |
} |
| 89 |
|
| 90 |
.. note:: |
| 91 |
|
| 92 |
``#define PY_SSIZE_T_CLEAN`` was used to indicate that ``Py_ssize_t`` should be |
| 93 |
used in some APIs instead of ``int``. |
| 94 |
It is not necessary since Python 3.13, but we keep it here for backward compatibility. |
| 95 |
See :ref:`arg-parsing-string-and-buffers` for a description of this macro. |
| 96 |
|
| 97 |
Setting :c:member:`PyConfig.program_name` should be called before |
| 98 |
:c:func:`Py_InitializeFromConfig` to inform the interpreter about paths to Python run-time |
| 99 |
libraries. Next, the Python interpreter is initialized with |
| 100 |
:c:func:`Py_Initialize`, followed by the execution of a hard-coded Python script |
| 101 |
that prints the date and time. Afterwards, the :c:func:`Py_FinalizeEx` call shuts |
| 102 |
the interpreter down, followed by the end of the program. In a real program, |
| 103 |
you may want to get the Python script from another source, perhaps a text-editor |
| 104 |
routine, a file, or a database. Getting the Python code from a file can better |
| 105 |
be done by using the :c:func:`PyRun_SimpleFile` function, which saves you the |
| 106 |
trouble of allocating memory space and loading the file contents. |
| 107 |
|
| 108 |
|
| 109 |
.. _lower-level-embedding: |
| 110 |
|
| 111 |
Beyond Very High Level Embedding: An overview |
| 112 |
============================================= |
| 113 |
|
| 114 |
The high level interface gives you the ability to execute arbitrary pieces of |
| 115 |
Python code from your application, but exchanging data values is quite |
| 116 |
cumbersome to say the least. If you want that, you should use lower level calls. |
| 117 |
At the cost of having to write more C code, you can achieve almost anything. |
| 118 |
|
| 119 |
It should be noted that extending Python and embedding Python is quite the same |
| 120 |
activity, despite the different intent. Most topics discussed in the previous |
| 121 |
chapters are still valid. To show this, consider what the extension code from |
| 122 |
Python to C really does: |
| 123 |
|
| 124 |
#. Convert data values from Python to C, |
| 125 |
|
| 126 |
#. Perform a function call to a C routine using the converted values, and |
| 127 |
|
| 128 |
#. Convert the data values from the call from C to Python. |
| 129 |
|
| 130 |
When embedding Python, the interface code does: |
| 131 |
|
| 132 |
#. Convert data values from C to Python, |
| 133 |
|
| 134 |
#. Perform a function call to a Python interface routine using the converted |
| 135 |
values, and |
| 136 |
|
| 137 |
#. Convert the data values from the call from Python to C. |
| 138 |
|
| 139 |
As you can see, the data conversion steps are simply swapped to accommodate the |
| 140 |
different direction of the cross-language transfer. The only difference is the |
| 141 |
routine that you call between both data conversions. When extending, you call a |
| 142 |
C routine, when embedding, you call a Python routine. |
| 143 |
|
| 144 |
This chapter will not discuss how to convert data from Python to C and vice |
| 145 |
versa. Also, proper use of references and dealing with errors is assumed to be |
| 146 |
understood. Since these aspects do not differ from extending the interpreter, |
| 147 |
you can refer to earlier chapters for the required information. |
| 148 |
|
| 149 |
|
| 150 |
.. _pure-embedding: |
| 151 |
|
| 152 |
Pure Embedding |
| 153 |
============== |
| 154 |
|
| 155 |
The first program aims to execute a function in a Python script. Like in the |
| 156 |
section about the very high level interface, the Python interpreter does not |
| 157 |
directly interact with the application (but that will change in the next |
| 158 |
section). |
| 159 |
|
| 160 |
The code to run a function defined in a Python script is: |
| 161 |
|
| 162 |
.. literalinclude:: ../includes/run-func.c |
| 163 |
|
| 164 |
|
| 165 |
This code loads a Python script using ``argv[1]``, and calls the function named |
| 166 |
in ``argv[2]``. Its integer arguments are the other values of the ``argv`` |
| 167 |
array. If you :ref:`compile and link <compiling>` this program (let's call |
| 168 |
the finished executable :program:`call`), and use it to execute a Python |
| 169 |
script, such as: |
| 170 |
|
| 171 |
.. code-block:: python |
| 172 |
|
| 173 |
def multiply(a,b): |
| 174 |
print("Will compute", a, "times", b) |
| 175 |
c = 0 |
| 176 |
for i in range(0, a): |
| 177 |
c = c + b |
| 178 |
return c |
| 179 |
|
| 180 |
then the result should be: |
| 181 |
|
| 182 |
.. code-block:: shell-session |
| 183 |
|
| 184 |
$ call multiply multiply 3 2 |
| 185 |
Will compute 3 times 2 |
| 186 |
Result of call: 6 |
| 187 |
|
| 188 |
Although the program is quite large for its functionality, most of the code is |
| 189 |
for data conversion between Python and C, and for error reporting. The |
| 190 |
interesting part with respect to embedding Python starts with :: |
| 191 |
|
| 192 |
Py_Initialize(); |
| 193 |
pName = PyUnicode_DecodeFSDefault(argv[1]); |
| 194 |
/* Error checking of pName left out */ |
| 195 |
pModule = PyImport_Import(pName); |
| 196 |
|
| 197 |
After initializing the interpreter, the script is loaded using |
| 198 |
:c:func:`PyImport_Import`. This routine needs a Python string as its argument, |
| 199 |
which is constructed using the :c:func:`PyUnicode_DecodeFSDefault` data |
| 200 |
conversion routine. :: |
| 201 |
|
| 202 |
pFunc = PyObject_GetAttrString(pModule, argv[2]); |
| 203 |
/* pFunc is a new reference */ |
| 204 |
|
| 205 |
if (pFunc && PyCallable_Check(pFunc)) { |
| 206 |
... |
| 207 |
} |
| 208 |
Py_XDECREF(pFunc); |
| 209 |
|
| 210 |
Once the script is loaded, the name we're looking for is retrieved using |
| 211 |
:c:func:`PyObject_GetAttrString`. If the name exists, and the object returned is |
| 212 |
callable, you can safely assume that it is a function. The program then |
| 213 |
proceeds by constructing a tuple of arguments as normal. The call to the Python |
| 214 |
function is then made with:: |
| 215 |
|
| 216 |
pValue = PyObject_CallObject(pFunc, pArgs); |
| 217 |
|
| 218 |
Upon return of the function, ``pValue`` is either ``NULL`` or it contains a |
| 219 |
reference to the return value of the function. Be sure to release the reference |
| 220 |
after examining the value. |
| 221 |
|
| 222 |
|
| 223 |
.. _extending-with-embedding: |
| 224 |
|
| 225 |
Extending Embedded Python |
| 226 |
========================= |
| 227 |
|
| 228 |
Until now, the embedded Python interpreter had no access to functionality from |
| 229 |
the application itself. The Python API allows this by extending the embedded |
| 230 |
interpreter. That is, the embedded interpreter gets extended with routines |
| 231 |
provided by the application. While it sounds complex, it is not so bad. Simply |
| 232 |
forget for a while that the application starts the Python interpreter. Instead, |
| 233 |
consider the application to be a set of subroutines, and write some glue code |
| 234 |
that gives Python access to those routines, just like you would write a normal |
| 235 |
Python extension. For example:: |
| 236 |
|
| 237 |
static int numargs=0; |
| 238 |
|
| 239 |
/* Return the number of arguments of the application command line */ |
| 240 |
static PyObject* |
| 241 |
emb_numargs(PyObject *self, PyObject *args) |
| 242 |
{ |
| 243 |
if(!PyArg_ParseTuple(args, ":numargs")) |
| 244 |
return NULL; |
| 245 |
return PyLong_FromLong(numargs); |
| 246 |
} |
| 247 |
|
| 248 |
static PyMethodDef emb_module_methods[] = { |
| 249 |
{"numargs", emb_numargs, METH_VARARGS, |
| 250 |
"Return the number of arguments received by the process."}, |
| 251 |
{NULL, NULL, 0, NULL} |
| 252 |
}; |
| 253 |
|
| 254 |
static struct PyModuleDef emb_module = { |
| 255 |
.m_base = PyModuleDef_HEAD_INIT, |
| 256 |
.m_name = "emb", |
| 257 |
.m_size = 0, |
| 258 |
.m_methods = emb_module_methods, |
| 259 |
}; |
| 260 |
|
| 261 |
static PyObject* |
| 262 |
PyInit_emb(void) |
| 263 |
{ |
| 264 |
return PyModuleDef_Init(&emb_module); |
| 265 |
} |
| 266 |
|
| 267 |
Insert the above code just above the :c:func:`main` function. Also, insert the |
| 268 |
following two statements before the call to :c:func:`Py_Initialize`:: |
| 269 |
|
| 270 |
numargs = argc; |
| 271 |
PyImport_AppendInittab("emb", &PyInit_emb); |
| 272 |
|
| 273 |
These two lines initialize the ``numargs`` variable, and make the |
| 274 |
:func:`!emb.numargs` function accessible to the embedded Python interpreter. |
| 275 |
With these extensions, the Python script can do things like |
| 276 |
|
| 277 |
.. code-block:: python |
| 278 |
|
| 279 |
import emb |
| 280 |
print("Number of arguments", emb.numargs()) |
| 281 |
|
| 282 |
In a real application, the methods will expose an API of the application to |
| 283 |
Python. |
| 284 |
|
| 285 |
.. TODO: threads, code examples do not really behave well if errors happen |
| 286 |
(what to watch out for) |
| 287 |
|
| 288 |
|
| 289 |
.. _embeddingincplusplus: |
| 290 |
|
| 291 |
Embedding Python in C++ |
| 292 |
======================= |
| 293 |
|
| 294 |
It is also possible to embed Python in a C++ program; precisely how this is done |
| 295 |
will depend on the details of the C++ system used; in general you will need to |
| 296 |
write the main program in C++, and use the C++ compiler to compile and link your |
| 297 |
program. There is no need to recompile Python itself using C++. |
| 298 |
|
| 299 |
|
| 300 |
.. _compiling: |
| 301 |
|
| 302 |
Compiling and Linking under Unix-like systems |
| 303 |
============================================= |
| 304 |
|
| 305 |
It is not necessarily trivial to find the right flags to pass to your |
| 306 |
compiler (and linker) in order to embed the Python interpreter into your |
| 307 |
application, particularly because Python needs to load library modules |
| 308 |
implemented as C dynamic extensions (:file:`.so` files) linked against |
| 309 |
it. |
| 310 |
|
| 311 |
To find out the required compiler and linker flags, you can execute the |
| 312 |
:file:`python{X.Y}-config` script which is generated as part of the |
| 313 |
installation process (a :file:`python3-config` script may also be |
| 314 |
available). This script has several options, of which the following will |
| 315 |
be directly useful to you: |
| 316 |
|
| 317 |
* ``pythonX.Y-config --cflags`` will give you the recommended flags when |
| 318 |
compiling: |
| 319 |
|
| 320 |
.. code-block:: shell-session |
| 321 |
|
| 322 |
$ /opt/bin/python3.11-config --cflags |
| 323 |
-I/opt/include/python3.11 -I/opt/include/python3.11 -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall |
| 324 |
|
| 325 |
* ``pythonX.Y-config --ldflags --embed`` will give you the recommended flags |
| 326 |
when linking: |
| 327 |
|
| 328 |
.. code-block:: shell-session |
| 329 |
|
| 330 |
$ /opt/bin/python3.11-config --ldflags --embed |
| 331 |
-L/opt/lib/python3.11/config-3.11-x86_64-linux-gnu -L/opt/lib -lpython3.11 -lpthread -ldl -lutil -lm |
| 332 |
|
| 333 |
.. note:: |
| 334 |
To avoid confusion between several Python installations (and especially |
| 335 |
between the system Python and your own compiled Python), it is recommended |
| 336 |
that you use the absolute path to :file:`python{X.Y}-config`, as in the above |
| 337 |
example. |
| 338 |
|
| 339 |
If this procedure doesn't work for you (it is not guaranteed to work for |
| 340 |
all Unix-like platforms; however, we welcome :ref:`bug reports <reporting-bugs>`) |
| 341 |
you will have to read your system's documentation about dynamic linking and/or |
| 342 |
examine Python's :file:`Makefile` (use :func:`sysconfig.get_makefile_filename` |
| 343 |
to find its location) and compilation |
| 344 |
options. In this case, the :mod:`sysconfig` module is a useful tool to |
| 345 |
programmatically extract the configuration values that you will want to |
| 346 |
combine together. For example: |
| 347 |
|
| 348 |
.. code-block:: pycon |
| 349 |
|
| 350 |
>>> import sysconfig |
| 351 |
>>> sysconfig.get_config_var('LIBS') |
| 352 |
'-lpthread -ldl -lutil' |
| 353 |
>>> sysconfig.get_config_var('LINKFORSHARED') |
| 354 |
'-Xlinker -export-dynamic' |
| 355 |
|
| 356 |
|
| 357 |
.. XXX similar documentation for Windows missing |