Debian Codegraph @

Doc/extending/embedding.rst

raw
1 .. highlight:: c
2
3
4 .. _embedding:
5
6 ***************************************
7 Embedding Python in Another Application
8 ***************************************
9
10 The previous chapters discussed how to extend Python, that is, how to extend the
11 functionality of Python by attaching a library of C functions to it. It is also
12 possible to do it the other way around: enrich your C/C++ application by
13 embedding Python in it. Embedding provides your application with the ability to
14 implement some of the functionality of your application in Python rather than C
15 or C++. This can be used for many purposes; one example would be to allow users
16 to tailor the application to their needs by writing some scripts in Python. You
17 can also use it yourself if some of the functionality can be written in Python
18 more easily.
19
20 Embedding Python is similar to extending it, but not quite. The difference is
21 that when you extend Python, the main program of the application is still the
22 Python interpreter, while if you embed Python, the main program may have nothing
23 to do with Python --- instead, some parts of the application occasionally call
24 the Python interpreter to run some Python code.
25
26 So if you are embedding Python, you are providing your own main program. One of
27 the things this main program has to do is initialize the Python interpreter. At
28 the very least, you have to call the function :c:func:`Py_Initialize`. There are
29 optional calls to pass command line arguments to Python. Then later you can
30 call the interpreter from any part of the application.
31
32 There are several different ways to call the interpreter: you can pass a string
33 containing Python statements to :c:func:`PyRun_SimpleString`, or you can pass a
34 stdio file pointer and a file name (for identification in error messages only)
35 to :c:func:`PyRun_SimpleFile`. You can also call the lower-level operations
36 described in the previous chapters to construct and use Python objects.
37
38
39 .. seealso::
40
41 :ref:`c-api-index`
42 The details of Python's C interface are given in this manual. A great deal of
43 necessary information can be found here.
44
45
46 .. _high-level-embedding:
47
48 Very High Level Embedding
49 =========================
50
51 The simplest form of embedding Python is the use of the very high level
52 interface. This interface is intended to execute a Python script without needing
53 to interact with the application directly. This can for example be used to
54 perform some operation on a file. ::
55
56 #define PY_SSIZE_T_CLEAN
57 #include <Python.h>
58
59 int
60 main(int argc, char *argv[])
61 {
62 PyStatus status;
63 PyConfig config;
64 PyConfig_InitPythonConfig(&config);
65
66 /* optional but recommended */
67 status = PyConfig_SetBytesString(&config, &config.program_name, argv[0]);
68 if (PyStatus_Exception(status)) {
69 goto exception;
70 }
71
72 status = Py_InitializeFromConfig(&config);
73 if (PyStatus_Exception(status)) {
74 goto exception;
75 }
76 PyConfig_Clear(&config);
77
78 PyRun_SimpleString("from time import time,ctime\n"
79 "print('Today is', ctime(time()))\n");
80 if (Py_FinalizeEx() < 0) {
81 exit(120);
82 }
83 return 0;
84
85 exception:
86 PyConfig_Clear(&config);
87 Py_ExitStatusException(status);
88 }
89
90 .. note::
91
92 ``#define PY_SSIZE_T_CLEAN`` was used to indicate that ``Py_ssize_t`` should be
93 used in some APIs instead of ``int``.
94 It is not necessary since Python 3.13, but we keep it here for backward compatibility.
95 See :ref:`arg-parsing-string-and-buffers` for a description of this macro.
96
97 Setting :c:member:`PyConfig.program_name` should be called before
98 :c:func:`Py_InitializeFromConfig` to inform the interpreter about paths to Python run-time
99 libraries. Next, the Python interpreter is initialized with
100 :c:func:`Py_Initialize`, followed by the execution of a hard-coded Python script
101 that prints the date and time. Afterwards, the :c:func:`Py_FinalizeEx` call shuts
102 the interpreter down, followed by the end of the program. In a real program,
103 you may want to get the Python script from another source, perhaps a text-editor
104 routine, a file, or a database. Getting the Python code from a file can better
105 be done by using the :c:func:`PyRun_SimpleFile` function, which saves you the
106 trouble of allocating memory space and loading the file contents.
107
108
109 .. _lower-level-embedding:
110
111 Beyond Very High Level Embedding: An overview
112 =============================================
113
114 The high level interface gives you the ability to execute arbitrary pieces of
115 Python code from your application, but exchanging data values is quite
116 cumbersome to say the least. If you want that, you should use lower level calls.
117 At the cost of having to write more C code, you can achieve almost anything.
118
119 It should be noted that extending Python and embedding Python is quite the same
120 activity, despite the different intent. Most topics discussed in the previous
121 chapters are still valid. To show this, consider what the extension code from
122 Python to C really does:
123
124 #. Convert data values from Python to C,
125
126 #. Perform a function call to a C routine using the converted values, and
127
128 #. Convert the data values from the call from C to Python.
129
130 When embedding Python, the interface code does:
131
132 #. Convert data values from C to Python,
133
134 #. Perform a function call to a Python interface routine using the converted
135 values, and
136
137 #. Convert the data values from the call from Python to C.
138
139 As you can see, the data conversion steps are simply swapped to accommodate the
140 different direction of the cross-language transfer. The only difference is the
141 routine that you call between both data conversions. When extending, you call a
142 C routine, when embedding, you call a Python routine.
143
144 This chapter will not discuss how to convert data from Python to C and vice
145 versa. Also, proper use of references and dealing with errors is assumed to be
146 understood. Since these aspects do not differ from extending the interpreter,
147 you can refer to earlier chapters for the required information.
148
149
150 .. _pure-embedding:
151
152 Pure Embedding
153 ==============
154
155 The first program aims to execute a function in a Python script. Like in the
156 section about the very high level interface, the Python interpreter does not
157 directly interact with the application (but that will change in the next
158 section).
159
160 The code to run a function defined in a Python script is:
161
162 .. literalinclude:: ../includes/run-func.c
163
164
165 This code loads a Python script using ``argv[1]``, and calls the function named
166 in ``argv[2]``. Its integer arguments are the other values of the ``argv``
167 array. If you :ref:`compile and link <compiling>` this program (let's call
168 the finished executable :program:`call`), and use it to execute a Python
169 script, such as:
170
171 .. code-block:: python
172
173 def multiply(a,b):
174 print("Will compute", a, "times", b)
175 c = 0
176 for i in range(0, a):
177 c = c + b
178 return c
179
180 then the result should be:
181
182 .. code-block:: shell-session
183
184 $ call multiply multiply 3 2
185 Will compute 3 times 2
186 Result of call: 6
187
188 Although the program is quite large for its functionality, most of the code is
189 for data conversion between Python and C, and for error reporting. The
190 interesting part with respect to embedding Python starts with ::
191
192 Py_Initialize();
193 pName = PyUnicode_DecodeFSDefault(argv[1]);
194 /* Error checking of pName left out */
195 pModule = PyImport_Import(pName);
196
197 After initializing the interpreter, the script is loaded using
198 :c:func:`PyImport_Import`. This routine needs a Python string as its argument,
199 which is constructed using the :c:func:`PyUnicode_DecodeFSDefault` data
200 conversion routine. ::
201
202 pFunc = PyObject_GetAttrString(pModule, argv[2]);
203 /* pFunc is a new reference */
204
205 if (pFunc && PyCallable_Check(pFunc)) {
206 ...
207 }
208 Py_XDECREF(pFunc);
209
210 Once the script is loaded, the name we're looking for is retrieved using
211 :c:func:`PyObject_GetAttrString`. If the name exists, and the object returned is
212 callable, you can safely assume that it is a function. The program then
213 proceeds by constructing a tuple of arguments as normal. The call to the Python
214 function is then made with::
215
216 pValue = PyObject_CallObject(pFunc, pArgs);
217
218 Upon return of the function, ``pValue`` is either ``NULL`` or it contains a
219 reference to the return value of the function. Be sure to release the reference
220 after examining the value.
221
222
223 .. _extending-with-embedding:
224
225 Extending Embedded Python
226 =========================
227
228 Until now, the embedded Python interpreter had no access to functionality from
229 the application itself. The Python API allows this by extending the embedded
230 interpreter. That is, the embedded interpreter gets extended with routines
231 provided by the application. While it sounds complex, it is not so bad. Simply
232 forget for a while that the application starts the Python interpreter. Instead,
233 consider the application to be a set of subroutines, and write some glue code
234 that gives Python access to those routines, just like you would write a normal
235 Python extension. For example::
236
237 static int numargs=0;
238
239 /* Return the number of arguments of the application command line */
240 static PyObject*
241 emb_numargs(PyObject *self, PyObject *args)
242 {
243 if(!PyArg_ParseTuple(args, ":numargs"))
244 return NULL;
245 return PyLong_FromLong(numargs);
246 }
247
248 static PyMethodDef emb_module_methods[] = {
249 {"numargs", emb_numargs, METH_VARARGS,
250 "Return the number of arguments received by the process."},
251 {NULL, NULL, 0, NULL}
252 };
253
254 static struct PyModuleDef emb_module = {
255 .m_base = PyModuleDef_HEAD_INIT,
256 .m_name = "emb",
257 .m_size = 0,
258 .m_methods = emb_module_methods,
259 };
260
261 static PyObject*
262 PyInit_emb(void)
263 {
264 return PyModuleDef_Init(&emb_module);
265 }
266
267 Insert the above code just above the :c:func:`main` function. Also, insert the
268 following two statements before the call to :c:func:`Py_Initialize`::
269
270 numargs = argc;
271 PyImport_AppendInittab("emb", &PyInit_emb);
272
273 These two lines initialize the ``numargs`` variable, and make the
274 :func:`!emb.numargs` function accessible to the embedded Python interpreter.
275 With these extensions, the Python script can do things like
276
277 .. code-block:: python
278
279 import emb
280 print("Number of arguments", emb.numargs())
281
282 In a real application, the methods will expose an API of the application to
283 Python.
284
285 .. TODO: threads, code examples do not really behave well if errors happen
286 (what to watch out for)
287
288
289 .. _embeddingincplusplus:
290
291 Embedding Python in C++
292 =======================
293
294 It is also possible to embed Python in a C++ program; precisely how this is done
295 will depend on the details of the C++ system used; in general you will need to
296 write the main program in C++, and use the C++ compiler to compile and link your
297 program. There is no need to recompile Python itself using C++.
298
299
300 .. _compiling:
301
302 Compiling and Linking under Unix-like systems
303 =============================================
304
305 It is not necessarily trivial to find the right flags to pass to your
306 compiler (and linker) in order to embed the Python interpreter into your
307 application, particularly because Python needs to load library modules
308 implemented as C dynamic extensions (:file:`.so` files) linked against
309 it.
310
311 To find out the required compiler and linker flags, you can execute the
312 :file:`python{X.Y}-config` script which is generated as part of the
313 installation process (a :file:`python3-config` script may also be
314 available). This script has several options, of which the following will
315 be directly useful to you:
316
317 * ``pythonX.Y-config --cflags`` will give you the recommended flags when
318 compiling:
319
320 .. code-block:: shell-session
321
322 $ /opt/bin/python3.11-config --cflags
323 -I/opt/include/python3.11 -I/opt/include/python3.11 -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall
324
325 * ``pythonX.Y-config --ldflags --embed`` will give you the recommended flags
326 when linking:
327
328 .. code-block:: shell-session
329
330 $ /opt/bin/python3.11-config --ldflags --embed
331 -L/opt/lib/python3.11/config-3.11-x86_64-linux-gnu -L/opt/lib -lpython3.11 -lpthread -ldl -lutil -lm
332
333 .. note::
334 To avoid confusion between several Python installations (and especially
335 between the system Python and your own compiled Python), it is recommended
336 that you use the absolute path to :file:`python{X.Y}-config`, as in the above
337 example.
338
339 If this procedure doesn't work for you (it is not guaranteed to work for
340 all Unix-like platforms; however, we welcome :ref:`bug reports <reporting-bugs>`)
341 you will have to read your system's documentation about dynamic linking and/or
342 examine Python's :file:`Makefile` (use :func:`sysconfig.get_makefile_filename`
343 to find its location) and compilation
344 options. In this case, the :mod:`sysconfig` module is a useful tool to
345 programmatically extract the configuration values that you will want to
346 combine together. For example:
347
348 .. code-block:: pycon
349
350 >>> import sysconfig
351 >>> sysconfig.get_config_var('LIBS')
352 '-lpthread -ldl -lutil'
353 >>> sysconfig.get_config_var('LINKFORSHARED')
354 '-Xlinker -export-dynamic'
355
356
357 .. XXX similar documentation for Windows missing