Debian Codegraph @

Doc/library/asyncio-subprocess.rst

raw
1 .. currentmodule:: asyncio
2
3 .. _asyncio-subprocess:
4
5 ============
6 Subprocesses
7 ============
8
9 **Source code:** :source:`Lib/asyncio/subprocess.py`,
10 :source:`Lib/asyncio/base_subprocess.py`
11
12 ----------------------------------------
13
14 This section describes high-level async/await asyncio APIs to
15 create and manage subprocesses.
16
17 .. _asyncio_example_subprocess_shell:
18
19 Here's an example of how asyncio can run a shell command and
20 obtain its result::
21
22 import asyncio
23
24 async def run(cmd):
25 proc = await asyncio.create_subprocess_shell(
26 cmd,
27 stdout=asyncio.subprocess.PIPE,
28 stderr=asyncio.subprocess.PIPE)
29
30 stdout, stderr = await proc.communicate()
31
32 print(f'[{cmd!r} exited with {proc.returncode}]')
33 if stdout:
34 print(f'[stdout]\n{stdout.decode()}')
35 if stderr:
36 print(f'[stderr]\n{stderr.decode()}')
37
38 asyncio.run(run('ls /zzz'))
39
40 will print::
41
42 ['ls /zzz' exited with 1]
43 [stderr]
44 ls: /zzz: No such file or directory
45
46 Because all asyncio subprocess functions are asynchronous and asyncio
47 provides many tools to work with such functions, it is easy to execute
48 and monitor multiple subprocesses in parallel. It is indeed trivial
49 to modify the above example to run several commands simultaneously::
50
51 async def main():
52 await asyncio.gather(
53 run('ls /zzz'),
54 run('sleep 1; echo "hello"'))
55
56 asyncio.run(main())
57
58 See also the `Examples`_ subsection.
59
60
61 Creating Subprocesses
62 =====================
63
64 .. function:: create_subprocess_exec(program, *args, stdin=None, \
65 stdout=None, stderr=None, limit=None, **kwds)
66 :async:
67
68 Create a subprocess.
69
70 The *limit* argument sets the buffer limit for :class:`StreamReader`
71 wrappers for :attr:`~asyncio.subprocess.Process.stdout` and :attr:`~asyncio.subprocess.Process.stderr`
72 (if :const:`subprocess.PIPE` is passed to *stdout* and *stderr* arguments).
73
74 Return a :class:`~asyncio.subprocess.Process` instance.
75
76 See the documentation of :meth:`loop.subprocess_exec` for other
77 parameters.
78
79 If the process object is garbage collected while the process is still
80 running, the child process will be killed.
81
82 .. versionchanged:: 3.10
83 Removed the *loop* parameter.
84
85
86 .. function:: create_subprocess_shell(cmd, stdin=None, \
87 stdout=None, stderr=None, limit=None, **kwds)
88 :async:
89
90 Run the *cmd* shell command.
91
92 The *limit* argument sets the buffer limit for :class:`StreamReader`
93 wrappers for :attr:`~asyncio.subprocess.Process.stdout` and :attr:`~asyncio.subprocess.Process.stderr`
94 (if :const:`subprocess.PIPE` is passed to *stdout* and *stderr* arguments).
95
96 Return a :class:`~asyncio.subprocess.Process` instance.
97
98 See the documentation of :meth:`loop.subprocess_shell` for other
99 parameters.
100
101 If the process object is garbage collected while the process is still
102 running, the child process will be killed.
103
104 .. important::
105
106 It is the application's responsibility to ensure that all whitespace and
107 special characters are quoted appropriately to avoid `shell injection
108 <https://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_
109 vulnerabilities. The :func:`shlex.quote` function can be used to properly
110 escape whitespace and special shell characters in strings that are going
111 to be used to construct shell commands.
112
113 .. versionchanged:: 3.10
114 Removed the *loop* parameter.
115
116 .. note::
117
118 Subprocesses are available for Windows if a :class:`ProactorEventLoop` is
119 used. See :ref:`Subprocess Support on Windows <asyncio-windows-subprocess>`
120 for details.
121
122 .. seealso::
123
124 asyncio also has the following *low-level* APIs to work with subprocesses:
125 :meth:`loop.subprocess_exec`, :meth:`loop.subprocess_shell`,
126 :meth:`loop.connect_read_pipe`, :meth:`loop.connect_write_pipe`,
127 as well as the :ref:`Subprocess Transports <asyncio-subprocess-transports>`
128 and :ref:`Subprocess Protocols <asyncio-subprocess-protocols>`.
129
130
131 Constants
132 =========
133
134 .. data:: asyncio.subprocess.PIPE
135 :module:
136
137 Can be passed to the *stdin*, *stdout* or *stderr* parameters.
138
139 If *PIPE* is passed to *stdin* argument, the
140 :attr:`Process.stdin <asyncio.subprocess.Process.stdin>` attribute
141 will point to a :class:`~asyncio.StreamWriter` instance.
142
143 If *PIPE* is passed to *stdout* or *stderr* arguments, the
144 :attr:`Process.stdout <asyncio.subprocess.Process.stdout>` and
145 :attr:`Process.stderr <asyncio.subprocess.Process.stderr>`
146 attributes will point to :class:`~asyncio.StreamReader` instances.
147
148 .. data:: asyncio.subprocess.STDOUT
149 :module:
150
151 Special value that can be used as the *stderr* argument and indicates
152 that standard error should be redirected into standard output.
153
154 .. data:: asyncio.subprocess.DEVNULL
155 :module:
156
157 Special value that can be used as the *stdin*, *stdout* or *stderr* argument
158 to process creation functions. It indicates that the special file
159 :data:`os.devnull` will be used for the corresponding subprocess stream.
160
161
162 Interacting with Subprocesses
163 =============================
164
165 Both :func:`create_subprocess_exec` and :func:`create_subprocess_shell`
166 functions return instances of the *Process* class. *Process* is a high-level
167 wrapper that allows communicating with subprocesses and watching for
168 their completion.
169
170 .. class:: asyncio.subprocess.Process
171 :module:
172
173 An object that wraps OS processes created by the
174 :func:`~asyncio.create_subprocess_exec` and :func:`~asyncio.create_subprocess_shell`
175 functions.
176
177 This class is designed to have a similar API to the
178 :class:`subprocess.Popen` class, but there are some
179 notable differences:
180
181 * unlike Popen, Process instances do not have an equivalent to
182 the :meth:`~subprocess.Popen.poll` method;
183
184 * the :meth:`~asyncio.subprocess.Process.communicate` and
185 :meth:`~asyncio.subprocess.Process.wait` methods don't have a
186 *timeout* parameter: use the :func:`~asyncio.wait_for` function;
187
188 * the :meth:`Process.wait() <asyncio.subprocess.Process.wait>` method
189 is asynchronous, whereas :meth:`subprocess.Popen.wait` method
190 is implemented as a blocking busy loop;
191
192 * the *universal_newlines* parameter is not supported.
193
194 This class is :ref:`not thread safe <asyncio-multithreading>`.
195
196 See also the :ref:`Subprocess and Threads <asyncio-subprocess-threads>`
197 section.
198
199 .. method:: wait()
200 :async:
201
202 Wait for the child process to terminate.
203
204 Set and return the :attr:`returncode` attribute.
205
206 .. note::
207
208 This method can deadlock when using ``stdout=PIPE`` or
209 ``stderr=PIPE`` and the child process generates so much output
210 that it blocks waiting for the OS pipe buffer to accept
211 more data. Use the :meth:`communicate` method when using pipes
212 to avoid this condition.
213
214 .. method:: communicate(input=None)
215 :async:
216
217 Interact with process:
218
219 1. send data to *stdin* (if *input* is not ``None``);
220 2. closes *stdin*;
221 3. read data from *stdout* and *stderr*, until EOF is reached;
222 4. wait for process to terminate.
223
224 The optional *input* argument is the data (:class:`bytes` object)
225 that will be sent to the child process.
226
227 Return a tuple ``(stdout_data, stderr_data)``.
228
229 If either :exc:`BrokenPipeError` or :exc:`ConnectionResetError`
230 exception is raised when writing *input* into *stdin*, the
231 exception is ignored. This condition occurs when the process
232 exits before all data are written into *stdin*.
233
234 If it is desired to send data to the process' *stdin*,
235 the process needs to be created with ``stdin=PIPE``. Similarly,
236 to get anything other than ``None`` in the result tuple, the
237 process has to be created with ``stdout=PIPE`` and/or
238 ``stderr=PIPE`` arguments.
239
240 Note, that the data read is buffered in memory, so do not use
241 this method if the data size is large or unlimited.
242
243 .. versionchanged:: 3.12
244
245 *stdin* gets closed when ``input=None`` too.
246
247 .. method:: send_signal(signal)
248
249 Sends the signal *signal* to the child process.
250
251 .. note::
252
253 On Windows, :py:const:`~signal.SIGTERM` is an alias for :meth:`terminate`.
254 ``CTRL_C_EVENT`` and ``CTRL_BREAK_EVENT`` can be sent to processes
255 started with a *creationflags* parameter which includes
256 ``CREATE_NEW_PROCESS_GROUP``.
257
258 .. method:: terminate()
259
260 Stop the child process.
261
262 On POSIX systems this method sends :py:const:`~signal.SIGTERM` to the
263 child process.
264
265 On Windows the Win32 API function :c:func:`!TerminateProcess` is
266 called to stop the child process.
267
268 .. method:: kill()
269
270 Kill the child process.
271
272 On POSIX systems this method sends :py:data:`~signal.SIGKILL` to the child
273 process.
274
275 On Windows this method is an alias for :meth:`terminate`.
276
277 .. attribute:: stdin
278
279 Standard input stream (:class:`~asyncio.StreamWriter`) or ``None``
280 if the process was created with ``stdin=None``.
281
282 .. attribute:: stdout
283
284 Standard output stream (:class:`~asyncio.StreamReader`) or ``None``
285 if the process was created with ``stdout=None``.
286
287 .. attribute:: stderr
288
289 Standard error stream (:class:`~asyncio.StreamReader`) or ``None``
290 if the process was created with ``stderr=None``.
291
292 .. warning::
293
294 Use the :meth:`communicate` method rather than
295 :attr:`process.stdin.write() <stdin>`,
296 :attr:`await process.stdout.read() <stdout>` or
297 :attr:`await process.stderr.read() <stderr>`.
298 This avoids deadlocks due to streams pausing reading or writing
299 and blocking the child process.
300
301 .. attribute:: pid
302
303 Process identification number (PID).
304
305 Note that for processes created by the :func:`~asyncio.create_subprocess_shell`
306 function, this attribute is the PID of the spawned shell.
307
308 .. attribute:: returncode
309
310 Return code of the process when it exits.
311
312 A ``None`` value indicates that the process has not terminated yet.
313
314 For processes created with :func:`~asyncio.create_subprocess_exec`, a negative
315 value ``-N`` indicates that the child was terminated by signal ``N``
316 (POSIX only).
317
318 For processes created with :func:`~asyncio.create_subprocess_shell`, the
319 return code reflects the exit status of the shell itself (e.g. ``/bin/sh``),
320 which may map signals to codes such as ``128+N``. See the
321 documentation of the shell (for example, the Bash manual's Exit Status)
322 for details.
323
324
325
326 .. _asyncio-subprocess-threads:
327
328 Subprocess and Threads
329 ----------------------
330
331 Standard asyncio event loop supports running subprocesses from different threads by
332 default.
333
334 On Windows subprocesses are provided by :class:`ProactorEventLoop` only (default),
335 :class:`SelectorEventLoop` has no subprocess support.
336
337 Note that alternative event loop implementations might have own limitations;
338 please refer to their documentation.
339
340 .. seealso::
341
342 The :ref:`Concurrency and multithreading in asyncio
343 <asyncio-multithreading>` section.
344
345
346 Examples
347 --------
348
349 An example using the :class:`~asyncio.subprocess.Process` class to
350 control a subprocess and the :class:`StreamReader` class to read from
351 its standard output.
352
353 .. _asyncio_example_create_subprocess_exec:
354
355 The subprocess is created by the :func:`create_subprocess_exec`
356 function::
357
358 import asyncio
359 import sys
360
361 async def get_date():
362 code = 'import datetime as dt; print(dt.datetime.now())'
363
364 # Create the subprocess; redirect the standard output
365 # into a pipe.
366 proc = await asyncio.create_subprocess_exec(
367 sys.executable, '-c', code,
368 stdout=asyncio.subprocess.PIPE)
369
370 # Read one line of output.
371 data = await proc.stdout.readline()
372 line = data.decode('ascii').rstrip()
373
374 # Wait for the subprocess exit.
375 await proc.wait()
376 return line
377
378 date = asyncio.run(get_date())
379 print(f"Current date: {date}")
380
381
382 See also the :ref:`same example <asyncio_example_subprocess_proto>`
383 written using low-level APIs.