| 1 |
.. currentmodule:: asyncio |
| 2 |
|
| 3 |
.. _asyncio-subprocess: |
| 4 |
|
| 5 |
============ |
| 6 |
Subprocesses |
| 7 |
============ |
| 8 |
|
| 9 |
**Source code:** :source:`Lib/asyncio/subprocess.py`, |
| 10 |
:source:`Lib/asyncio/base_subprocess.py` |
| 11 |
|
| 12 |
---------------------------------------- |
| 13 |
|
| 14 |
This section describes high-level async/await asyncio APIs to |
| 15 |
create and manage subprocesses. |
| 16 |
|
| 17 |
.. _asyncio_example_subprocess_shell: |
| 18 |
|
| 19 |
Here's an example of how asyncio can run a shell command and |
| 20 |
obtain its result:: |
| 21 |
|
| 22 |
import asyncio |
| 23 |
|
| 24 |
async def run(cmd): |
| 25 |
proc = await asyncio.create_subprocess_shell( |
| 26 |
cmd, |
| 27 |
stdout=asyncio.subprocess.PIPE, |
| 28 |
stderr=asyncio.subprocess.PIPE) |
| 29 |
|
| 30 |
stdout, stderr = await proc.communicate() |
| 31 |
|
| 32 |
print(f'[{cmd!r} exited with {proc.returncode}]') |
| 33 |
if stdout: |
| 34 |
print(f'[stdout]\n{stdout.decode()}') |
| 35 |
if stderr: |
| 36 |
print(f'[stderr]\n{stderr.decode()}') |
| 37 |
|
| 38 |
asyncio.run(run('ls /zzz')) |
| 39 |
|
| 40 |
will print:: |
| 41 |
|
| 42 |
['ls /zzz' exited with 1] |
| 43 |
[stderr] |
| 44 |
ls: /zzz: No such file or directory |
| 45 |
|
| 46 |
Because all asyncio subprocess functions are asynchronous and asyncio |
| 47 |
provides many tools to work with such functions, it is easy to execute |
| 48 |
and monitor multiple subprocesses in parallel. It is indeed trivial |
| 49 |
to modify the above example to run several commands simultaneously:: |
| 50 |
|
| 51 |
async def main(): |
| 52 |
await asyncio.gather( |
| 53 |
run('ls /zzz'), |
| 54 |
run('sleep 1; echo "hello"')) |
| 55 |
|
| 56 |
asyncio.run(main()) |
| 57 |
|
| 58 |
See also the `Examples`_ subsection. |
| 59 |
|
| 60 |
|
| 61 |
Creating Subprocesses |
| 62 |
===================== |
| 63 |
|
| 64 |
.. function:: create_subprocess_exec(program, *args, stdin=None, \ |
| 65 |
stdout=None, stderr=None, limit=None, **kwds) |
| 66 |
:async: |
| 67 |
|
| 68 |
Create a subprocess. |
| 69 |
|
| 70 |
The *limit* argument sets the buffer limit for :class:`StreamReader` |
| 71 |
wrappers for :attr:`~asyncio.subprocess.Process.stdout` and :attr:`~asyncio.subprocess.Process.stderr` |
| 72 |
(if :const:`subprocess.PIPE` is passed to *stdout* and *stderr* arguments). |
| 73 |
|
| 74 |
Return a :class:`~asyncio.subprocess.Process` instance. |
| 75 |
|
| 76 |
See the documentation of :meth:`loop.subprocess_exec` for other |
| 77 |
parameters. |
| 78 |
|
| 79 |
If the process object is garbage collected while the process is still |
| 80 |
running, the child process will be killed. |
| 81 |
|
| 82 |
.. versionchanged:: 3.10 |
| 83 |
Removed the *loop* parameter. |
| 84 |
|
| 85 |
|
| 86 |
.. function:: create_subprocess_shell(cmd, stdin=None, \ |
| 87 |
stdout=None, stderr=None, limit=None, **kwds) |
| 88 |
:async: |
| 89 |
|
| 90 |
Run the *cmd* shell command. |
| 91 |
|
| 92 |
The *limit* argument sets the buffer limit for :class:`StreamReader` |
| 93 |
wrappers for :attr:`~asyncio.subprocess.Process.stdout` and :attr:`~asyncio.subprocess.Process.stderr` |
| 94 |
(if :const:`subprocess.PIPE` is passed to *stdout* and *stderr* arguments). |
| 95 |
|
| 96 |
Return a :class:`~asyncio.subprocess.Process` instance. |
| 97 |
|
| 98 |
See the documentation of :meth:`loop.subprocess_shell` for other |
| 99 |
parameters. |
| 100 |
|
| 101 |
If the process object is garbage collected while the process is still |
| 102 |
running, the child process will be killed. |
| 103 |
|
| 104 |
.. important:: |
| 105 |
|
| 106 |
It is the application's responsibility to ensure that all whitespace and |
| 107 |
special characters are quoted appropriately to avoid `shell injection |
| 108 |
<https://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_ |
| 109 |
vulnerabilities. The :func:`shlex.quote` function can be used to properly |
| 110 |
escape whitespace and special shell characters in strings that are going |
| 111 |
to be used to construct shell commands. |
| 112 |
|
| 113 |
.. versionchanged:: 3.10 |
| 114 |
Removed the *loop* parameter. |
| 115 |
|
| 116 |
.. note:: |
| 117 |
|
| 118 |
Subprocesses are available for Windows if a :class:`ProactorEventLoop` is |
| 119 |
used. See :ref:`Subprocess Support on Windows <asyncio-windows-subprocess>` |
| 120 |
for details. |
| 121 |
|
| 122 |
.. seealso:: |
| 123 |
|
| 124 |
asyncio also has the following *low-level* APIs to work with subprocesses: |
| 125 |
:meth:`loop.subprocess_exec`, :meth:`loop.subprocess_shell`, |
| 126 |
:meth:`loop.connect_read_pipe`, :meth:`loop.connect_write_pipe`, |
| 127 |
as well as the :ref:`Subprocess Transports <asyncio-subprocess-transports>` |
| 128 |
and :ref:`Subprocess Protocols <asyncio-subprocess-protocols>`. |
| 129 |
|
| 130 |
|
| 131 |
Constants |
| 132 |
========= |
| 133 |
|
| 134 |
.. data:: asyncio.subprocess.PIPE |
| 135 |
:module: |
| 136 |
|
| 137 |
Can be passed to the *stdin*, *stdout* or *stderr* parameters. |
| 138 |
|
| 139 |
If *PIPE* is passed to *stdin* argument, the |
| 140 |
:attr:`Process.stdin <asyncio.subprocess.Process.stdin>` attribute |
| 141 |
will point to a :class:`~asyncio.StreamWriter` instance. |
| 142 |
|
| 143 |
If *PIPE* is passed to *stdout* or *stderr* arguments, the |
| 144 |
:attr:`Process.stdout <asyncio.subprocess.Process.stdout>` and |
| 145 |
:attr:`Process.stderr <asyncio.subprocess.Process.stderr>` |
| 146 |
attributes will point to :class:`~asyncio.StreamReader` instances. |
| 147 |
|
| 148 |
.. data:: asyncio.subprocess.STDOUT |
| 149 |
:module: |
| 150 |
|
| 151 |
Special value that can be used as the *stderr* argument and indicates |
| 152 |
that standard error should be redirected into standard output. |
| 153 |
|
| 154 |
.. data:: asyncio.subprocess.DEVNULL |
| 155 |
:module: |
| 156 |
|
| 157 |
Special value that can be used as the *stdin*, *stdout* or *stderr* argument |
| 158 |
to process creation functions. It indicates that the special file |
| 159 |
:data:`os.devnull` will be used for the corresponding subprocess stream. |
| 160 |
|
| 161 |
|
| 162 |
Interacting with Subprocesses |
| 163 |
============================= |
| 164 |
|
| 165 |
Both :func:`create_subprocess_exec` and :func:`create_subprocess_shell` |
| 166 |
functions return instances of the *Process* class. *Process* is a high-level |
| 167 |
wrapper that allows communicating with subprocesses and watching for |
| 168 |
their completion. |
| 169 |
|
| 170 |
.. class:: asyncio.subprocess.Process |
| 171 |
:module: |
| 172 |
|
| 173 |
An object that wraps OS processes created by the |
| 174 |
:func:`~asyncio.create_subprocess_exec` and :func:`~asyncio.create_subprocess_shell` |
| 175 |
functions. |
| 176 |
|
| 177 |
This class is designed to have a similar API to the |
| 178 |
:class:`subprocess.Popen` class, but there are some |
| 179 |
notable differences: |
| 180 |
|
| 181 |
* unlike Popen, Process instances do not have an equivalent to |
| 182 |
the :meth:`~subprocess.Popen.poll` method; |
| 183 |
|
| 184 |
* the :meth:`~asyncio.subprocess.Process.communicate` and |
| 185 |
:meth:`~asyncio.subprocess.Process.wait` methods don't have a |
| 186 |
*timeout* parameter: use the :func:`~asyncio.wait_for` function; |
| 187 |
|
| 188 |
* the :meth:`Process.wait() <asyncio.subprocess.Process.wait>` method |
| 189 |
is asynchronous, whereas :meth:`subprocess.Popen.wait` method |
| 190 |
is implemented as a blocking busy loop; |
| 191 |
|
| 192 |
* the *universal_newlines* parameter is not supported. |
| 193 |
|
| 194 |
This class is :ref:`not thread safe <asyncio-multithreading>`. |
| 195 |
|
| 196 |
See also the :ref:`Subprocess and Threads <asyncio-subprocess-threads>` |
| 197 |
section. |
| 198 |
|
| 199 |
.. method:: wait() |
| 200 |
:async: |
| 201 |
|
| 202 |
Wait for the child process to terminate. |
| 203 |
|
| 204 |
Set and return the :attr:`returncode` attribute. |
| 205 |
|
| 206 |
.. note:: |
| 207 |
|
| 208 |
This method can deadlock when using ``stdout=PIPE`` or |
| 209 |
``stderr=PIPE`` and the child process generates so much output |
| 210 |
that it blocks waiting for the OS pipe buffer to accept |
| 211 |
more data. Use the :meth:`communicate` method when using pipes |
| 212 |
to avoid this condition. |
| 213 |
|
| 214 |
.. method:: communicate(input=None) |
| 215 |
:async: |
| 216 |
|
| 217 |
Interact with process: |
| 218 |
|
| 219 |
1. send data to *stdin* (if *input* is not ``None``); |
| 220 |
2. closes *stdin*; |
| 221 |
3. read data from *stdout* and *stderr*, until EOF is reached; |
| 222 |
4. wait for process to terminate. |
| 223 |
|
| 224 |
The optional *input* argument is the data (:class:`bytes` object) |
| 225 |
that will be sent to the child process. |
| 226 |
|
| 227 |
Return a tuple ``(stdout_data, stderr_data)``. |
| 228 |
|
| 229 |
If either :exc:`BrokenPipeError` or :exc:`ConnectionResetError` |
| 230 |
exception is raised when writing *input* into *stdin*, the |
| 231 |
exception is ignored. This condition occurs when the process |
| 232 |
exits before all data are written into *stdin*. |
| 233 |
|
| 234 |
If it is desired to send data to the process' *stdin*, |
| 235 |
the process needs to be created with ``stdin=PIPE``. Similarly, |
| 236 |
to get anything other than ``None`` in the result tuple, the |
| 237 |
process has to be created with ``stdout=PIPE`` and/or |
| 238 |
``stderr=PIPE`` arguments. |
| 239 |
|
| 240 |
Note, that the data read is buffered in memory, so do not use |
| 241 |
this method if the data size is large or unlimited. |
| 242 |
|
| 243 |
.. versionchanged:: 3.12 |
| 244 |
|
| 245 |
*stdin* gets closed when ``input=None`` too. |
| 246 |
|
| 247 |
.. method:: send_signal(signal) |
| 248 |
|
| 249 |
Sends the signal *signal* to the child process. |
| 250 |
|
| 251 |
.. note:: |
| 252 |
|
| 253 |
On Windows, :py:const:`~signal.SIGTERM` is an alias for :meth:`terminate`. |
| 254 |
``CTRL_C_EVENT`` and ``CTRL_BREAK_EVENT`` can be sent to processes |
| 255 |
started with a *creationflags* parameter which includes |
| 256 |
``CREATE_NEW_PROCESS_GROUP``. |
| 257 |
|
| 258 |
.. method:: terminate() |
| 259 |
|
| 260 |
Stop the child process. |
| 261 |
|
| 262 |
On POSIX systems this method sends :py:const:`~signal.SIGTERM` to the |
| 263 |
child process. |
| 264 |
|
| 265 |
On Windows the Win32 API function :c:func:`!TerminateProcess` is |
| 266 |
called to stop the child process. |
| 267 |
|
| 268 |
.. method:: kill() |
| 269 |
|
| 270 |
Kill the child process. |
| 271 |
|
| 272 |
On POSIX systems this method sends :py:data:`~signal.SIGKILL` to the child |
| 273 |
process. |
| 274 |
|
| 275 |
On Windows this method is an alias for :meth:`terminate`. |
| 276 |
|
| 277 |
.. attribute:: stdin |
| 278 |
|
| 279 |
Standard input stream (:class:`~asyncio.StreamWriter`) or ``None`` |
| 280 |
if the process was created with ``stdin=None``. |
| 281 |
|
| 282 |
.. attribute:: stdout |
| 283 |
|
| 284 |
Standard output stream (:class:`~asyncio.StreamReader`) or ``None`` |
| 285 |
if the process was created with ``stdout=None``. |
| 286 |
|
| 287 |
.. attribute:: stderr |
| 288 |
|
| 289 |
Standard error stream (:class:`~asyncio.StreamReader`) or ``None`` |
| 290 |
if the process was created with ``stderr=None``. |
| 291 |
|
| 292 |
.. warning:: |
| 293 |
|
| 294 |
Use the :meth:`communicate` method rather than |
| 295 |
:attr:`process.stdin.write() <stdin>`, |
| 296 |
:attr:`await process.stdout.read() <stdout>` or |
| 297 |
:attr:`await process.stderr.read() <stderr>`. |
| 298 |
This avoids deadlocks due to streams pausing reading or writing |
| 299 |
and blocking the child process. |
| 300 |
|
| 301 |
.. attribute:: pid |
| 302 |
|
| 303 |
Process identification number (PID). |
| 304 |
|
| 305 |
Note that for processes created by the :func:`~asyncio.create_subprocess_shell` |
| 306 |
function, this attribute is the PID of the spawned shell. |
| 307 |
|
| 308 |
.. attribute:: returncode |
| 309 |
|
| 310 |
Return code of the process when it exits. |
| 311 |
|
| 312 |
A ``None`` value indicates that the process has not terminated yet. |
| 313 |
|
| 314 |
For processes created with :func:`~asyncio.create_subprocess_exec`, a negative |
| 315 |
value ``-N`` indicates that the child was terminated by signal ``N`` |
| 316 |
(POSIX only). |
| 317 |
|
| 318 |
For processes created with :func:`~asyncio.create_subprocess_shell`, the |
| 319 |
return code reflects the exit status of the shell itself (e.g. ``/bin/sh``), |
| 320 |
which may map signals to codes such as ``128+N``. See the |
| 321 |
documentation of the shell (for example, the Bash manual's Exit Status) |
| 322 |
for details. |
| 323 |
|
| 324 |
|
| 325 |
|
| 326 |
.. _asyncio-subprocess-threads: |
| 327 |
|
| 328 |
Subprocess and Threads |
| 329 |
---------------------- |
| 330 |
|
| 331 |
Standard asyncio event loop supports running subprocesses from different threads by |
| 332 |
default. |
| 333 |
|
| 334 |
On Windows subprocesses are provided by :class:`ProactorEventLoop` only (default), |
| 335 |
:class:`SelectorEventLoop` has no subprocess support. |
| 336 |
|
| 337 |
Note that alternative event loop implementations might have own limitations; |
| 338 |
please refer to their documentation. |
| 339 |
|
| 340 |
.. seealso:: |
| 341 |
|
| 342 |
The :ref:`Concurrency and multithreading in asyncio |
| 343 |
<asyncio-multithreading>` section. |
| 344 |
|
| 345 |
|
| 346 |
Examples |
| 347 |
-------- |
| 348 |
|
| 349 |
An example using the :class:`~asyncio.subprocess.Process` class to |
| 350 |
control a subprocess and the :class:`StreamReader` class to read from |
| 351 |
its standard output. |
| 352 |
|
| 353 |
.. _asyncio_example_create_subprocess_exec: |
| 354 |
|
| 355 |
The subprocess is created by the :func:`create_subprocess_exec` |
| 356 |
function:: |
| 357 |
|
| 358 |
import asyncio |
| 359 |
import sys |
| 360 |
|
| 361 |
async def get_date(): |
| 362 |
code = 'import datetime as dt; print(dt.datetime.now())' |
| 363 |
|
| 364 |
# Create the subprocess; redirect the standard output |
| 365 |
# into a pipe. |
| 366 |
proc = await asyncio.create_subprocess_exec( |
| 367 |
sys.executable, '-c', code, |
| 368 |
stdout=asyncio.subprocess.PIPE) |
| 369 |
|
| 370 |
# Read one line of output. |
| 371 |
data = await proc.stdout.readline() |
| 372 |
line = data.decode('ascii').rstrip() |
| 373 |
|
| 374 |
# Wait for the subprocess exit. |
| 375 |
await proc.wait() |
| 376 |
return line |
| 377 |
|
| 378 |
date = asyncio.run(get_date()) |
| 379 |
print(f"Current date: {date}") |
| 380 |
|
| 381 |
|
| 382 |
See also the :ref:`same example <asyncio_example_subprocess_proto>` |
| 383 |
written using low-level APIs. |