| 1 |
:mod:`!concurrent.futures` --- Launching parallel tasks |
| 2 |
======================================================= |
| 3 |
|
| 4 |
.. module:: concurrent.futures |
| 5 |
:synopsis: Execute computations concurrently using threads or processes. |
| 6 |
|
| 7 |
.. versionadded:: 3.2 |
| 8 |
|
| 9 |
**Source code:** :source:`Lib/concurrent/futures/thread.py`, |
| 10 |
:source:`Lib/concurrent/futures/process.py`, |
| 11 |
and :source:`Lib/concurrent/futures/interpreter.py` |
| 12 |
|
| 13 |
-------------- |
| 14 |
|
| 15 |
The :mod:`!concurrent.futures` module provides a high-level interface for |
| 16 |
asynchronously executing callables. |
| 17 |
|
| 18 |
The asynchronous execution can be performed with threads, using |
| 19 |
:class:`ThreadPoolExecutor` or :class:`InterpreterPoolExecutor`, |
| 20 |
or separate processes, using :class:`ProcessPoolExecutor`. |
| 21 |
Each implements the same interface, which is defined |
| 22 |
by the abstract :class:`Executor` class. |
| 23 |
|
| 24 |
:class:`concurrent.futures.Future` must not be confused with |
| 25 |
:class:`asyncio.Future`, which is designed for use with :mod:`asyncio` |
| 26 |
tasks and coroutines. See the :doc:`asyncio's Future <asyncio-future>` |
| 27 |
documentation for a detailed comparison of the two. |
| 28 |
|
| 29 |
.. include:: ../includes/wasm-notavail.rst |
| 30 |
|
| 31 |
Executor Objects |
| 32 |
---------------- |
| 33 |
|
| 34 |
.. class:: Executor |
| 35 |
|
| 36 |
An abstract class that provides methods to execute calls asynchronously. It |
| 37 |
should not be used directly, but through its concrete subclasses. |
| 38 |
|
| 39 |
.. method:: submit(fn, /, *args, **kwargs) |
| 40 |
|
| 41 |
Schedules the callable, *fn*, to be executed as ``fn(*args, **kwargs)`` |
| 42 |
and returns a :class:`Future` object representing the execution of the |
| 43 |
callable. :: |
| 44 |
|
| 45 |
with ThreadPoolExecutor(max_workers=1) as executor: |
| 46 |
future = executor.submit(pow, 323, 1235) |
| 47 |
print(future.result()) |
| 48 |
|
| 49 |
.. method:: map(fn, *iterables, timeout=None, chunksize=1, buffersize=None) |
| 50 |
|
| 51 |
Similar to :func:`map(fn, *iterables) <map>` except: |
| 52 |
|
| 53 |
* The *iterables* are collected immediately rather than lazily, unless a |
| 54 |
*buffersize* is specified to limit the number of submitted tasks whose |
| 55 |
results have not yet been yielded. If the buffer is full, iteration over |
| 56 |
the *iterables* pauses until a result is yielded from the buffer. |
| 57 |
|
| 58 |
* *fn* is executed asynchronously and several calls to |
| 59 |
*fn* may be made concurrently. |
| 60 |
|
| 61 |
The returned iterator raises a :exc:`TimeoutError` |
| 62 |
if :meth:`~iterator.__next__` is called and the result isn't available |
| 63 |
after *timeout* seconds from the original call to :meth:`Executor.map`. |
| 64 |
*timeout* can be an int or a float. If *timeout* is not specified or |
| 65 |
``None``, there is no limit to the wait time. |
| 66 |
|
| 67 |
If a *fn* call raises an exception, then that exception will be |
| 68 |
raised when its value is retrieved from the iterator. |
| 69 |
|
| 70 |
When using :class:`ProcessPoolExecutor`, this method chops *iterables* |
| 71 |
into a number of chunks which it submits to the pool as separate |
| 72 |
tasks. The (approximate) size of these chunks can be specified by |
| 73 |
setting *chunksize* to a positive integer. For very long iterables, |
| 74 |
using a large value for *chunksize* can significantly improve |
| 75 |
performance compared to the default size of 1. With |
| 76 |
:class:`ThreadPoolExecutor` and :class:`InterpreterPoolExecutor`, |
| 77 |
*chunksize* has no effect. |
| 78 |
|
| 79 |
.. versionchanged:: 3.5 |
| 80 |
Added the *chunksize* parameter. |
| 81 |
|
| 82 |
.. versionchanged:: 3.14 |
| 83 |
Added the *buffersize* parameter. |
| 84 |
|
| 85 |
.. method:: shutdown(wait=True, *, cancel_futures=False) |
| 86 |
|
| 87 |
Signal the executor that it should free any resources that it is using |
| 88 |
when the currently pending futures are done executing. Calls to |
| 89 |
:meth:`Executor.submit` and :meth:`Executor.map` made after shutdown will |
| 90 |
raise :exc:`RuntimeError`. |
| 91 |
|
| 92 |
If *wait* is ``True`` then this method will not return until all the |
| 93 |
pending futures are done executing and the resources associated with the |
| 94 |
executor have been freed. If *wait* is ``False`` then this method will |
| 95 |
return immediately and the resources associated with the executor will be |
| 96 |
freed when all pending futures are done executing. Regardless of the |
| 97 |
value of *wait*, the entire Python program will not exit until all |
| 98 |
pending futures are done executing. |
| 99 |
|
| 100 |
If *cancel_futures* is ``True``, this method will cancel all pending |
| 101 |
futures that the executor has not started running. Any futures that |
| 102 |
are completed or running won't be cancelled, regardless of the value |
| 103 |
of *cancel_futures*. |
| 104 |
|
| 105 |
If both *cancel_futures* and *wait* are ``True``, all futures that the |
| 106 |
executor has started running will be completed prior to this method |
| 107 |
returning. The remaining futures are cancelled. |
| 108 |
|
| 109 |
You can avoid having to call this method explicitly if you use the executor |
| 110 |
as a :term:`context manager` via the :keyword:`with` statement, which |
| 111 |
will shutdown the :class:`Executor` (waiting as if :meth:`Executor.shutdown` |
| 112 |
were called with *wait* set to ``True``):: |
| 113 |
|
| 114 |
import shutil |
| 115 |
with ThreadPoolExecutor(max_workers=4) as e: |
| 116 |
e.submit(shutil.copy, 'src1.txt', 'dest1.txt') |
| 117 |
e.submit(shutil.copy, 'src2.txt', 'dest2.txt') |
| 118 |
e.submit(shutil.copy, 'src3.txt', 'dest3.txt') |
| 119 |
e.submit(shutil.copy, 'src4.txt', 'dest4.txt') |
| 120 |
|
| 121 |
.. versionchanged:: 3.9 |
| 122 |
Added *cancel_futures*. |
| 123 |
|
| 124 |
|
| 125 |
ThreadPoolExecutor |
| 126 |
------------------ |
| 127 |
|
| 128 |
:class:`ThreadPoolExecutor` is an :class:`Executor` subclass that uses a pool of |
| 129 |
threads to execute calls asynchronously. |
| 130 |
|
| 131 |
Deadlocks can occur when the callable associated with a :class:`Future` waits on |
| 132 |
the results of another :class:`Future`. For example:: |
| 133 |
|
| 134 |
import time |
| 135 |
def wait_on_b(): |
| 136 |
time.sleep(5) |
| 137 |
print(b.result()) # b will never complete because it is waiting on a. |
| 138 |
return 5 |
| 139 |
|
| 140 |
def wait_on_a(): |
| 141 |
time.sleep(5) |
| 142 |
print(a.result()) # a will never complete because it is waiting on b. |
| 143 |
return 6 |
| 144 |
|
| 145 |
|
| 146 |
executor = ThreadPoolExecutor(max_workers=2) |
| 147 |
a = executor.submit(wait_on_b) |
| 148 |
b = executor.submit(wait_on_a) |
| 149 |
|
| 150 |
And:: |
| 151 |
|
| 152 |
def wait_on_future(): |
| 153 |
f = executor.submit(pow, 5, 2) |
| 154 |
# This will never complete because there is only one worker thread and |
| 155 |
# it is executing this function. |
| 156 |
print(f.result()) |
| 157 |
|
| 158 |
executor = ThreadPoolExecutor(max_workers=1) |
| 159 |
future = executor.submit(wait_on_future) |
| 160 |
# Note: calling future.result() would also cause a deadlock because |
| 161 |
# the single worker thread is already waiting for wait_on_future(). |
| 162 |
|
| 163 |
|
| 164 |
.. class:: ThreadPoolExecutor(max_workers=None, thread_name_prefix='', initializer=None, initargs=()) |
| 165 |
|
| 166 |
An :class:`Executor` subclass that uses a pool of at most *max_workers* |
| 167 |
threads to execute calls asynchronously. |
| 168 |
|
| 169 |
All threads enqueued to ``ThreadPoolExecutor`` will be joined before the |
| 170 |
interpreter can exit. Note that the exit handler which does this is |
| 171 |
executed *before* any exit handlers added using ``atexit``. This means |
| 172 |
exceptions in the main thread must be caught and handled in order to |
| 173 |
signal threads to exit gracefully. For this reason, it is recommended |
| 174 |
that ``ThreadPoolExecutor`` not be used for long-running tasks. |
| 175 |
|
| 176 |
*initializer* is an optional callable that is called at the start of |
| 177 |
each worker thread; *initargs* is a tuple of arguments passed to the |
| 178 |
initializer. Should *initializer* raise an exception, all currently |
| 179 |
pending jobs will raise a :exc:`~concurrent.futures.thread.BrokenThreadPool`, |
| 180 |
as well as any attempt to submit more jobs to the pool. |
| 181 |
|
| 182 |
.. versionchanged:: 3.5 |
| 183 |
If *max_workers* is ``None`` or |
| 184 |
not given, it will default to the number of processors on the machine, |
| 185 |
multiplied by ``5``, assuming that :class:`ThreadPoolExecutor` is often |
| 186 |
used to overlap I/O instead of CPU work and the number of workers |
| 187 |
should be higher than the number of workers |
| 188 |
for :class:`ProcessPoolExecutor`. |
| 189 |
|
| 190 |
.. versionchanged:: 3.6 |
| 191 |
Added the *thread_name_prefix* parameter to allow users to |
| 192 |
control the :class:`threading.Thread` names for worker threads created by |
| 193 |
the pool for easier debugging. |
| 194 |
|
| 195 |
.. versionchanged:: 3.7 |
| 196 |
Added the *initializer* and *initargs* arguments. |
| 197 |
|
| 198 |
.. versionchanged:: 3.8 |
| 199 |
Default value of *max_workers* is changed to ``min(32, os.cpu_count() + 4)``. |
| 200 |
This default value preserves at least 5 workers for I/O bound tasks. |
| 201 |
It utilizes at most 32 CPU cores for CPU bound tasks which release the GIL. |
| 202 |
And it avoids using very large resources implicitly on many-core machines. |
| 203 |
|
| 204 |
ThreadPoolExecutor now reuses idle worker threads before starting |
| 205 |
*max_workers* worker threads too. |
| 206 |
|
| 207 |
.. versionchanged:: 3.13 |
| 208 |
Default value of *max_workers* is changed to |
| 209 |
``min(32, (os.process_cpu_count() or 1) + 4)``. |
| 210 |
|
| 211 |
|
| 212 |
.. _threadpoolexecutor-example: |
| 213 |
|
| 214 |
ThreadPoolExecutor Example |
| 215 |
~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 216 |
:: |
| 217 |
|
| 218 |
import concurrent.futures |
| 219 |
import urllib.request |
| 220 |
|
| 221 |
URLS = ['http://www.foxnews.com/', |
| 222 |
'http://www.cnn.com/', |
| 223 |
'http://europe.wsj.com/', |
| 224 |
'http://www.bbc.co.uk/', |
| 225 |
'http://nonexistent-subdomain.python.org/'] |
| 226 |
|
| 227 |
# Retrieve a single page and report the URL and contents |
| 228 |
def load_url(url, timeout): |
| 229 |
with urllib.request.urlopen(url, timeout=timeout) as conn: |
| 230 |
return conn.read() |
| 231 |
|
| 232 |
# We can use a with statement to ensure threads are cleaned up promptly |
| 233 |
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor: |
| 234 |
# Start the load operations and mark each future with its URL |
| 235 |
future_to_url = {executor.submit(load_url, url, 60): url for url in URLS} |
| 236 |
for future in concurrent.futures.as_completed(future_to_url): |
| 237 |
url = future_to_url[future] |
| 238 |
try: |
| 239 |
data = future.result() |
| 240 |
except Exception as exc: |
| 241 |
print('%r generated an exception: %s' % (url, exc)) |
| 242 |
else: |
| 243 |
print('%r page is %d bytes' % (url, len(data))) |
| 244 |
|
| 245 |
|
| 246 |
InterpreterPoolExecutor |
| 247 |
----------------------- |
| 248 |
|
| 249 |
.. versionadded:: 3.14 |
| 250 |
|
| 251 |
The :class:`InterpreterPoolExecutor` class uses a pool of interpreters |
| 252 |
to execute calls asynchronously. It is a :class:`ThreadPoolExecutor` |
| 253 |
subclass, which means each worker is running in its own thread. |
| 254 |
The difference here is that each worker has its own interpreter, |
| 255 |
and runs each task using that interpreter. |
| 256 |
|
| 257 |
The biggest benefit to using interpreters instead of only threads |
| 258 |
is true multi-core parallelism. Each interpreter has its own |
| 259 |
:term:`Global Interpreter Lock <global interpreter lock>`, so code |
| 260 |
running in one interpreter can run on one CPU core, while code in |
| 261 |
another interpreter runs unblocked on a different core. |
| 262 |
|
| 263 |
The tradeoff is that writing concurrent code for use with multiple |
| 264 |
interpreters can take extra effort. However, this is because it |
| 265 |
forces you to be deliberate about how and when interpreters interact, |
| 266 |
and to be explicit about what data is shared between interpreters. |
| 267 |
This results in several benefits that help balance the extra effort, |
| 268 |
including true multi-core parallelism, For example, code written |
| 269 |
this way can make it easier to reason about concurrency. Another |
| 270 |
major benefit is that you don't have to deal with several of the |
| 271 |
big pain points of using threads, like race conditions. |
| 272 |
|
| 273 |
Each worker's interpreter is isolated from all the other interpreters. |
| 274 |
"Isolated" means each interpreter has its own runtime state and |
| 275 |
operates completely independently. For example, if you redirect |
| 276 |
:data:`sys.stdout` in one interpreter, it will not be automatically |
| 277 |
redirected to any other interpreter. If you import a module in one |
| 278 |
interpreter, it is not automatically imported in any other. You |
| 279 |
would need to import the module separately in interpreter where |
| 280 |
you need it. In fact, each module imported in an interpreter is |
| 281 |
a completely separate object from the same module in a different |
| 282 |
interpreter, including :mod:`sys`, :mod:`builtins`, |
| 283 |
and even ``__main__``. |
| 284 |
|
| 285 |
Isolation means a mutable object, or other data, cannot be used |
| 286 |
by more than one interpreter at the same time. That effectively means |
| 287 |
interpreters cannot actually share such objects or data. Instead, |
| 288 |
each interpreter must have its own copy, and you will have to |
| 289 |
synchronize any changes between the copies manually. Immutable |
| 290 |
objects and data, like the builtin singletons, strings, and tuples |
| 291 |
of immutable objects, don't have these limitations. |
| 292 |
|
| 293 |
Communicating and synchronizing between interpreters is most effectively |
| 294 |
done using dedicated tools, like those proposed in :pep:`734`. One less |
| 295 |
efficient alternative is to serialize with :mod:`pickle` and then send |
| 296 |
the bytes over a shared :mod:`socket <socket>` or |
| 297 |
:func:`pipe <os.pipe>`. |
| 298 |
|
| 299 |
.. class:: InterpreterPoolExecutor(max_workers=None, thread_name_prefix='', initializer=None, initargs=()) |
| 300 |
|
| 301 |
A :class:`ThreadPoolExecutor` subclass that executes calls asynchronously |
| 302 |
using a pool of at most *max_workers* threads. Each thread runs |
| 303 |
tasks in its own interpreter. The worker interpreters are isolated |
| 304 |
from each other, which means each has its own runtime state and that |
| 305 |
they can't share any mutable objects or other data. Each interpreter |
| 306 |
has its own :term:`Global Interpreter Lock <global interpreter lock>`, |
| 307 |
which means code run with this executor has true multi-core parallelism. |
| 308 |
|
| 309 |
The optional *initializer* and *initargs* arguments have the same |
| 310 |
meaning as for :class:`!ThreadPoolExecutor`: the initializer is run |
| 311 |
when each worker is created, though in this case it is run in |
| 312 |
the worker's interpreter. The executor serializes the *initializer* |
| 313 |
and *initargs* using :mod:`pickle` when sending them to the worker's |
| 314 |
interpreter. |
| 315 |
|
| 316 |
.. note:: |
| 317 |
The executor may replace uncaught exceptions from *initializer* |
| 318 |
with :class:`~concurrent.interpreters.ExecutionFailed`. |
| 319 |
|
| 320 |
Other caveats from parent :class:`ThreadPoolExecutor` apply here. |
| 321 |
|
| 322 |
:meth:`~Executor.submit` and :meth:`~Executor.map` work like normal, |
| 323 |
except the worker serializes the callable and arguments using |
| 324 |
:mod:`pickle` when sending them to its interpreter. The worker |
| 325 |
likewise serializes the return value when sending it back. |
| 326 |
|
| 327 |
When a worker's current task raises an uncaught exception, the worker |
| 328 |
always tries to preserve the exception as-is. If that is successful |
| 329 |
then it also sets the ``__cause__`` to a corresponding |
| 330 |
:class:`~concurrent.interpreters.ExecutionFailed` |
| 331 |
instance, which contains a summary of the original exception. |
| 332 |
In the uncommon case that the worker is not able to preserve the |
| 333 |
original as-is then it directly preserves the corresponding |
| 334 |
:class:`~concurrent.interpreters.ExecutionFailed` |
| 335 |
instance instead. |
| 336 |
|
| 337 |
|
| 338 |
ProcessPoolExecutor |
| 339 |
------------------- |
| 340 |
|
| 341 |
The :class:`ProcessPoolExecutor` class is an :class:`Executor` subclass that |
| 342 |
uses a pool of processes to execute calls asynchronously. |
| 343 |
:class:`ProcessPoolExecutor` uses the :mod:`multiprocessing` module, which |
| 344 |
allows it to side-step the :term:`Global Interpreter Lock |
| 345 |
<global interpreter lock>` but also means that |
| 346 |
only picklable objects can be executed and returned. |
| 347 |
|
| 348 |
The ``__main__`` module must be importable by worker subprocesses. This means |
| 349 |
that :class:`ProcessPoolExecutor` will not work in the interactive interpreter. |
| 350 |
|
| 351 |
Calling :class:`Executor` or :class:`Future` methods from a callable submitted |
| 352 |
to a :class:`ProcessPoolExecutor` will result in deadlock. |
| 353 |
|
| 354 |
Note that the restrictions on functions and arguments needing to picklable as |
| 355 |
per :class:`multiprocessing.Process` apply when using :meth:`~Executor.submit` |
| 356 |
and :meth:`~Executor.map` on a :class:`ProcessPoolExecutor`. A function defined |
| 357 |
in a REPL or a lambda should not be expected to work. |
| 358 |
|
| 359 |
.. class:: ProcessPoolExecutor(max_workers=None, mp_context=None, initializer=None, initargs=(), max_tasks_per_child=None) |
| 360 |
|
| 361 |
An :class:`Executor` subclass that executes calls asynchronously using a pool |
| 362 |
of at most *max_workers* processes. If *max_workers* is ``None`` or not |
| 363 |
given, it will default to :func:`os.process_cpu_count`. |
| 364 |
If *max_workers* is less than or equal to ``0``, then a :exc:`ValueError` |
| 365 |
will be raised. |
| 366 |
On Windows, *max_workers* must be less than or equal to ``61``. If it is not |
| 367 |
then :exc:`ValueError` will be raised. If *max_workers* is ``None``, then |
| 368 |
the default chosen will be at most ``61``, even if more processors are |
| 369 |
available. |
| 370 |
*mp_context* can be a :mod:`multiprocessing` context or ``None``. It will be |
| 371 |
used to launch the workers. If *mp_context* is ``None`` or not given, the |
| 372 |
default :mod:`multiprocessing` context is used. |
| 373 |
See :ref:`multiprocessing-start-methods`. |
| 374 |
|
| 375 |
*initializer* is an optional callable that is called at the start of |
| 376 |
each worker process; *initargs* is a tuple of arguments passed to the |
| 377 |
initializer. Should *initializer* raise an exception, all currently |
| 378 |
pending jobs will raise a :exc:`~concurrent.futures.process.BrokenProcessPool`, |
| 379 |
as well as any attempt to submit more jobs to the pool. |
| 380 |
|
| 381 |
*max_tasks_per_child* is an optional argument that specifies the maximum |
| 382 |
number of tasks a single process can execute before it will exit and be |
| 383 |
replaced with a fresh worker process. By default *max_tasks_per_child* is |
| 384 |
``None`` which means worker processes will live as long as the pool. When |
| 385 |
a max is specified, the "spawn" multiprocessing start method will be used by |
| 386 |
default in absence of a *mp_context* parameter. This feature is incompatible |
| 387 |
with the "fork" start method. |
| 388 |
|
| 389 |
.. note:: |
| 390 |
Bugs have been reported when using the *max_tasks_per_child* feature that |
| 391 |
can result in the :class:`ProcessPoolExecutor` hanging in some |
| 392 |
circumstances. Follow its eventual resolution in :gh:`115634`. |
| 393 |
|
| 394 |
.. versionchanged:: 3.3 |
| 395 |
When one of the worker processes terminates abruptly, a |
| 396 |
:exc:`~concurrent.futures.process.BrokenProcessPool` error is now raised. |
| 397 |
Previously, behaviour |
| 398 |
was undefined but operations on the executor or its futures would often |
| 399 |
freeze or deadlock. |
| 400 |
|
| 401 |
.. versionchanged:: 3.7 |
| 402 |
The *mp_context* argument was added to allow users to control the |
| 403 |
start_method for worker processes created by the pool. |
| 404 |
|
| 405 |
Added the *initializer* and *initargs* arguments. |
| 406 |
|
| 407 |
.. versionchanged:: 3.11 |
| 408 |
The *max_tasks_per_child* argument was added to allow users to |
| 409 |
control the lifetime of workers in the pool. |
| 410 |
|
| 411 |
.. versionchanged:: 3.12 |
| 412 |
On POSIX systems, if your application has multiple threads and the |
| 413 |
:mod:`multiprocessing` context uses the ``"fork"`` start method: |
| 414 |
The :func:`os.fork` function called internally to spawn workers may raise a |
| 415 |
:exc:`DeprecationWarning`. Pass a *mp_context* configured to use a |
| 416 |
different start method. See the :func:`os.fork` documentation for |
| 417 |
further explanation. |
| 418 |
|
| 419 |
.. versionchanged:: 3.13 |
| 420 |
*max_workers* uses :func:`os.process_cpu_count` by default, instead of |
| 421 |
:func:`os.cpu_count`. |
| 422 |
|
| 423 |
.. versionchanged:: 3.14 |
| 424 |
The default process start method (see |
| 425 |
:ref:`multiprocessing-start-methods`) changed away from *fork*. If you |
| 426 |
require the *fork* start method for :class:`ProcessPoolExecutor` you must |
| 427 |
explicitly pass ``mp_context=multiprocessing.get_context("fork")``. |
| 428 |
|
| 429 |
.. method:: terminate_workers() |
| 430 |
|
| 431 |
Attempt to terminate all living worker processes immediately by calling |
| 432 |
:meth:`Process.terminate <multiprocessing.Process.terminate>` on each of them. |
| 433 |
Internally, it will also call :meth:`Executor.shutdown` to ensure that all |
| 434 |
other resources associated with the executor are freed. |
| 435 |
|
| 436 |
After calling this method the caller should no longer submit tasks to the |
| 437 |
executor. |
| 438 |
|
| 439 |
.. versionadded:: 3.14 |
| 440 |
|
| 441 |
.. method:: kill_workers() |
| 442 |
|
| 443 |
Attempt to kill all living worker processes immediately by calling |
| 444 |
:meth:`Process.kill <multiprocessing.Process.kill>` on each of them. |
| 445 |
Internally, it will also call :meth:`Executor.shutdown` to ensure that all |
| 446 |
other resources associated with the executor are freed. |
| 447 |
|
| 448 |
After calling this method the caller should no longer submit tasks to the |
| 449 |
executor. |
| 450 |
|
| 451 |
.. versionadded:: 3.14 |
| 452 |
|
| 453 |
.. _processpoolexecutor-example: |
| 454 |
|
| 455 |
ProcessPoolExecutor Example |
| 456 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 457 |
:: |
| 458 |
|
| 459 |
import concurrent.futures |
| 460 |
import math |
| 461 |
|
| 462 |
PRIMES = [ |
| 463 |
112272535095293, |
| 464 |
112582705942171, |
| 465 |
112272535095293, |
| 466 |
115280095190773, |
| 467 |
115797848077099, |
| 468 |
1099726899285419] |
| 469 |
|
| 470 |
def is_prime(n): |
| 471 |
if n < 2: |
| 472 |
return False |
| 473 |
if n == 2: |
| 474 |
return True |
| 475 |
if n % 2 == 0: |
| 476 |
return False |
| 477 |
|
| 478 |
sqrt_n = int(math.floor(math.sqrt(n))) |
| 479 |
for i in range(3, sqrt_n + 1, 2): |
| 480 |
if n % i == 0: |
| 481 |
return False |
| 482 |
return True |
| 483 |
|
| 484 |
def main(): |
| 485 |
with concurrent.futures.ProcessPoolExecutor() as executor: |
| 486 |
for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)): |
| 487 |
print('%d is prime: %s' % (number, prime)) |
| 488 |
|
| 489 |
if __name__ == '__main__': |
| 490 |
main() |
| 491 |
|
| 492 |
|
| 493 |
Future Objects |
| 494 |
-------------- |
| 495 |
|
| 496 |
The :class:`Future` class encapsulates the asynchronous execution of a callable. |
| 497 |
:class:`Future` instances are created by :meth:`Executor.submit`. |
| 498 |
|
| 499 |
.. class:: Future |
| 500 |
|
| 501 |
Encapsulates the asynchronous execution of a callable. :class:`Future` |
| 502 |
instances are created by :meth:`Executor.submit` and should not be created |
| 503 |
directly except for testing. |
| 504 |
|
| 505 |
.. method:: cancel() |
| 506 |
|
| 507 |
Attempt to cancel the call. If the call is currently being executed or |
| 508 |
finished running and cannot be cancelled then the method will return |
| 509 |
``False``, otherwise the call will be cancelled and the method will |
| 510 |
return ``True``. |
| 511 |
|
| 512 |
.. method:: cancelled() |
| 513 |
|
| 514 |
Return ``True`` if the call was successfully cancelled. |
| 515 |
|
| 516 |
.. method:: running() |
| 517 |
|
| 518 |
Return ``True`` if the call is currently being executed and cannot be |
| 519 |
cancelled. |
| 520 |
|
| 521 |
.. method:: done() |
| 522 |
|
| 523 |
Return ``True`` if the call was successfully cancelled or finished |
| 524 |
running. |
| 525 |
|
| 526 |
.. method:: result(timeout=None) |
| 527 |
|
| 528 |
Return the value returned by the call. If the call hasn't yet completed |
| 529 |
then this method will wait up to *timeout* seconds. If the call hasn't |
| 530 |
completed in *timeout* seconds, then a |
| 531 |
:exc:`TimeoutError` will be raised. *timeout* can be |
| 532 |
an int or float. If *timeout* is not specified or ``None``, there is no |
| 533 |
limit to the wait time. |
| 534 |
|
| 535 |
If the future is cancelled before completing then :exc:`.CancelledError` |
| 536 |
will be raised. |
| 537 |
|
| 538 |
If the call raised an exception, this method will raise the same exception. |
| 539 |
|
| 540 |
.. method:: exception(timeout=None) |
| 541 |
|
| 542 |
Return the exception raised by the call. If the call hasn't yet |
| 543 |
completed then this method will wait up to *timeout* seconds. If the |
| 544 |
call hasn't completed in *timeout* seconds, then a |
| 545 |
:exc:`TimeoutError` will be raised. *timeout* can be |
| 546 |
an int or float. If *timeout* is not specified or ``None``, there is no |
| 547 |
limit to the wait time. |
| 548 |
|
| 549 |
If the future is cancelled before completing then :exc:`.CancelledError` |
| 550 |
will be raised. |
| 551 |
|
| 552 |
If the call completed without raising, ``None`` is returned. |
| 553 |
|
| 554 |
.. method:: add_done_callback(fn) |
| 555 |
|
| 556 |
Attaches the callable *fn* to the future. *fn* will be called, with the |
| 557 |
future as its only argument, when the future is cancelled or finishes |
| 558 |
running. |
| 559 |
|
| 560 |
Added callables are called in the order that they were added and are |
| 561 |
always called in a thread belonging to the process that added them. If |
| 562 |
the callable raises an :exc:`Exception` subclass, it will be logged and |
| 563 |
ignored. If the callable raises a :exc:`BaseException` subclass, the |
| 564 |
behavior is undefined. |
| 565 |
|
| 566 |
If the future has already completed or been cancelled, *fn* will be |
| 567 |
called immediately. |
| 568 |
|
| 569 |
The following :class:`Future` methods are meant for use in unit tests and |
| 570 |
:class:`Executor` implementations. |
| 571 |
|
| 572 |
.. method:: set_running_or_notify_cancel() |
| 573 |
|
| 574 |
This method should only be called by :class:`Executor` implementations |
| 575 |
before executing the work associated with the :class:`Future` and by unit |
| 576 |
tests. |
| 577 |
|
| 578 |
If the method returns ``False`` then the :class:`Future` was cancelled, |
| 579 |
i.e. :meth:`Future.cancel` was called and returned ``True``. Any threads |
| 580 |
waiting on the :class:`Future` completing (i.e. through |
| 581 |
:func:`as_completed` or :func:`wait`) will be woken up. |
| 582 |
|
| 583 |
If the method returns ``True`` then the :class:`Future` was not cancelled |
| 584 |
and has been put in the running state, i.e. calls to |
| 585 |
:meth:`Future.running` will return ``True``. |
| 586 |
|
| 587 |
This method can only be called once and cannot be called after |
| 588 |
:meth:`Future.set_result` or :meth:`Future.set_exception` have been |
| 589 |
called. |
| 590 |
|
| 591 |
.. method:: set_result(result) |
| 592 |
|
| 593 |
Sets the result of the work associated with the :class:`Future` to |
| 594 |
*result*. |
| 595 |
|
| 596 |
This method should only be used by :class:`Executor` implementations and |
| 597 |
unit tests. |
| 598 |
|
| 599 |
.. versionchanged:: 3.8 |
| 600 |
This method raises |
| 601 |
:exc:`concurrent.futures.InvalidStateError` if the :class:`Future` is |
| 602 |
already done. |
| 603 |
|
| 604 |
.. method:: set_exception(exception) |
| 605 |
|
| 606 |
Sets the result of the work associated with the :class:`Future` to the |
| 607 |
:class:`Exception` *exception*. |
| 608 |
|
| 609 |
This method should only be used by :class:`Executor` implementations and |
| 610 |
unit tests. |
| 611 |
|
| 612 |
.. versionchanged:: 3.8 |
| 613 |
This method raises |
| 614 |
:exc:`concurrent.futures.InvalidStateError` if the :class:`Future` is |
| 615 |
already done. |
| 616 |
|
| 617 |
Module Functions |
| 618 |
---------------- |
| 619 |
|
| 620 |
.. function:: wait(fs, timeout=None, return_when=ALL_COMPLETED) |
| 621 |
|
| 622 |
Wait for the :class:`Future` instances (possibly created by different |
| 623 |
:class:`Executor` instances) given by *fs* to complete. Duplicate futures |
| 624 |
given to *fs* are removed and will be returned only once. Returns a named |
| 625 |
2-tuple of sets. The first set, named ``done``, contains the futures that |
| 626 |
completed (finished or cancelled futures) before the wait completed. The |
| 627 |
second set, named ``not_done``, contains the futures that did not complete |
| 628 |
(pending or running futures). |
| 629 |
|
| 630 |
*timeout* can be used to control the maximum number of seconds to wait before |
| 631 |
returning. *timeout* can be an int or float. If *timeout* is not specified |
| 632 |
or ``None``, there is no limit to the wait time. |
| 633 |
|
| 634 |
*return_when* indicates when this function should return. It must be one of |
| 635 |
the following constants: |
| 636 |
|
| 637 |
.. list-table:: |
| 638 |
:header-rows: 1 |
| 639 |
|
| 640 |
* - Constant |
| 641 |
- Description |
| 642 |
|
| 643 |
* - .. data:: FIRST_COMPLETED |
| 644 |
- The function will return when any future finishes or is cancelled. |
| 645 |
|
| 646 |
* - .. data:: FIRST_EXCEPTION |
| 647 |
- The function will return when any future finishes by raising an |
| 648 |
exception. If no future raises an exception |
| 649 |
then it is equivalent to :const:`ALL_COMPLETED`. |
| 650 |
|
| 651 |
* - .. data:: ALL_COMPLETED |
| 652 |
- The function will return when all futures finish or are cancelled. |
| 653 |
|
| 654 |
.. function:: as_completed(fs, timeout=None) |
| 655 |
|
| 656 |
Returns an iterator over the :class:`Future` instances (possibly created by |
| 657 |
different :class:`Executor` instances) given by *fs* that yields futures as |
| 658 |
they complete (finished or cancelled futures). Any futures given by *fs* that |
| 659 |
are duplicated will be returned once. Any futures that completed before |
| 660 |
:func:`as_completed` is called will be yielded first. The returned iterator |
| 661 |
raises a :exc:`TimeoutError` if :meth:`~iterator.__next__` |
| 662 |
is called and the result isn't available after *timeout* seconds from the |
| 663 |
original call to :func:`as_completed`. *timeout* can be an int or float. If |
| 664 |
*timeout* is not specified or ``None``, there is no limit to the wait time. |
| 665 |
|
| 666 |
|
| 667 |
.. seealso:: |
| 668 |
|
| 669 |
:pep:`3148` -- futures - execute computations asynchronously |
| 670 |
The proposal which described this feature for inclusion in the Python |
| 671 |
standard library. |
| 672 |
|
| 673 |
|
| 674 |
Exception classes |
| 675 |
----------------- |
| 676 |
|
| 677 |
.. currentmodule:: concurrent.futures |
| 678 |
|
| 679 |
.. exception:: CancelledError |
| 680 |
|
| 681 |
Raised when a future is cancelled. |
| 682 |
|
| 683 |
.. exception:: TimeoutError |
| 684 |
|
| 685 |
A deprecated alias of :exc:`TimeoutError`, |
| 686 |
raised when a future operation exceeds the given timeout. |
| 687 |
|
| 688 |
.. versionchanged:: 3.11 |
| 689 |
|
| 690 |
This class was made an alias of :exc:`TimeoutError`. |
| 691 |
|
| 692 |
|
| 693 |
.. exception:: BrokenExecutor |
| 694 |
|
| 695 |
Derived from :exc:`RuntimeError`, this exception class is raised |
| 696 |
when an executor is broken for some reason, and cannot be used |
| 697 |
to submit or execute new tasks. |
| 698 |
|
| 699 |
.. versionadded:: 3.7 |
| 700 |
|
| 701 |
.. exception:: InvalidStateError |
| 702 |
|
| 703 |
Raised when an operation is performed on a future that is not allowed |
| 704 |
in the current state. |
| 705 |
|
| 706 |
.. versionadded:: 3.8 |
| 707 |
|
| 708 |
.. currentmodule:: concurrent.futures.thread |
| 709 |
|
| 710 |
.. exception:: BrokenThreadPool |
| 711 |
|
| 712 |
Derived from :exc:`~concurrent.futures.BrokenExecutor`, this exception |
| 713 |
class is raised when one of the workers |
| 714 |
of a :class:`~concurrent.futures.ThreadPoolExecutor` |
| 715 |
has failed initializing. |
| 716 |
|
| 717 |
.. versionadded:: 3.7 |
| 718 |
|
| 719 |
.. currentmodule:: concurrent.futures.interpreter |
| 720 |
|
| 721 |
.. exception:: BrokenInterpreterPool |
| 722 |
|
| 723 |
Derived from :exc:`~concurrent.futures.thread.BrokenThreadPool`, |
| 724 |
this exception class is raised when one of the workers |
| 725 |
of a :class:`~concurrent.futures.InterpreterPoolExecutor` |
| 726 |
has failed initializing. |
| 727 |
|
| 728 |
.. versionadded:: 3.14 |
| 729 |
|
| 730 |
.. currentmodule:: concurrent.futures.process |
| 731 |
|
| 732 |
.. exception:: BrokenProcessPool |
| 733 |
|
| 734 |
Derived from :exc:`~concurrent.futures.BrokenExecutor` (formerly |
| 735 |
:exc:`RuntimeError`), this exception class is raised when one of the |
| 736 |
workers of a :class:`~concurrent.futures.ProcessPoolExecutor` |
| 737 |
has terminated in a non-clean |
| 738 |
fashion (for example, if it was killed from the outside). |
| 739 |
|
| 740 |
.. versionadded:: 3.3 |