Debian Codegraph @

Doc/library/concurrent.futures.rst

raw
1 :mod:`!concurrent.futures` --- Launching parallel tasks
2 =======================================================
3
4 .. module:: concurrent.futures
5 :synopsis: Execute computations concurrently using threads or processes.
6
7 .. versionadded:: 3.2
8
9 **Source code:** :source:`Lib/concurrent/futures/thread.py`
10 and :source:`Lib/concurrent/futures/process.py`
11
12 --------------
13
14 The :mod:`concurrent.futures` module provides a high-level interface for
15 asynchronously executing callables.
16
17 The asynchronous execution can be performed with threads, using
18 :class:`ThreadPoolExecutor`, or separate processes, using
19 :class:`ProcessPoolExecutor`. Both implement the same interface, which is
20 defined by the abstract :class:`Executor` class.
21
22 :class:`concurrent.futures.Future` must not be confused with
23 :class:`asyncio.Future`, which is designed for use with :mod:`asyncio`
24 tasks and coroutines. See the :doc:`asyncio's Future <asyncio-future>`
25 documentation for a detailed comparison of the two.
26
27 .. include:: ../includes/wasm-notavail.rst
28
29 Executor Objects
30 ----------------
31
32 .. class:: Executor
33
34 An abstract class that provides methods to execute calls asynchronously. It
35 should not be used directly, but through its concrete subclasses.
36
37 .. method:: submit(fn, /, *args, **kwargs)
38
39 Schedules the callable, *fn*, to be executed as ``fn(*args, **kwargs)``
40 and returns a :class:`Future` object representing the execution of the
41 callable. ::
42
43 with ThreadPoolExecutor(max_workers=1) as executor:
44 future = executor.submit(pow, 323, 1235)
45 print(future.result())
46
47 .. method:: map(fn, *iterables, timeout=None, chunksize=1)
48
49 Similar to :func:`map(fn, *iterables) <map>` except:
50
51 * the *iterables* are collected immediately rather than lazily;
52
53 * *fn* is executed asynchronously and several calls to
54 *fn* may be made concurrently.
55
56 The returned iterator raises a :exc:`TimeoutError`
57 if :meth:`~iterator.__next__` is called and the result isn't available
58 after *timeout* seconds from the original call to :meth:`Executor.map`.
59 *timeout* can be an int or a float. If *timeout* is not specified or
60 ``None``, there is no limit to the wait time.
61
62 If a *fn* call raises an exception, then that exception will be
63 raised when its value is retrieved from the iterator.
64
65 When using :class:`ProcessPoolExecutor`, this method chops *iterables*
66 into a number of chunks which it submits to the pool as separate
67 tasks. The (approximate) size of these chunks can be specified by
68 setting *chunksize* to a positive integer. For very long iterables,
69 using a large value for *chunksize* can significantly improve
70 performance compared to the default size of 1. With
71 :class:`ThreadPoolExecutor`, *chunksize* has no effect.
72
73 .. versionchanged:: 3.5
74 Added the *chunksize* argument.
75
76 .. method:: shutdown(wait=True, *, cancel_futures=False)
77
78 Signal the executor that it should free any resources that it is using
79 when the currently pending futures are done executing. Calls to
80 :meth:`Executor.submit` and :meth:`Executor.map` made after shutdown will
81 raise :exc:`RuntimeError`.
82
83 If *wait* is ``True`` then this method will not return until all the
84 pending futures are done executing and the resources associated with the
85 executor have been freed. If *wait* is ``False`` then this method will
86 return immediately and the resources associated with the executor will be
87 freed when all pending futures are done executing. Regardless of the
88 value of *wait*, the entire Python program will not exit until all
89 pending futures are done executing.
90
91 If *cancel_futures* is ``True``, this method will cancel all pending
92 futures that the executor has not started running. Any futures that
93 are completed or running won't be cancelled, regardless of the value
94 of *cancel_futures*.
95
96 If both *cancel_futures* and *wait* are ``True``, all futures that the
97 executor has started running will be completed prior to this method
98 returning. The remaining futures are cancelled.
99
100 You can avoid having to call this method explicitly if you use the executor
101 as a :term:`context manager` via the :keyword:`with` statement, which
102 will shutdown the :class:`Executor` (waiting as if :meth:`Executor.shutdown`
103 were called with *wait* set to ``True``)::
104
105 import shutil
106 with ThreadPoolExecutor(max_workers=4) as e:
107 e.submit(shutil.copy, 'src1.txt', 'dest1.txt')
108 e.submit(shutil.copy, 'src2.txt', 'dest2.txt')
109 e.submit(shutil.copy, 'src3.txt', 'dest3.txt')
110 e.submit(shutil.copy, 'src4.txt', 'dest4.txt')
111
112 .. versionchanged:: 3.9
113 Added *cancel_futures*.
114
115
116 ThreadPoolExecutor
117 ------------------
118
119 :class:`ThreadPoolExecutor` is an :class:`Executor` subclass that uses a pool of
120 threads to execute calls asynchronously.
121
122 Deadlocks can occur when the callable associated with a :class:`Future` waits on
123 the results of another :class:`Future`. For example::
124
125 import time
126 def wait_on_b():
127 time.sleep(5)
128 print(b.result()) # b will never complete because it is waiting on a.
129 return 5
130
131 def wait_on_a():
132 time.sleep(5)
133 print(a.result()) # a will never complete because it is waiting on b.
134 return 6
135
136
137 executor = ThreadPoolExecutor(max_workers=2)
138 a = executor.submit(wait_on_b)
139 b = executor.submit(wait_on_a)
140
141 And::
142
143 def wait_on_future():
144 f = executor.submit(pow, 5, 2)
145 # This will never complete because there is only one worker thread and
146 # it is executing this function.
147 print(f.result())
148
149 executor = ThreadPoolExecutor(max_workers=1)
150 future = executor.submit(wait_on_future)
151 # Note: calling future.result() would also cause a deadlock because
152 # the single worker thread is already waiting for wait_on_future().
153
154
155 .. class:: ThreadPoolExecutor(max_workers=None, thread_name_prefix='', initializer=None, initargs=())
156
157 An :class:`Executor` subclass that uses a pool of at most *max_workers*
158 threads to execute calls asynchronously.
159
160 All threads enqueued to ``ThreadPoolExecutor`` will be joined before the
161 interpreter can exit. Note that the exit handler which does this is
162 executed *before* any exit handlers added using ``atexit``. This means
163 exceptions in the main thread must be caught and handled in order to
164 signal threads to exit gracefully. For this reason, it is recommended
165 that ``ThreadPoolExecutor`` not be used for long-running tasks.
166
167 *initializer* is an optional callable that is called at the start of
168 each worker thread; *initargs* is a tuple of arguments passed to the
169 initializer. Should *initializer* raise an exception, all currently
170 pending jobs will raise a :exc:`~concurrent.futures.thread.BrokenThreadPool`,
171 as well as any attempt to submit more jobs to the pool.
172
173 .. versionchanged:: 3.5
174 If *max_workers* is ``None`` or
175 not given, it will default to the number of processors on the machine,
176 multiplied by ``5``, assuming that :class:`ThreadPoolExecutor` is often
177 used to overlap I/O instead of CPU work and the number of workers
178 should be higher than the number of workers
179 for :class:`ProcessPoolExecutor`.
180
181 .. versionchanged:: 3.6
182 Added the *thread_name_prefix* parameter to allow users to
183 control the :class:`threading.Thread` names for worker threads created by
184 the pool for easier debugging.
185
186 .. versionchanged:: 3.7
187 Added the *initializer* and *initargs* arguments.
188
189 .. versionchanged:: 3.8
190 Default value of *max_workers* is changed to ``min(32, os.cpu_count() + 4)``.
191 This default value preserves at least 5 workers for I/O bound tasks.
192 It utilizes at most 32 CPU cores for CPU bound tasks which release the GIL.
193 And it avoids using very large resources implicitly on many-core machines.
194
195 ThreadPoolExecutor now reuses idle worker threads before starting
196 *max_workers* worker threads too.
197
198 .. versionchanged:: 3.13
199 Default value of *max_workers* is changed to
200 ``min(32, (os.process_cpu_count() or 1) + 4)``.
201
202
203 .. _threadpoolexecutor-example:
204
205 ThreadPoolExecutor Example
206 ~~~~~~~~~~~~~~~~~~~~~~~~~~
207 ::
208
209 import concurrent.futures
210 import urllib.request
211
212 URLS = ['http://www.foxnews.com/',
213 'http://www.cnn.com/',
214 'http://europe.wsj.com/',
215 'http://www.bbc.co.uk/',
216 'http://nonexistent-subdomain.python.org/']
217
218 # Retrieve a single page and report the URL and contents
219 def load_url(url, timeout):
220 with urllib.request.urlopen(url, timeout=timeout) as conn:
221 return conn.read()
222
223 # We can use a with statement to ensure threads are cleaned up promptly
224 with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
225 # Start the load operations and mark each future with its URL
226 future_to_url = {executor.submit(load_url, url, 60): url for url in URLS}
227 for future in concurrent.futures.as_completed(future_to_url):
228 url = future_to_url[future]
229 try:
230 data = future.result()
231 except Exception as exc:
232 print('%r generated an exception: %s' % (url, exc))
233 else:
234 print('%r page is %d bytes' % (url, len(data)))
235
236
237 ProcessPoolExecutor
238 -------------------
239
240 The :class:`ProcessPoolExecutor` class is an :class:`Executor` subclass that
241 uses a pool of processes to execute calls asynchronously.
242 :class:`ProcessPoolExecutor` uses the :mod:`multiprocessing` module, which
243 allows it to side-step the :term:`Global Interpreter Lock
244 <global interpreter lock>` but also means that
245 only picklable objects can be executed and returned.
246
247 The ``__main__`` module must be importable by worker subprocesses. This means
248 that :class:`ProcessPoolExecutor` will not work in the interactive interpreter.
249
250 Calling :class:`Executor` or :class:`Future` methods from a callable submitted
251 to a :class:`ProcessPoolExecutor` will result in deadlock.
252
253 Note that the restrictions on functions and arguments needing to picklable as
254 per :class:`multiprocessing.Process` apply when using :meth:`~Executor.submit`
255 and :meth:`~Executor.map` on a :class:`ProcessPoolExecutor`. A function defined
256 in a REPL or a lambda should not be expected to work.
257
258 .. class:: ProcessPoolExecutor(max_workers=None, mp_context=None, initializer=None, initargs=(), max_tasks_per_child=None)
259
260 An :class:`Executor` subclass that executes calls asynchronously using a pool
261 of at most *max_workers* processes. If *max_workers* is ``None`` or not
262 given, it will default to :func:`os.process_cpu_count`.
263 If *max_workers* is less than or equal to ``0``, then a :exc:`ValueError`
264 will be raised.
265 On Windows, *max_workers* must be less than or equal to ``61``. If it is not
266 then :exc:`ValueError` will be raised. If *max_workers* is ``None``, then
267 the default chosen will be at most ``61``, even if more processors are
268 available.
269 *mp_context* can be a :mod:`multiprocessing` context or ``None``. It will be
270 used to launch the workers. If *mp_context* is ``None`` or not given, the
271 default :mod:`multiprocessing` context is used.
272 See :ref:`multiprocessing-start-methods`.
273
274 *initializer* is an optional callable that is called at the start of
275 each worker process; *initargs* is a tuple of arguments passed to the
276 initializer. Should *initializer* raise an exception, all currently
277 pending jobs will raise a :exc:`~concurrent.futures.process.BrokenProcessPool`,
278 as well as any attempt to submit more jobs to the pool.
279
280 *max_tasks_per_child* is an optional argument that specifies the maximum
281 number of tasks a single process can execute before it will exit and be
282 replaced with a fresh worker process. By default *max_tasks_per_child* is
283 ``None`` which means worker processes will live as long as the pool. When
284 a max is specified, the "spawn" multiprocessing start method will be used by
285 default in absence of a *mp_context* parameter. This feature is incompatible
286 with the "fork" start method.
287
288 .. versionchanged:: 3.3
289 When one of the worker processes terminates abruptly, a
290 :exc:`~concurrent.futures.process.BrokenProcessPool` error is now raised.
291 Previously, behaviour
292 was undefined but operations on the executor or its futures would often
293 freeze or deadlock.
294
295 .. versionchanged:: 3.7
296 The *mp_context* argument was added to allow users to control the
297 start_method for worker processes created by the pool.
298
299 Added the *initializer* and *initargs* arguments.
300
301 .. note::
302 The default :mod:`multiprocessing` start method
303 (see :ref:`multiprocessing-start-methods`) will change away from
304 *fork* in Python 3.14. Code that requires *fork* be used for their
305 :class:`ProcessPoolExecutor` should explicitly specify that by
306 passing a ``mp_context=multiprocessing.get_context("fork")``
307 parameter.
308
309 .. versionchanged:: 3.11
310 The *max_tasks_per_child* argument was added to allow users to
311 control the lifetime of workers in the pool.
312
313 .. versionchanged:: 3.12
314 On POSIX systems, if your application has multiple threads and the
315 :mod:`multiprocessing` context uses the ``"fork"`` start method:
316 The :func:`os.fork` function called internally to spawn workers may raise a
317 :exc:`DeprecationWarning`. Pass a *mp_context* configured to use a
318 different start method. See the :func:`os.fork` documentation for
319 further explanation.
320
321 .. versionchanged:: 3.13
322 *max_workers* uses :func:`os.process_cpu_count` by default, instead of
323 :func:`os.cpu_count`.
324
325 .. versionchanged:: 3.13.15
326 Fixed a deadlock (:gh:`115634`) where the executor could hang after
327 a worker process exited upon reaching its *max_tasks_per_child*
328 limit while tasks remained queued.
329
330 .. _processpoolexecutor-example:
331
332 ProcessPoolExecutor Example
333 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
334 ::
335
336 import concurrent.futures
337 import math
338
339 PRIMES = [
340 112272535095293,
341 112582705942171,
342 112272535095293,
343 115280095190773,
344 115797848077099,
345 1099726899285419]
346
347 def is_prime(n):
348 if n < 2:
349 return False
350 if n == 2:
351 return True
352 if n % 2 == 0:
353 return False
354
355 sqrt_n = int(math.floor(math.sqrt(n)))
356 for i in range(3, sqrt_n + 1, 2):
357 if n % i == 0:
358 return False
359 return True
360
361 def main():
362 with concurrent.futures.ProcessPoolExecutor() as executor:
363 for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
364 print('%d is prime: %s' % (number, prime))
365
366 if __name__ == '__main__':
367 main()
368
369
370 Future Objects
371 --------------
372
373 The :class:`Future` class encapsulates the asynchronous execution of a callable.
374 :class:`Future` instances are created by :meth:`Executor.submit`.
375
376 .. class:: Future
377
378 Encapsulates the asynchronous execution of a callable. :class:`Future`
379 instances are created by :meth:`Executor.submit` and should not be created
380 directly except for testing.
381
382 .. method:: cancel()
383
384 Attempt to cancel the call. If the call is currently being executed or
385 finished running and cannot be cancelled then the method will return
386 ``False``, otherwise the call will be cancelled and the method will
387 return ``True``.
388
389 .. method:: cancelled()
390
391 Return ``True`` if the call was successfully cancelled.
392
393 .. method:: running()
394
395 Return ``True`` if the call is currently being executed and cannot be
396 cancelled.
397
398 .. method:: done()
399
400 Return ``True`` if the call was successfully cancelled or finished
401 running.
402
403 .. method:: result(timeout=None)
404
405 Return the value returned by the call. If the call hasn't yet completed
406 then this method will wait up to *timeout* seconds. If the call hasn't
407 completed in *timeout* seconds, then a
408 :exc:`TimeoutError` will be raised. *timeout* can be
409 an int or float. If *timeout* is not specified or ``None``, there is no
410 limit to the wait time.
411
412 If the future is cancelled before completing then :exc:`.CancelledError`
413 will be raised.
414
415 If the call raised an exception, this method will raise the same exception.
416
417 .. method:: exception(timeout=None)
418
419 Return the exception raised by the call. If the call hasn't yet
420 completed then this method will wait up to *timeout* seconds. If the
421 call hasn't completed in *timeout* seconds, then a
422 :exc:`TimeoutError` will be raised. *timeout* can be
423 an int or float. If *timeout* is not specified or ``None``, there is no
424 limit to the wait time.
425
426 If the future is cancelled before completing then :exc:`.CancelledError`
427 will be raised.
428
429 If the call completed without raising, ``None`` is returned.
430
431 .. method:: add_done_callback(fn)
432
433 Attaches the callable *fn* to the future. *fn* will be called, with the
434 future as its only argument, when the future is cancelled or finishes
435 running.
436
437 Added callables are called in the order that they were added and are
438 always called in a thread belonging to the process that added them. If
439 the callable raises an :exc:`Exception` subclass, it will be logged and
440 ignored. If the callable raises a :exc:`BaseException` subclass, the
441 behavior is undefined.
442
443 If the future has already completed or been cancelled, *fn* will be
444 called immediately.
445
446 The following :class:`Future` methods are meant for use in unit tests and
447 :class:`Executor` implementations.
448
449 .. method:: set_running_or_notify_cancel()
450
451 This method should only be called by :class:`Executor` implementations
452 before executing the work associated with the :class:`Future` and by unit
453 tests.
454
455 If the method returns ``False`` then the :class:`Future` was cancelled,
456 i.e. :meth:`Future.cancel` was called and returned ``True``. Any threads
457 waiting on the :class:`Future` completing (i.e. through
458 :func:`as_completed` or :func:`wait`) will be woken up.
459
460 If the method returns ``True`` then the :class:`Future` was not cancelled
461 and has been put in the running state, i.e. calls to
462 :meth:`Future.running` will return ``True``.
463
464 This method can only be called once and cannot be called after
465 :meth:`Future.set_result` or :meth:`Future.set_exception` have been
466 called.
467
468 .. method:: set_result(result)
469
470 Sets the result of the work associated with the :class:`Future` to
471 *result*.
472
473 This method should only be used by :class:`Executor` implementations and
474 unit tests.
475
476 .. versionchanged:: 3.8
477 This method raises
478 :exc:`concurrent.futures.InvalidStateError` if the :class:`Future` is
479 already done.
480
481 .. method:: set_exception(exception)
482
483 Sets the result of the work associated with the :class:`Future` to the
484 :class:`Exception` *exception*.
485
486 This method should only be used by :class:`Executor` implementations and
487 unit tests.
488
489 .. versionchanged:: 3.8
490 This method raises
491 :exc:`concurrent.futures.InvalidStateError` if the :class:`Future` is
492 already done.
493
494 Module Functions
495 ----------------
496
497 .. function:: wait(fs, timeout=None, return_when=ALL_COMPLETED)
498
499 Wait for the :class:`Future` instances (possibly created by different
500 :class:`Executor` instances) given by *fs* to complete. Duplicate futures
501 given to *fs* are removed and will be returned only once. Returns a named
502 2-tuple of sets. The first set, named ``done``, contains the futures that
503 completed (finished or cancelled futures) before the wait completed. The
504 second set, named ``not_done``, contains the futures that did not complete
505 (pending or running futures).
506
507 *timeout* can be used to control the maximum number of seconds to wait before
508 returning. *timeout* can be an int or float. If *timeout* is not specified
509 or ``None``, there is no limit to the wait time.
510
511 *return_when* indicates when this function should return. It must be one of
512 the following constants:
513
514 .. list-table::
515 :header-rows: 1
516
517 * - Constant
518 - Description
519
520 * - .. data:: FIRST_COMPLETED
521 - The function will return when any future finishes or is cancelled.
522
523 * - .. data:: FIRST_EXCEPTION
524 - The function will return when any future finishes by raising an
525 exception. If no future raises an exception
526 then it is equivalent to :const:`ALL_COMPLETED`.
527
528 * - .. data:: ALL_COMPLETED
529 - The function will return when all futures finish or are cancelled.
530
531 .. function:: as_completed(fs, timeout=None)
532
533 Returns an iterator over the :class:`Future` instances (possibly created by
534 different :class:`Executor` instances) given by *fs* that yields futures as
535 they complete (finished or cancelled futures). Any futures given by *fs* that
536 are duplicated will be returned once. Any futures that completed before
537 :func:`as_completed` is called will be yielded first. The returned iterator
538 raises a :exc:`TimeoutError` if :meth:`~iterator.__next__`
539 is called and the result isn't available after *timeout* seconds from the
540 original call to :func:`as_completed`. *timeout* can be an int or float. If
541 *timeout* is not specified or ``None``, there is no limit to the wait time.
542
543
544 .. seealso::
545
546 :pep:`3148` -- futures - execute computations asynchronously
547 The proposal which described this feature for inclusion in the Python
548 standard library.
549
550
551 Exception classes
552 -----------------
553
554 .. currentmodule:: concurrent.futures
555
556 .. exception:: CancelledError
557
558 Raised when a future is cancelled.
559
560 .. exception:: TimeoutError
561
562 A deprecated alias of :exc:`TimeoutError`,
563 raised when a future operation exceeds the given timeout.
564
565 .. versionchanged:: 3.11
566
567 This class was made an alias of :exc:`TimeoutError`.
568
569
570 .. exception:: BrokenExecutor
571
572 Derived from :exc:`RuntimeError`, this exception class is raised
573 when an executor is broken for some reason, and cannot be used
574 to submit or execute new tasks.
575
576 .. versionadded:: 3.7
577
578 .. exception:: InvalidStateError
579
580 Raised when an operation is performed on a future that is not allowed
581 in the current state.
582
583 .. versionadded:: 3.8
584
585 .. currentmodule:: concurrent.futures.thread
586
587 .. exception:: BrokenThreadPool
588
589 Derived from :exc:`~concurrent.futures.BrokenExecutor`, this exception
590 class is raised when one of the workers
591 of a :class:`~concurrent.futures.ThreadPoolExecutor`
592 has failed initializing.
593
594 .. versionadded:: 3.7
595
596 .. currentmodule:: concurrent.futures.process
597
598 .. exception:: BrokenProcessPool
599
600 Derived from :exc:`~concurrent.futures.BrokenExecutor` (formerly
601 :exc:`RuntimeError`), this exception class is raised when one of the
602 workers of a :class:`~concurrent.futures.ProcessPoolExecutor`
603 has terminated in a non-clean
604 fashion (for example, if it was killed from the outside).
605
606 .. versionadded:: 3.3