| 1 |
<?xml version="1.0" encoding="utf-8" ?> |
| 2 |
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
| 3 |
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> |
| 4 |
<head> |
| 5 |
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> |
| 6 |
<meta name="generator" content="Docutils 0.4: http://docutils.sourceforge.net/" /> |
| 7 |
<title>Process Pools</title> |
| 8 |
<link rel="stylesheet" href="html4css1.css" type="text/css" /> |
| 9 |
</head> |
| 10 |
<body> |
| 11 |
<div class="header"> |
| 12 |
<a class="reference" href="proxy-objects.html">Prev</a> <a class="reference" href="processing-ref.html">Up</a> <a class="reference" href="sharedctypes.html">Next</a> |
| 13 |
<hr class="header"/> |
| 14 |
</div> |
| 15 |
<div class="document" id="process-pools"> |
| 16 |
<h1 class="title">Process Pools</h1> |
| 17 |
<p>The <tt class="docutils literal"><span class="pre">processing.pool</span></tt> module has one public class:</p> |
| 18 |
<blockquote> |
| 19 |
<dl class="docutils"> |
| 20 |
<dt><strong>class</strong> <tt class="docutils literal"><span class="pre">Pool(processes=None,</span> <span class="pre">initializer=None,</span> <span class="pre">initargs=())</span></tt></dt> |
| 21 |
<dd><p class="first">A class representing a pool of worker processes.</p> |
| 22 |
<p>Tasks can be offloaded to the pool and the results dealt with |
| 23 |
when they become available.</p> |
| 24 |
<p>Note that tasks can only be submitted (or retrieved) by the |
| 25 |
process which created the pool object.</p> |
| 26 |
<p class="last"><tt class="docutils literal"><span class="pre">processes</span></tt> is the number of worker processes to use. If |
| 27 |
<tt class="docutils literal"><span class="pre">processes</span></tt> is <tt class="docutils literal"><span class="pre">None</span></tt> then the number returned by <tt class="docutils literal"><span class="pre">cpuCount()</span></tt> |
| 28 |
is used. If <tt class="docutils literal"><span class="pre">initializer</span></tt> is not <tt class="docutils literal"><span class="pre">None</span></tt> then each worker |
| 29 |
process will call <tt class="docutils literal"><span class="pre">initializer(*initargs)</span></tt> when it starts.</p> |
| 30 |
</dd> |
| 31 |
</dl> |
| 32 |
</blockquote> |
| 33 |
<div class="section"> |
| 34 |
<h1><a id="pool-objects" name="pool-objects">Pool objects</a></h1> |
| 35 |
<p><tt class="docutils literal"><span class="pre">Pool</span></tt> has the following public methods:</p> |
| 36 |
<blockquote> |
| 37 |
<dl class="docutils"> |
| 38 |
<dt><tt class="docutils literal"><span class="pre">__init__(processes=None)</span></tt></dt> |
| 39 |
<dd>The constructor creates and starts <tt class="docutils literal"><span class="pre">processes</span></tt> worker |
| 40 |
processes. If <tt class="docutils literal"><span class="pre">processes</span></tt> is <tt class="docutils literal"><span class="pre">None</span></tt> then <tt class="docutils literal"><span class="pre">cpuCount()</span></tt> is used |
| 41 |
to find a default or 1 if <tt class="docutils literal"><span class="pre">cpuCount()</span></tt> raises <tt class="docutils literal"><span class="pre">NotImplemented</span></tt>.</dd> |
| 42 |
<dt><tt class="docutils literal"><span class="pre">apply(func,</span> <span class="pre">args=(),</span> <span class="pre">kwds={})</span></tt></dt> |
| 43 |
<dd>Equivalent of the <tt class="docutils literal"><span class="pre">apply()</span></tt> builtin function. It blocks till |
| 44 |
the result is ready.</dd> |
| 45 |
<dt><tt class="docutils literal"><span class="pre">applyAsync(func,</span> <span class="pre">args=(),</span> <span class="pre">kwds={},</span> <span class="pre">callback=None)</span></tt></dt> |
| 46 |
<dd><p class="first">A variant of the <tt class="docutils literal"><span class="pre">apply()</span></tt> method which returns a |
| 47 |
result object --- see <a class="reference" href="#asynchronous-result-objects">Asynchronous result objects</a>.</p> |
| 48 |
<p class="last">If <tt class="docutils literal"><span class="pre">callback</span></tt> is specified then it should be a callable which |
| 49 |
accepts a single argument. When the result becomes ready |
| 50 |
<tt class="docutils literal"><span class="pre">callback</span></tt> is applied to it (unless the call failed). |
| 51 |
<tt class="docutils literal"><span class="pre">callback</span></tt> should complete immediately since otherwise the |
| 52 |
thread which handles the results will get blocked.</p> |
| 53 |
</dd> |
| 54 |
<dt><tt class="docutils literal"><span class="pre">map(func,</span> <span class="pre">iterable,</span> <span class="pre">chunksize=None)</span></tt></dt> |
| 55 |
<dd><p class="first">A parallel equivalent of the <tt class="docutils literal"><span class="pre">map()</span></tt> builtin function. It |
| 56 |
blocks till the result is ready.</p> |
| 57 |
<p class="last">This method chops the iterable into a number of chunks which |
| 58 |
it submits to the process pool as separate tasks. The |
| 59 |
(approximate) size of these chunks can be specified by setting |
| 60 |
<tt class="docutils literal"><span class="pre">chunksize</span></tt> to a positive integer.</p> |
| 61 |
</dd> |
| 62 |
<dt><tt class="docutils literal"><span class="pre">mapAsync(func,</span> <span class="pre">iterable,</span> <span class="pre">chunksize=None,</span> <span class="pre">callback=None)</span></tt></dt> |
| 63 |
<dd><p class="first">A variant of the <tt class="docutils literal"><span class="pre">map()</span></tt> method which returns a result object |
| 64 |
--- see <a class="reference" href="#asynchronous-result-objects">Asynchronous result objects</a>.</p> |
| 65 |
<p class="last">If <tt class="docutils literal"><span class="pre">callback</span></tt> is specified then it should be a callable which |
| 66 |
accepts a single argument. When the result becomes ready |
| 67 |
<tt class="docutils literal"><span class="pre">callback</span></tt> is applied to it (unless the call failed). |
| 68 |
<tt class="docutils literal"><span class="pre">callback</span></tt> should complete immediately since otherwise the |
| 69 |
thread which handles the results will get blocked.</p> |
| 70 |
</dd> |
| 71 |
<dt><tt class="docutils literal"><span class="pre">imap(func,</span> <span class="pre">iterable,</span> <span class="pre">chunksize=1)</span></tt></dt> |
| 72 |
<dd><p class="first">An equivalent of <tt class="docutils literal"><span class="pre">itertools.imap()</span></tt>.</p> |
| 73 |
<p>The <tt class="docutils literal"><span class="pre">chunksize</span></tt> argument is the same as the one used by the |
| 74 |
<tt class="docutils literal"><span class="pre">map()</span></tt> method. For very long iterables using a large value |
| 75 |
for <tt class="docutils literal"><span class="pre">chunksize</span></tt> can make make the job complete <strong>much</strong> faster |
| 76 |
than using the default value of <tt class="docutils literal"><span class="pre">1</span></tt>.</p> |
| 77 |
<p class="last">Also if <tt class="docutils literal"><span class="pre">chunksize</span></tt> is <tt class="docutils literal"><span class="pre">1</span></tt> then the <tt class="docutils literal"><span class="pre">next()</span></tt> method of the |
| 78 |
iterator returned by the <tt class="docutils literal"><span class="pre">imap()</span></tt> method has an optional |
| 79 |
<tt class="docutils literal"><span class="pre">timeout</span></tt> parameter: <tt class="docutils literal"><span class="pre">next(timeout)</span></tt> will raise |
| 80 |
<tt class="docutils literal"><span class="pre">processing.TimeoutError</span></tt> if the result cannot be returned |
| 81 |
within <tt class="docutils literal"><span class="pre">timeout</span></tt> seconds.</p> |
| 82 |
</dd> |
| 83 |
<dt><tt class="docutils literal"><span class="pre">imapUnordered(func,</span> <span class="pre">iterable,</span> <span class="pre">chunksize=1)</span></tt></dt> |
| 84 |
<dd>The same as <tt class="docutils literal"><span class="pre">imap()</span></tt> except that the ordering of the results |
| 85 |
from the returned iterator should be considered arbitrary. |
| 86 |
(Only when there is only one worker process is the order |
| 87 |
guaranteed to be "correct".)</dd> |
| 88 |
<dt><tt class="docutils literal"><span class="pre">close()</span></tt></dt> |
| 89 |
<dd>Prevents any more tasks from being submitted to the pool. |
| 90 |
Once all the tasks have been completed the worker processes |
| 91 |
will exit.</dd> |
| 92 |
<dt><tt class="docutils literal"><span class="pre">terminate()</span></tt></dt> |
| 93 |
<dd>Stops the worker processes immediately without completing |
| 94 |
outstanding work. When the pool object is garbage collected |
| 95 |
<tt class="docutils literal"><span class="pre">terminate()</span></tt> will be called immediately.</dd> |
| 96 |
<dt><tt class="docutils literal"><span class="pre">join()</span></tt></dt> |
| 97 |
<dd>Wait for the worker processes to exit. One must call |
| 98 |
<tt class="docutils literal"><span class="pre">close()</span></tt> or <tt class="docutils literal"><span class="pre">terminate()</span></tt> before using <tt class="docutils literal"><span class="pre">join()</span></tt>.</dd> |
| 99 |
</dl> |
| 100 |
</blockquote> |
| 101 |
</div> |
| 102 |
<div class="section"> |
| 103 |
<h1><a id="asynchronous-result-objects" name="asynchronous-result-objects">Asynchronous result objects</a></h1> |
| 104 |
<p>The result objects returns by <tt class="docutils literal"><span class="pre">applyAsync()</span></tt> and <tt class="docutils literal"><span class="pre">mapAsync()</span></tt> have |
| 105 |
the following public methods:</p> |
| 106 |
<blockquote> |
| 107 |
<dl class="docutils"> |
| 108 |
<dt><tt class="docutils literal"><span class="pre">get(timeout=None)</span></tt></dt> |
| 109 |
<dd>Returns the result when it arrives. If <tt class="docutils literal"><span class="pre">timeout</span></tt> is not |
| 110 |
<tt class="docutils literal"><span class="pre">None</span></tt> and the result does not arrive within <tt class="docutils literal"><span class="pre">timeout</span></tt> seconds |
| 111 |
then <tt class="docutils literal"><span class="pre">processing.TimeoutError</span></tt> is raised. If the remote call |
| 112 |
raised an exception then that exception will be reraised by <tt class="docutils literal"><span class="pre">get()</span></tt>.</dd> |
| 113 |
<dt><tt class="docutils literal"><span class="pre">wait(timeout=None)</span></tt></dt> |
| 114 |
<dd>Waits until the result is available or until <tt class="docutils literal"><span class="pre">timeout</span></tt> seconds |
| 115 |
pass.</dd> |
| 116 |
<dt><tt class="docutils literal"><span class="pre">ready()</span></tt></dt> |
| 117 |
<dd>Returns whether the call has completed.</dd> |
| 118 |
<dt><tt class="docutils literal"><span class="pre">successful()</span></tt></dt> |
| 119 |
<dd>Returns whether the call completed without raising an |
| 120 |
exception. Will raise <tt class="docutils literal"><span class="pre">AssertionError</span></tt> if the result is not |
| 121 |
ready.</dd> |
| 122 |
</dl> |
| 123 |
</blockquote> |
| 124 |
</div> |
| 125 |
<div class="section"> |
| 126 |
<h1><a id="examples" name="examples">Examples</a></h1> |
| 127 |
<p>The following example demonstrates the use of a pool:</p> |
| 128 |
<pre class="literal-block"> |
| 129 |
from processing import Pool |
| 130 |
|
| 131 |
def f(x): |
| 132 |
return x*x |
| 133 |
|
| 134 |
if __name__ == '__main__': |
| 135 |
pool = Pool(processes=4) # start 4 worker processes |
| 136 |
|
| 137 |
result = pool.applyAsync(f, (10,)) # evaluate "f(10)" asynchronously |
| 138 |
print result.get(timeout=1) # prints "100" unless your computer is *very* slow |
| 139 |
|
| 140 |
print pool.map(f, range(10)) # prints "[0, 1, 4,..., 81]" |
| 141 |
|
| 142 |
it = pool.imap(f, range(10)) |
| 143 |
print it.next() # prints "0" |
| 144 |
print it.next() # prints "1" |
| 145 |
print it.next(timeout=1) # prints "4" unless your computer is *very* slow |
| 146 |
|
| 147 |
import time |
| 148 |
result = pool.applyAsync(time.sleep, (10,)) |
| 149 |
print result.get(timeout=1) # raises `TimeoutError` |
| 150 |
</pre> |
| 151 |
<p>See also <a class="reference" href="../examples/ex_pool.py">ex_pool.py</a>.</p> |
| 152 |
</div> |
| 153 |
</div> |
| 154 |
<div class="footer"> |
| 155 |
<hr class="footer" /> |
| 156 |
<a class="reference" href="proxy-objects.html">Prev</a> <a class="reference" href="processing-ref.html">Up</a> <a class="reference" href="sharedctypes.html">Next</a> |
| 157 |
</div> |
| 158 |
</body> |
| 159 |
</html> |