| 1 |
<?xml version="1.0" encoding="utf-8" ?> |
| 2 |
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
| 3 |
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> |
| 4 |
<head> |
| 5 |
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> |
| 6 |
<meta name="generator" content="Docutils 0.4: http://docutils.sourceforge.net/" /> |
| 7 |
<title>Introduction</title> |
| 8 |
<link rel="stylesheet" href="html4css1.css" type="text/css" /> |
| 9 |
</head> |
| 10 |
<body> |
| 11 |
<div class="header"> |
| 12 |
<a class="reference" href="index.html">Prev</a> <a class="reference" href="index.html">Up</a> <a class="reference" href="processing-ref.html">Next</a> |
| 13 |
<hr class="header"/> |
| 14 |
</div> |
| 15 |
<div class="document" id="introduction"> |
| 16 |
<h1 class="title">Introduction</h1> |
| 17 |
<div class="section"> |
| 18 |
<h1><a id="threads-processes-and-the-gil" name="threads-processes-and-the-gil">Threads, processes and the GIL</a></h1> |
| 19 |
<p>To run more than one piece of code at the same time on the same |
| 20 |
computer one has the choice of either using multiple processes or |
| 21 |
multiple threads.</p> |
| 22 |
<p>Although a program can be made up of multiple processes, these |
| 23 |
processes are in effect completely independent of one another: |
| 24 |
different processes are not able to cooperate with one another unless |
| 25 |
one sets up some means of communication between them (such as by using |
| 26 |
sockets). If a lot of data must be transferred between processes then |
| 27 |
this can be inefficient.</p> |
| 28 |
<p>On the other hand, multiple threads within a single process are |
| 29 |
intimately connected: they share their data but often can interfere |
| 30 |
badly with one another. It is often argued that the only way to make |
| 31 |
multithreaded programming "easy" is to avoid relying on any shared |
| 32 |
state and for the threads to only communicate by passing messages to |
| 33 |
each other.</p> |
| 34 |
<p>CPython has a <em>Global Interpreter Lock</em> (GIL) which in many ways makes |
| 35 |
threading easier than it is in most languages by making sure that only |
| 36 |
one thread can manipulate the interpreter's objects at a time. As a |
| 37 |
result, it is often safe to let multiple threads access data without |
| 38 |
using any additional locking as one would need to in a language such |
| 39 |
as C.</p> |
| 40 |
<p>One downside of the GIL is that on multi-processor (or multi-core) |
| 41 |
systems a multithreaded Python program can only make use of one |
| 42 |
processor at a time. This is a problem that can be overcome by using |
| 43 |
multiple processes instead.</p> |
| 44 |
<p>Python gives little direct support for writing programs using multiple |
| 45 |
process. This package allows one to write multi-process programs |
| 46 |
using much the same API that one uses for writing threaded programs.</p> |
| 47 |
</div> |
| 48 |
<div class="section"> |
| 49 |
<h1><a id="forking-and-spawning" name="forking-and-spawning">Forking and spawning</a></h1> |
| 50 |
<p>There are two ways of creating a new process in Python:</p> |
| 51 |
<ul> |
| 52 |
<li><p class="first">The current process can <em>fork</em> a new child process by using the |
| 53 |
<tt class="docutils literal"><span class="pre">os.fork()</span></tt> function. This effectively creates an identical copy |
| 54 |
of the current process which is now able to go off and perform some |
| 55 |
task set by the parent process. This means that the child process |
| 56 |
inherits <em>copies</em> of all variables that the parent process had.</p> |
| 57 |
<p>However, <tt class="docutils literal"><span class="pre">os.fork()</span></tt> is not available on every platform: in |
| 58 |
particular Windows does not support it.</p> |
| 59 |
</li> |
| 60 |
<li><p class="first">Alternatively, the current process can spawn a completely new Python |
| 61 |
interpreter by using the <tt class="docutils literal"><span class="pre">subprocess</span></tt> module or one of the |
| 62 |
<tt class="docutils literal"><span class="pre">os.spawn*()</span></tt> functions.</p> |
| 63 |
<p>Getting this new interpreter in to a fit state to perform the task |
| 64 |
set for it by its parent process is, however, a bit of a challenge.</p> |
| 65 |
</li> |
| 66 |
</ul> |
| 67 |
<p>The <tt class="docutils literal"><span class="pre">processing</span></tt> package uses <tt class="docutils literal"><span class="pre">os.fork()</span></tt> if it is available since |
| 68 |
it makes life a lot simpler. Forking the process is also more |
| 69 |
efficient in terms of memory usage and the time needed to create the |
| 70 |
new process.</p> |
| 71 |
</div> |
| 72 |
<div class="section"> |
| 73 |
<h1><a id="the-process-class" name="the-process-class">The Process class</a></h1> |
| 74 |
<p>In the <tt class="docutils literal"><span class="pre">processing</span></tt> package processes are spawned by creating a |
| 75 |
<tt class="docutils literal"><span class="pre">Process</span></tt> object and then calling its <tt class="docutils literal"><span class="pre">start()</span></tt> method. |
| 76 |
<tt class="docutils literal"><span class="pre">processing.Process</span></tt> follows the API of <tt class="docutils literal"><span class="pre">threading.Thread</span></tt>. A |
| 77 |
trivial example of a multiprocess program is</p> |
| 78 |
<pre class="literal-block"> |
| 79 |
from processing import Process |
| 80 |
|
| 81 |
def f(name): |
| 82 |
print 'hello', name |
| 83 |
|
| 84 |
if __name__ == '__main__': |
| 85 |
p = Process(target=f, args=('bob',)) |
| 86 |
p.start() |
| 87 |
p.join() |
| 88 |
</pre> |
| 89 |
<p>Here the function <tt class="docutils literal"><span class="pre">f</span></tt> is run in a child process.</p> |
| 90 |
<p>For an explanation of why (on Windows) the <tt class="docutils literal"><span class="pre">if</span> <span class="pre">__name__</span> <span class="pre">==</span> <span class="pre">'__main__'</span></tt> |
| 91 |
part is necessary see <a class="reference" href="programming-guidelines.html">Programming guidelines</a>.</p> |
| 92 |
</div> |
| 93 |
<div class="section"> |
| 94 |
<h1><a id="exchanging-objects-between-processes" name="exchanging-objects-between-processes">Exchanging objects between processes</a></h1> |
| 95 |
<p><tt class="docutils literal"><span class="pre">processing</span></tt> supports two types of communication channel between |
| 96 |
processes:</p> |
| 97 |
<dl class="docutils"> |
| 98 |
<dt><strong>Queues</strong>:</dt> |
| 99 |
<dd><p class="first">The function <tt class="docutils literal"><span class="pre">Queue()</span></tt> returns a near clone of <tt class="docutils literal"><span class="pre">Queue.Queue</span></tt> |
| 100 |
-- see the Python standard documentation. For example</p> |
| 101 |
<pre class="literal-block"> |
| 102 |
from processing import Process, Queue |
| 103 |
|
| 104 |
def f(q): |
| 105 |
q.put([42, None, 'hello']) |
| 106 |
|
| 107 |
if __name__ == '__main__': |
| 108 |
q = Queue() |
| 109 |
p = Process(target=f, args=(q,)) |
| 110 |
p.start() |
| 111 |
print q.get() # prints "[42, None, 'hello']" |
| 112 |
p.join() |
| 113 |
</pre> |
| 114 |
<p class="last">Queues are thread and process safe. See <a class="reference" href="processing-ref.html#pipes-and-queues">Queues</a>.</p> |
| 115 |
</dd> |
| 116 |
<dt><strong>Pipes</strong>:</dt> |
| 117 |
<dd><p class="first">The <tt class="docutils literal"><span class="pre">Pipe()</span></tt> function returns a pair of connection objects |
| 118 |
connected by a pipe which by default is duplex (two-way). For |
| 119 |
example</p> |
| 120 |
<pre class="literal-block"> |
| 121 |
from processing import Process, Pipe |
| 122 |
|
| 123 |
def f(conn): |
| 124 |
conn.send([42, None, 'hello']) |
| 125 |
conn.close() |
| 126 |
|
| 127 |
if __name__ == '__main__': |
| 128 |
parent_conn, child_conn = Pipe() |
| 129 |
p = Process(target=f, args=(child_conn,)) |
| 130 |
p.start() |
| 131 |
print parent_conn.recv() # prints "[42, None, 'hello']" |
| 132 |
p.join() |
| 133 |
</pre> |
| 134 |
<p class="last">The two connection objects returned by <tt class="docutils literal"><span class="pre">Pipe()</span></tt> represent the two |
| 135 |
ends of the pipe. Each connection object has <tt class="docutils literal"><span class="pre">send()</span></tt> and |
| 136 |
<tt class="docutils literal"><span class="pre">recv()</span></tt> methods (among others). Note that data in a pipe may |
| 137 |
become corrupted if two processes (or threads) try to read from or |
| 138 |
write to the <em>same</em> end of the pipe at the same time. Of course |
| 139 |
there is no risk of corruption from processes using different ends |
| 140 |
of the pipe at the same time. See <a class="reference" href="processing-ref.html#pipes-and-queues">Pipes</a>.</p> |
| 141 |
</dd> |
| 142 |
</dl> |
| 143 |
</div> |
| 144 |
<div class="section"> |
| 145 |
<h1><a id="synchronization-between-processes" name="synchronization-between-processes">Synchronization between processes</a></h1> |
| 146 |
<p><tt class="docutils literal"><span class="pre">processing</span></tt> contains equivalents of all the synchronization |
| 147 |
primitives from <tt class="docutils literal"><span class="pre">threading</span></tt>. For instance one can use a lock to |
| 148 |
ensure that only one process prints to standard output at a time:</p> |
| 149 |
<pre class="literal-block"> |
| 150 |
from processing import Process, Lock |
| 151 |
|
| 152 |
def f(l, i): |
| 153 |
l.acquire() |
| 154 |
print 'hello world', i |
| 155 |
l.release() |
| 156 |
|
| 157 |
if __name__ == '__main__': |
| 158 |
lock = Lock() |
| 159 |
|
| 160 |
for num in range(10): |
| 161 |
Process(target=f, args=(lock, num)).start() |
| 162 |
</pre> |
| 163 |
<p>Without using the lock output from the different processes is liable |
| 164 |
to get all mixed up.</p> |
| 165 |
</div> |
| 166 |
<div class="section"> |
| 167 |
<h1><a id="sharing-state-between-processes" name="sharing-state-between-processes">Sharing state between processes</a></h1> |
| 168 |
<p>As mentioned above, when doing concurrent programming it is usually |
| 169 |
best to avoid using shared state as far as possible. This is |
| 170 |
particularly true when using multiple processes.</p> |
| 171 |
<p>However, if you really do need to use some shared data then |
| 172 |
<tt class="docutils literal"><span class="pre">processing</span></tt> provides a couple of ways of doing so.</p> |
| 173 |
<dl class="docutils"> |
| 174 |
<dt><strong>Shared memory</strong>:</dt> |
| 175 |
<dd><p class="first">Data can be stored in a shared memory map using <tt class="docutils literal"><span class="pre">Value</span></tt> or <tt class="docutils literal"><span class="pre">Array</span></tt>. |
| 176 |
For example the following code</p> |
| 177 |
<pre class="literal-block"> |
| 178 |
from processing import Process, Value, Array |
| 179 |
|
| 180 |
def f(n, a): |
| 181 |
n.value = 3.1415927 |
| 182 |
for i in range(len(a)): |
| 183 |
a[i] = -a[i] |
| 184 |
|
| 185 |
if __name__ == '__main__': |
| 186 |
num = Value('d', 0.0) |
| 187 |
arr = Array('i', range(10)) |
| 188 |
|
| 189 |
p = Process(target=f, args=(num, arr)) |
| 190 |
p.start() |
| 191 |
p.join() |
| 192 |
|
| 193 |
print num.value |
| 194 |
print arr[:] |
| 195 |
</pre> |
| 196 |
<p>will print</p> |
| 197 |
<pre class="literal-block"> |
| 198 |
3.1415927 |
| 199 |
[0, -1, -2, -3, -4, -5, -6, -7, -8, -9] |
| 200 |
</pre> |
| 201 |
<p>The <tt class="docutils literal"><span class="pre">'d'</span></tt> and <tt class="docutils literal"><span class="pre">'i'</span></tt> arguments used when creating <tt class="docutils literal"><span class="pre">num</span></tt> and <tt class="docutils literal"><span class="pre">arr</span></tt> |
| 202 |
are typecodes of the kind used by the <tt class="docutils literal"><span class="pre">array</span></tt> module: <tt class="docutils literal"><span class="pre">'d'</span></tt> |
| 203 |
indicates a double precision float and <tt class="docutils literal"><span class="pre">'i'</span></tt> inidicates a signed |
| 204 |
integer. These shared objects will be process and thread safe.</p> |
| 205 |
<p class="last">For more flexibility in using shared memory one can use the |
| 206 |
<tt class="docutils literal"><span class="pre">processing.sharedctypes</span></tt> module which supports the creation of |
| 207 |
arbitrary <a class="reference" href="sharedctypes.html">ctypes objects allocated from shared memory</a>.</p> |
| 208 |
</dd> |
| 209 |
<dt><strong>Server process</strong>:</dt> |
| 210 |
<dd><p class="first">A manager object returned by <tt class="docutils literal"><span class="pre">Manager()</span></tt> controls a server process |
| 211 |
which holds python objects and allows other processes to manipulate |
| 212 |
them using proxies.</p> |
| 213 |
<p>A manager returned by <tt class="docutils literal"><span class="pre">Manager()</span></tt> will support types <tt class="docutils literal"><span class="pre">list</span></tt>, |
| 214 |
<tt class="docutils literal"><span class="pre">dict</span></tt>, <tt class="docutils literal"><span class="pre">Namespace</span></tt>, <tt class="docutils literal"><span class="pre">Lock</span></tt>, <tt class="docutils literal"><span class="pre">RLock</span></tt>, <tt class="docutils literal"><span class="pre">Semaphore</span></tt>, |
| 215 |
<tt class="docutils literal"><span class="pre">BoundedSemaphore</span></tt>, <tt class="docutils literal"><span class="pre">Condition</span></tt>, <tt class="docutils literal"><span class="pre">Event</span></tt>, <tt class="docutils literal"><span class="pre">Queue</span></tt>, <tt class="docutils literal"><span class="pre">Value</span></tt> |
| 216 |
and <tt class="docutils literal"><span class="pre">Array</span></tt>. For example:</p> |
| 217 |
<pre class="literal-block"> |
| 218 |
from processing import Process, Manager |
| 219 |
|
| 220 |
def f(d, l): |
| 221 |
d[1] = '1' |
| 222 |
d['2'] = 2 |
| 223 |
d[0.25] = None |
| 224 |
l.reverse() |
| 225 |
|
| 226 |
if __name__ == '__main__': |
| 227 |
manager = Manager() |
| 228 |
|
| 229 |
d = manager.dict() |
| 230 |
l = manager.list(range(10)) |
| 231 |
|
| 232 |
p = Process(target=f, args=(d, l)) |
| 233 |
p.start() |
| 234 |
p.join() |
| 235 |
|
| 236 |
print d |
| 237 |
print l |
| 238 |
</pre> |
| 239 |
<p>will print</p> |
| 240 |
<pre class="literal-block"> |
| 241 |
{0.25: None, 1: '1', '2': 2} |
| 242 |
[9, 8, 7, 6, 5, 4, 3, 2, 1, 0] |
| 243 |
</pre> |
| 244 |
<p>Creating managers which support other types is not hard --- see |
| 245 |
<a class="reference" href="manager-objects.html#customized-managers">Customized managers</a>.</p> |
| 246 |
<p class="last">Server process managers are more flexible than using shared memory |
| 247 |
objects because they can be made to support arbitrary object types. |
| 248 |
Also, a single manager can be shared by processes on different |
| 249 |
computers over a network. They are, however, slower than using |
| 250 |
shared memory. See <a class="reference" href="manager-objects.html#server-process-managers">Server process managers</a>.</p> |
| 251 |
</dd> |
| 252 |
</dl> |
| 253 |
</div> |
| 254 |
<div class="section"> |
| 255 |
<h1><a id="using-a-pool-of-workers" name="using-a-pool-of-workers">Using a pool of workers</a></h1> |
| 256 |
<p>The <tt class="docutils literal"><span class="pre">Pool()</span></tt> function returns an object representing a pool of worker |
| 257 |
processes. It has methods which allows tasks to be offloaded to the |
| 258 |
worker processes in a few different ways.</p> |
| 259 |
<p>For example:</p> |
| 260 |
<pre class="literal-block"> |
| 261 |
from processing import Pool |
| 262 |
|
| 263 |
def f(x): |
| 264 |
return x*x |
| 265 |
|
| 266 |
if __name__ == '__main__': |
| 267 |
pool = Pool(processes=4) # start 4 worker processes |
| 268 |
result = pool.applyAsync(f, [10]) # evaluate "f(10)" asynchronously |
| 269 |
print result.get(timeout=1) # prints "100" unless your computer is *very* slow |
| 270 |
print pool.map(f, range(10)) # prints "[0, 1, 4,..., 81]" |
| 271 |
</pre> |
| 272 |
<p>See <a class="reference" href="pool-objects.html">Process pools</a>.</p> |
| 273 |
</div> |
| 274 |
<div class="section"> |
| 275 |
<h1><a id="speed" name="speed">Speed</a></h1> |
| 276 |
<p>The following benchmarks were performed on a single core Pentium 4, |
| 277 |
2.5Ghz laptop running Windows XP and Ubuntu Linux 6.10 --- see |
| 278 |
<a class="reference" href="../examples/benchmarks.py">benchmarks.py</a>.</p> |
| 279 |
<p><em>Number of 256 byte string objects passed between processes/threads per sec</em>:</p> |
| 280 |
<table border="1" class="docutils"> |
| 281 |
<colgroup> |
| 282 |
<col width="55%" /> |
| 283 |
<col width="16%" /> |
| 284 |
<col width="29%" /> |
| 285 |
</colgroup> |
| 286 |
<thead valign="bottom"> |
| 287 |
<tr><th class="head">Connection type</th> |
| 288 |
<th class="head">Windows</th> |
| 289 |
<th class="head">Linux</th> |
| 290 |
</tr> |
| 291 |
</thead> |
| 292 |
<tbody valign="top"> |
| 293 |
<tr><td>Queue.Queue</td> |
| 294 |
<td>49,000</td> |
| 295 |
<td>17,000-50,000 <a class="footnote-reference" href="#id2" id="id1" name="id1">[1]</a></td> |
| 296 |
</tr> |
| 297 |
<tr><td>processing.Queue</td> |
| 298 |
<td>22,000</td> |
| 299 |
<td>21,000</td> |
| 300 |
</tr> |
| 301 |
<tr><td>Queue managed by server</td> |
| 302 |
<td>6,900</td> |
| 303 |
<td>6,500</td> |
| 304 |
</tr> |
| 305 |
<tr><td>processing.Pipe</td> |
| 306 |
<td>52,000</td> |
| 307 |
<td>57,000</td> |
| 308 |
</tr> |
| 309 |
</tbody> |
| 310 |
</table> |
| 311 |
<table class="docutils footnote" frame="void" id="id2" rules="none"> |
| 312 |
<colgroup><col class="label" /><col /></colgroup> |
| 313 |
<tbody valign="top"> |
| 314 |
<tr><td class="label"><a class="fn-backref" href="#id1" name="id2">[1]</a></td><td>For some reason the performance of <tt class="docutils literal"><span class="pre">Queue.Queue</span></tt> is very |
| 315 |
variable on Linux.</td></tr> |
| 316 |
</tbody> |
| 317 |
</table> |
| 318 |
<p><em>Number of acquires/releases of a lock per sec</em>:</p> |
| 319 |
<table border="1" class="docutils"> |
| 320 |
<colgroup> |
| 321 |
<col width="60%" /> |
| 322 |
<col width="20%" /> |
| 323 |
<col width="20%" /> |
| 324 |
</colgroup> |
| 325 |
<thead valign="bottom"> |
| 326 |
<tr><th class="head">Lock type</th> |
| 327 |
<th class="head">Windows</th> |
| 328 |
<th class="head">Linux</th> |
| 329 |
</tr> |
| 330 |
</thead> |
| 331 |
<tbody valign="top"> |
| 332 |
<tr><td>threading.Lock</td> |
| 333 |
<td>850,000</td> |
| 334 |
<td>560,000</td> |
| 335 |
</tr> |
| 336 |
<tr><td>processing.Lock</td> |
| 337 |
<td>420,000</td> |
| 338 |
<td>510,000</td> |
| 339 |
</tr> |
| 340 |
<tr><td>Lock managed by server</td> |
| 341 |
<td>10,000</td> |
| 342 |
<td>8,400</td> |
| 343 |
</tr> |
| 344 |
<tr><td>threading.RLock</td> |
| 345 |
<td>93,000</td> |
| 346 |
<td>76,000</td> |
| 347 |
</tr> |
| 348 |
<tr><td>processing.RLock</td> |
| 349 |
<td>420,000</td> |
| 350 |
<td>500,000</td> |
| 351 |
</tr> |
| 352 |
<tr><td>RLock managed by server</td> |
| 353 |
<td>8,800</td> |
| 354 |
<td>7,400</td> |
| 355 |
</tr> |
| 356 |
</tbody> |
| 357 |
</table> |
| 358 |
<p><em>Number of interleaved waits/notifies per sec on a |
| 359 |
condition variable by two processes</em>:</p> |
| 360 |
<table border="1" class="docutils"> |
| 361 |
<colgroup> |
| 362 |
<col width="60%" /> |
| 363 |
<col width="20%" /> |
| 364 |
<col width="20%" /> |
| 365 |
</colgroup> |
| 366 |
<thead valign="bottom"> |
| 367 |
<tr><th class="head">Condition type</th> |
| 368 |
<th class="head">Windows</th> |
| 369 |
<th class="head">Linux</th> |
| 370 |
</tr> |
| 371 |
</thead> |
| 372 |
<tbody valign="top"> |
| 373 |
<tr><td>threading.Condition</td> |
| 374 |
<td>27,000</td> |
| 375 |
<td>31,000</td> |
| 376 |
</tr> |
| 377 |
<tr><td>processing.Condition</td> |
| 378 |
<td>26,000</td> |
| 379 |
<td>25,000</td> |
| 380 |
</tr> |
| 381 |
<tr><td>Condition managed by server</td> |
| 382 |
<td>6,600</td> |
| 383 |
<td>6,000</td> |
| 384 |
</tr> |
| 385 |
</tbody> |
| 386 |
</table> |
| 387 |
<p><em>Number of integers retrieved from a sequence per sec</em>:</p> |
| 388 |
<table border="1" class="docutils"> |
| 389 |
<colgroup> |
| 390 |
<col width="60%" /> |
| 391 |
<col width="20%" /> |
| 392 |
<col width="20%" /> |
| 393 |
</colgroup> |
| 394 |
<thead valign="bottom"> |
| 395 |
<tr><th class="head">Sequence type</th> |
| 396 |
<th class="head">Windows</th> |
| 397 |
<th class="head">Linux</th> |
| 398 |
</tr> |
| 399 |
</thead> |
| 400 |
<tbody valign="top"> |
| 401 |
<tr><td>list</td> |
| 402 |
<td>6,400,000</td> |
| 403 |
<td>5,100,000</td> |
| 404 |
</tr> |
| 405 |
<tr><td>unsynchornized shared array</td> |
| 406 |
<td>3,900,000</td> |
| 407 |
<td>3,100,000</td> |
| 408 |
</tr> |
| 409 |
<tr><td>synchronized shared array</td> |
| 410 |
<td>200,000</td> |
| 411 |
<td>220,000</td> |
| 412 |
</tr> |
| 413 |
<tr><td>list managed by server</td> |
| 414 |
<td>20,000</td> |
| 415 |
<td>17,000</td> |
| 416 |
</tr> |
| 417 |
</tbody> |
| 418 |
</table> |
| 419 |
</div> |
| 420 |
</div> |
| 421 |
<div class="footer"> |
| 422 |
<hr class="footer" /> |
| 423 |
<a class="reference" href="index.html">Prev</a> <a class="reference" href="index.html">Up</a> <a class="reference" href="processing-ref.html">Next</a> |
| 424 |
</div> |
| 425 |
</body> |
| 426 |
</html> |