| 1 |
<?xml version="1.0" encoding="utf-8" ?> |
| 2 |
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
| 3 |
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> |
| 4 |
<head> |
| 5 |
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> |
| 6 |
<meta name="generator" content="Docutils 0.4: http://docutils.sourceforge.net/" /> |
| 7 |
<title>Programming guidelines</title> |
| 8 |
<link rel="stylesheet" href="html4css1.css" type="text/css" /> |
| 9 |
</head> |
| 10 |
<body> |
| 11 |
<div class="header"> |
| 12 |
<a class="reference" href="connection-ref.html">Prev</a> <a class="reference" href="index.html">Up</a> <a class="reference" href="tests.html">Next</a> |
| 13 |
<hr class="header"/> |
| 14 |
</div> |
| 15 |
<div class="document" id="programming-guidelines"> |
| 16 |
<h1 class="title">Programming guidelines</h1> |
| 17 |
<p>There are certain guidelines and idioms which should be adhered to |
| 18 |
when using the <tt class="docutils literal"><span class="pre">processing</span></tt> package.</p> |
| 19 |
<div class="section"> |
| 20 |
<h1><a id="all-platforms" name="all-platforms">All platforms</a></h1> |
| 21 |
<dl class="docutils"> |
| 22 |
<dt><em>Avoid shared state</em></dt> |
| 23 |
<dd><p class="first">As far as possible one should try to avoid shifting large amounts |
| 24 |
of data between processes.</p> |
| 25 |
<p class="last">It is probably best to stick to using queues or pipes for |
| 26 |
communication between processes rather than using the lower level |
| 27 |
synchronization primitives from the <tt class="docutils literal"><span class="pre">threading</span></tt> module.</p> |
| 28 |
</dd> |
| 29 |
<dt><em>Picklability</em>:</dt> |
| 30 |
<dd>Ensure that the arguments to the methods of proxies are |
| 31 |
picklable.</dd> |
| 32 |
<dt><em>Thread safety of proxies</em>:</dt> |
| 33 |
<dd><p class="first">Do not use a proxy object from more than one thread unless you |
| 34 |
protect it with a lock.</p> |
| 35 |
<p class="last">(There is never a problem with different processes using the |
| 36 |
'same' proxy.)</p> |
| 37 |
</dd> |
| 38 |
<dt><em>Joining zombie processes</em></dt> |
| 39 |
<dd>On Unix when a process finishes but has not been joined it becomes |
| 40 |
a zombie. There should never be very many because each time a new |
| 41 |
process starts (or <tt class="docutils literal"><span class="pre">activeChildren()</span></tt> is called) all completed |
| 42 |
processes which have not yet been joined will be joined. Also |
| 43 |
calling a finished process's <tt class="docutils literal"><span class="pre">isAlive()</span></tt> will join the process. |
| 44 |
Even so it is probably good practice to explicitly join all the |
| 45 |
processes that you start.</dd> |
| 46 |
<dt><em>Better to inherit than pickle/unpickle</em></dt> |
| 47 |
<dd>On Windows many of types from the <tt class="docutils literal"><span class="pre">processing</span></tt> package need to be |
| 48 |
picklable so that child processes can use them. However, one |
| 49 |
should generally avoid sending shared objects to other processes |
| 50 |
using pipes or queues. Instead you should arrange the program so |
| 51 |
that a process which need access to a shared resource created |
| 52 |
elsewhere can inherit it from an ancestor process.</dd> |
| 53 |
<dt><em>Avoid terminating processes</em></dt> |
| 54 |
<dd><p class="first">Using the <tt class="docutils literal"><span class="pre">terminate()</span></tt> method to stop a process is liable to |
| 55 |
cause any shared resources (such as locks, semaphores, pipes and |
| 56 |
queues) currently being used by the process to become broken or |
| 57 |
unavailable to other processes.</p> |
| 58 |
<p class="last">Therefore it is probably best to only consider using <tt class="docutils literal"><span class="pre">terminate()</span></tt> |
| 59 |
on processes which never use any shared resources.</p> |
| 60 |
</dd> |
| 61 |
<dt><em>Joining processes that use queues</em></dt> |
| 62 |
<dd><p class="first">Bear in mind that a process that has put items in a queue will |
| 63 |
wait before terminating until all the buffered items are fed by |
| 64 |
the "feeder" thread to the underlying pipe. (The child process |
| 65 |
can call the <tt class="docutils literal"><span class="pre">cancelJoin()</span></tt> method of the queue to avoid this |
| 66 |
behaviour.)</p> |
| 67 |
<p>This means that whenever you use a queue you need to make sure |
| 68 |
that all items which have been put on the queue will eventually be |
| 69 |
removed before the process is joined. Otherwise you cannot be |
| 70 |
sure that processes which have put items on the queue will |
| 71 |
terminate. Remember also that non-daemonic processes will be |
| 72 |
automatically be joined.</p> |
| 73 |
<p>An example which will deadlock is the following:</p> |
| 74 |
<pre class="literal-block"> |
| 75 |
from processing import Process, Queue |
| 76 |
|
| 77 |
def f(q): |
| 78 |
q.put('X' * 1000000) |
| 79 |
|
| 80 |
if __name__ == '__main__': |
| 81 |
queue = Queue() |
| 82 |
p = Process(target=f, args=(queue,)) |
| 83 |
p.start() |
| 84 |
p.join() # this deadlocks |
| 85 |
obj = queue.get() |
| 86 |
</pre> |
| 87 |
<p class="last">A fix here would be to swap the last two lines round (or simply |
| 88 |
remove the <tt class="docutils literal"><span class="pre">p.join()</span></tt> line).</p> |
| 89 |
</dd> |
| 90 |
<dt><em>Explicity pass resources to child processes</em></dt> |
| 91 |
<dd><p class="first">On Unix a child process can make use of a shared resource created |
| 92 |
in a parent process using a global resource. However, it is |
| 93 |
better to pass the object as an argument to the constructor for |
| 94 |
the child process.</p> |
| 95 |
<p>Apart from making the code (potentially) compatible with Windows |
| 96 |
this also ensures that as long as the child process is still alive |
| 97 |
the object will not be garbage collected in the parent process. |
| 98 |
This might be important if some resource is freed when the object |
| 99 |
is garbage collected in the parent process.</p> |
| 100 |
<p>So for instance</p> |
| 101 |
<pre class="literal-block"> |
| 102 |
from processing import Process, Lock |
| 103 |
|
| 104 |
def f(): |
| 105 |
... do something using "lock" ... |
| 106 |
|
| 107 |
if __name__ == '__main__': |
| 108 |
lock = Lock() |
| 109 |
for i in range(10): |
| 110 |
Process(target=f).start() |
| 111 |
</pre> |
| 112 |
<p>should be rewritten as</p> |
| 113 |
<pre class="last literal-block"> |
| 114 |
from processing import Process, Lock |
| 115 |
|
| 116 |
def f(l): |
| 117 |
... do something using "l" ... |
| 118 |
|
| 119 |
if __name__ == '__main__': |
| 120 |
lock = Lock() |
| 121 |
for i in range(10): |
| 122 |
Process(target=f, args=(lock,)).start() |
| 123 |
</pre> |
| 124 |
</dd> |
| 125 |
</dl> |
| 126 |
</div> |
| 127 |
<div class="section"> |
| 128 |
<h1><a id="windows" name="windows">Windows</a></h1> |
| 129 |
<p>Since Windows lacks <tt class="docutils literal"><span class="pre">os.fork()</span></tt> it has a few extra restrictions:</p> |
| 130 |
<p><em>More picklability</em>:</p> |
| 131 |
<blockquote> |
| 132 |
<p>Ensure that all arguments to <tt class="docutils literal"><span class="pre">Process.__init__()</span></tt> are picklable. |
| 133 |
This means, in particular, that bound or unbound methods cannot be |
| 134 |
used directly as the <tt class="docutils literal"><span class="pre">target</span></tt> argument on Windows --- just define |
| 135 |
a function and use that instead.</p> |
| 136 |
<p>Also, if you subclass <tt class="docutils literal"><span class="pre">Process</span></tt> then make sure that instances |
| 137 |
will be picklable when the <tt class="docutils literal"><span class="pre">start()</span></tt> method is called.</p> |
| 138 |
</blockquote> |
| 139 |
<dl class="docutils"> |
| 140 |
<dt><em>Global variables</em>:</dt> |
| 141 |
<dd><p class="first">Bear in mind that if code run in a child process tries to access a |
| 142 |
global variable, then the value it sees (if any) may not be the |
| 143 |
same as the value in the parent process at the time that |
| 144 |
<tt class="docutils literal"><span class="pre">start()</span></tt> was called.</p> |
| 145 |
<p class="last">However, global variables which are just module level constants |
| 146 |
cause no problems.</p> |
| 147 |
</dd> |
| 148 |
<dt><em>Safe importing of main module</em>:</dt> |
| 149 |
<dd><p class="first">Make sure that the main module can be safely imported by a new |
| 150 |
Python interpreter without causing unintended side effects (such a |
| 151 |
starting a new process).</p> |
| 152 |
<p>For example, under Windows running the following module would |
| 153 |
fail with a <tt class="docutils literal"><span class="pre">RuntimeError</span></tt>:</p> |
| 154 |
<pre class="literal-block"> |
| 155 |
from processing import Process |
| 156 |
|
| 157 |
def foo(): |
| 158 |
print 'hello' |
| 159 |
|
| 160 |
p = Process(target=foo) |
| 161 |
p.start() |
| 162 |
</pre> |
| 163 |
<p>Instead one should protect the "entry point" of the program by |
| 164 |
using <tt class="docutils literal"><span class="pre">if</span> <span class="pre">__name__</span> <span class="pre">==</span> <span class="pre">'__main__':</span></tt> as follows:</p> |
| 165 |
<pre class="literal-block"> |
| 166 |
from processing import Process |
| 167 |
|
| 168 |
def foo(): |
| 169 |
print 'hello' |
| 170 |
|
| 171 |
if __name__ == '__main__': |
| 172 |
freezeSupport() |
| 173 |
p = Process(target=foo) |
| 174 |
p.start() |
| 175 |
</pre> |
| 176 |
<p>(The <tt class="docutils literal"><span class="pre">freezeSupport()</span></tt> line can be ommitted if the program will |
| 177 |
be run normally instead of frozen.)</p> |
| 178 |
<p>This allows the newly spawned Python interpreter to safely import |
| 179 |
the module and then run the module's <tt class="docutils literal"><span class="pre">foo()</span></tt> function.</p> |
| 180 |
<p class="last">Similar restrictions apply if a pool or manager is created in the |
| 181 |
main module.</p> |
| 182 |
</dd> |
| 183 |
</dl> |
| 184 |
</div> |
| 185 |
</div> |
| 186 |
<div class="footer"> |
| 187 |
<hr class="footer" /> |
| 188 |
<a class="reference" href="connection-ref.html">Prev</a> <a class="reference" href="index.html">Up</a> <a class="reference" href="tests.html">Next</a> |
| 189 |
</div> |
| 190 |
</body> |
| 191 |
</html> |