Debian Codegraph @

pypy3.11/doc/programming-guidelines.html

raw
1 <?xml version="1.0" encoding="utf-8" ?>
2 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
3 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
4 <head>
5 <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
6 <meta name="generator" content="Docutils 0.4: http://docutils.sourceforge.net/" />
7 <title>Programming guidelines</title>
8 <link rel="stylesheet" href="html4css1.css" type="text/css" />
9 </head>
10 <body>
11 <div class="header">
12 <a class="reference" href="connection-ref.html">Prev</a> &nbsp; &nbsp; &nbsp; &nbsp; <a class="reference" href="index.html">Up</a> &nbsp; &nbsp; &nbsp; &nbsp; <a class="reference" href="tests.html">Next</a>
13 <hr class="header"/>
14 </div>
15 <div class="document" id="programming-guidelines">
16 <h1 class="title">Programming guidelines</h1>
17 <p>There are certain guidelines and idioms which should be adhered to
18 when using the <tt class="docutils literal"><span class="pre">processing</span></tt> package.</p>
19 <div class="section">
20 <h1><a id="all-platforms" name="all-platforms">All platforms</a></h1>
21 <dl class="docutils">
22 <dt><em>Avoid shared state</em></dt>
23 <dd><p class="first">As far as possible one should try to avoid shifting large amounts
24 of data between processes.</p>
25 <p class="last">It is probably best to stick to using queues or pipes for
26 communication between processes rather than using the lower level
27 synchronization primitives from the <tt class="docutils literal"><span class="pre">threading</span></tt> module.</p>
28 </dd>
29 <dt><em>Picklability</em>:</dt>
30 <dd>Ensure that the arguments to the methods of proxies are
31 picklable.</dd>
32 <dt><em>Thread safety of proxies</em>:</dt>
33 <dd><p class="first">Do not use a proxy object from more than one thread unless you
34 protect it with a lock.</p>
35 <p class="last">(There is never a problem with different processes using the
36 'same' proxy.)</p>
37 </dd>
38 <dt><em>Joining zombie processes</em></dt>
39 <dd>On Unix when a process finishes but has not been joined it becomes
40 a zombie. There should never be very many because each time a new
41 process starts (or <tt class="docutils literal"><span class="pre">activeChildren()</span></tt> is called) all completed
42 processes which have not yet been joined will be joined. Also
43 calling a finished process's <tt class="docutils literal"><span class="pre">isAlive()</span></tt> will join the process.
44 Even so it is probably good practice to explicitly join all the
45 processes that you start.</dd>
46 <dt><em>Better to inherit than pickle/unpickle</em></dt>
47 <dd>On Windows many of types from the <tt class="docutils literal"><span class="pre">processing</span></tt> package need to be
48 picklable so that child processes can use them. However, one
49 should generally avoid sending shared objects to other processes
50 using pipes or queues. Instead you should arrange the program so
51 that a process which need access to a shared resource created
52 elsewhere can inherit it from an ancestor process.</dd>
53 <dt><em>Avoid terminating processes</em></dt>
54 <dd><p class="first">Using the <tt class="docutils literal"><span class="pre">terminate()</span></tt> method to stop a process is liable to
55 cause any shared resources (such as locks, semaphores, pipes and
56 queues) currently being used by the process to become broken or
57 unavailable to other processes.</p>
58 <p class="last">Therefore it is probably best to only consider using <tt class="docutils literal"><span class="pre">terminate()</span></tt>
59 on processes which never use any shared resources.</p>
60 </dd>
61 <dt><em>Joining processes that use queues</em></dt>
62 <dd><p class="first">Bear in mind that a process that has put items in a queue will
63 wait before terminating until all the buffered items are fed by
64 the &quot;feeder&quot; thread to the underlying pipe. (The child process
65 can call the <tt class="docutils literal"><span class="pre">cancelJoin()</span></tt> method of the queue to avoid this
66 behaviour.)</p>
67 <p>This means that whenever you use a queue you need to make sure
68 that all items which have been put on the queue will eventually be
69 removed before the process is joined. Otherwise you cannot be
70 sure that processes which have put items on the queue will
71 terminate. Remember also that non-daemonic processes will be
72 automatically be joined.</p>
73 <p>An example which will deadlock is the following:</p>
74 <pre class="literal-block">
75 from processing import Process, Queue
76
77 def f(q):
78 q.put('X' * 1000000)
79
80 if __name__ == '__main__':
81 queue = Queue()
82 p = Process(target=f, args=(queue,))
83 p.start()
84 p.join() # this deadlocks
85 obj = queue.get()
86 </pre>
87 <p class="last">A fix here would be to swap the last two lines round (or simply
88 remove the <tt class="docutils literal"><span class="pre">p.join()</span></tt> line).</p>
89 </dd>
90 <dt><em>Explicity pass resources to child processes</em></dt>
91 <dd><p class="first">On Unix a child process can make use of a shared resource created
92 in a parent process using a global resource. However, it is
93 better to pass the object as an argument to the constructor for
94 the child process.</p>
95 <p>Apart from making the code (potentially) compatible with Windows
96 this also ensures that as long as the child process is still alive
97 the object will not be garbage collected in the parent process.
98 This might be important if some resource is freed when the object
99 is garbage collected in the parent process.</p>
100 <p>So for instance</p>
101 <pre class="literal-block">
102 from processing import Process, Lock
103
104 def f():
105 ... do something using &quot;lock&quot; ...
106
107 if __name__ == '__main__':
108 lock = Lock()
109 for i in range(10):
110 Process(target=f).start()
111 </pre>
112 <p>should be rewritten as</p>
113 <pre class="last literal-block">
114 from processing import Process, Lock
115
116 def f(l):
117 ... do something using &quot;l&quot; ...
118
119 if __name__ == '__main__':
120 lock = Lock()
121 for i in range(10):
122 Process(target=f, args=(lock,)).start()
123 </pre>
124 </dd>
125 </dl>
126 </div>
127 <div class="section">
128 <h1><a id="windows" name="windows">Windows</a></h1>
129 <p>Since Windows lacks <tt class="docutils literal"><span class="pre">os.fork()</span></tt> it has a few extra restrictions:</p>
130 <p><em>More picklability</em>:</p>
131 <blockquote>
132 <p>Ensure that all arguments to <tt class="docutils literal"><span class="pre">Process.__init__()</span></tt> are picklable.
133 This means, in particular, that bound or unbound methods cannot be
134 used directly as the <tt class="docutils literal"><span class="pre">target</span></tt> argument on Windows --- just define
135 a function and use that instead.</p>
136 <p>Also, if you subclass <tt class="docutils literal"><span class="pre">Process</span></tt> then make sure that instances
137 will be picklable when the <tt class="docutils literal"><span class="pre">start()</span></tt> method is called.</p>
138 </blockquote>
139 <dl class="docutils">
140 <dt><em>Global variables</em>:</dt>
141 <dd><p class="first">Bear in mind that if code run in a child process tries to access a
142 global variable, then the value it sees (if any) may not be the
143 same as the value in the parent process at the time that
144 <tt class="docutils literal"><span class="pre">start()</span></tt> was called.</p>
145 <p class="last">However, global variables which are just module level constants
146 cause no problems.</p>
147 </dd>
148 <dt><em>Safe importing of main module</em>:</dt>
149 <dd><p class="first">Make sure that the main module can be safely imported by a new
150 Python interpreter without causing unintended side effects (such a
151 starting a new process).</p>
152 <p>For example, under Windows running the following module would
153 fail with a <tt class="docutils literal"><span class="pre">RuntimeError</span></tt>:</p>
154 <pre class="literal-block">
155 from processing import Process
156
157 def foo():
158 print 'hello'
159
160 p = Process(target=foo)
161 p.start()
162 </pre>
163 <p>Instead one should protect the &quot;entry point&quot; of the program by
164 using <tt class="docutils literal"><span class="pre">if</span> <span class="pre">__name__</span> <span class="pre">==</span> <span class="pre">'__main__':</span></tt> as follows:</p>
165 <pre class="literal-block">
166 from processing import Process
167
168 def foo():
169 print 'hello'
170
171 if __name__ == '__main__':
172 freezeSupport()
173 p = Process(target=foo)
174 p.start()
175 </pre>
176 <p>(The <tt class="docutils literal"><span class="pre">freezeSupport()</span></tt> line can be ommitted if the program will
177 be run normally instead of frozen.)</p>
178 <p>This allows the newly spawned Python interpreter to safely import
179 the module and then run the module's <tt class="docutils literal"><span class="pre">foo()</span></tt> function.</p>
180 <p class="last">Similar restrictions apply if a pool or manager is created in the
181 main module.</p>
182 </dd>
183 </dl>
184 </div>
185 </div>
186 <div class="footer">
187 <hr class="footer" />
188 <a class="reference" href="connection-ref.html">Prev</a> &nbsp; &nbsp; &nbsp; &nbsp; <a class="reference" href="index.html">Up</a> &nbsp; &nbsp; &nbsp; &nbsp; <a class="reference" href="tests.html">Next</a>
189 </div>
190 </body>
191 </html>