Debian Codegraph @

docs/getting_started.rst

raw
1
2 Getting Started
3 ===============
4
5 .. warning::
6
7 This section is incomplete.
8
9
10 Liability Waiver
11 ----------------
12
13 Before proceeding, it is critical you understand what you're involving yourself
14 and possibly your team and its successors with:
15
16 .. image:: images/pandora.svg
17 :class: mitogen-right-350
18
19 * Constructing the most fundamental class, :py:class:`Broker
20 <mitogen.master.Broker>`, causes a new thread to be spawned, exposing a huge
21 class of difficult to analyse behaviours that Python software generally does
22 not suffer from.
23
24 While every effort is made to hide this complexity, you should expect
25 threading-related encounters during development, and crucially, years after
26 your program reached production. See :ref:`troubleshooting` for more
27 information.
28
29 * While high-level abstractions are provided, they are only a convenience, you
30 must still understand :ref:`how Mitogen works <howitworks>` before depending
31 on it. Mitogen interacts with many aspects of the operating system,
32 threading, SSH, sudo, sockets, TTYs, shell, Python runtime, and timing and
33 ordering uncertainty introduced through interaction with the network, GIL and
34 OS scheduling.
35
36 Knowledge of this domain is typically attained through painful years of
37 failed attempts hacking system-level programs, and learning through continual
38 suffering how to debug the atrocities left behind. If you feel you lack
39 resources or willpower to diagnose problems independently, Mitogen is not
40 appropriate, prefer a higher level solution instead.
41
42
43 First Principles
44 ----------------
45
46 Before starting, take a moment to reflect on writing a program that will
47 operate across machines and privilege domains:
48
49 * As with multithreaded programming, writing a program that spans multiple
50 hosts is exposed to many asynchrony issues. Unlike multithreaded programming,
51 the margin for unexpected failures is much higher, even between only two
52 peers, as communication may be fail at any moment, since that communication
53 depends on reliability of an external network.
54
55 * Since a multi-host program always spans trust and privilege domains, trust
56 must be taken into consideration in your design from the outset. Mitogen
57 attempts to protect the consuming application by default where possible,
58 however it is paramount that trust considerations are always in mind when
59 exposing any privileged functionality to a potentially untrusted network of
60 peers.
61
62 A parent must always assume data received from a child is suspect, and must
63 not base privileged control decisions on that data. As a small example, a
64 parent should not form a command to execute in a subprocess using strings
65 received from a child.
66
67 * As the program spans multiple hosts, its design will benefit from a strict
68 separation of program and data. This entails avoiding some common Python
69 idioms that rely on its ability to manipulate functions and closures as if
70 they were data, such as passing a lambda closed over some program state as a
71 callback parameter.
72
73 In the general case this is both difficult and unsafe to support in a
74 distributed program, and so (for now at least) it should be assumed this
75 functionality is unlikely to appear in future.
76
77
78 Broker And Router
79 -----------------
80
81 .. image:: images/layout.svg
82 :class: mitogen-full-width
83
84 .. currentmodule:: mitogen.core
85
86 Execution starts when your program constructs a :py:class:`Broker` and
87 associated :py:class:`Router`. The broker is responsible for multiplexing IO to
88 children from a private thread, while in children, it is additionally
89 responsible for ensuring robust destruction if communication with the master
90 is lost.
91
92 :py:class:`Router` is responsible for receiving messages and dispatching them
93 to a callback from the broker thread (registered by :py:meth:`add_handler()
94 <mitogen.core.Router.add_handler>`), or forwarding them to a :py:class:`Stream
95 <mitogen.core.Stream>`. See :ref:`routing` for an in-depth description.
96 :py:class:`Router` also doubles as the entry point to Mitogen's public API::
97
98 >>> import mitogen.master
99
100 >>> broker = mitogen.master.Broker()
101 >>> router = mitogen.master.Router(broker)
102
103 >>> try:
104 ... # Your code here.
105 ... pass
106 ... finally:
107 ... broker.shutdown()
108
109 As Python will not stop if threads still exist after the main thread exits,
110 :py:meth:`Broker.shutdown` must be called reliably at exit. Helpers are
111 provided by :py:mod:`mitogen.utils` to ensure :py:class:`Broker` is reliably
112 destroyed::
113
114 def do_mitogen_stuff(router):
115 # Your code here.
116
117 mitogen.utils.run_with_router(do_mitogen_stuff)
118
119 If your program cannot live beneath :py:func:`mitogen.utils.run_with_router` on
120 the stack, you must arrange for :py:meth:`Broker.shutdown` to be called
121 anywhere the main thread may exit.
122
123
124 Enable Logging
125 --------------
126
127 Mitogen makes heavy use of the :py:mod:`logging` package, both for child
128 ``stdio`` redirection, and soft errors and warnings that may be generated.
129
130 You should always configure the :py:mod:`logging` package in any program that
131 integrates Mitogen. If your program does not otherwise use the
132 :py:mod:`logging` package, a basic configuration can be performed by calling
133 :py:func:`mitogen.utils.log_to_file`::
134
135 >>> import mitogen.utils
136
137 # Errors, warnings, and child stdio will be written to stderr.
138 >>> mitogen.utils.log_to_file()
139
140 Additionally, if your program has :py:const:`logging.DEBUG` as the default
141 logging level, you may wish to update its configuration to restrict the
142 ``mitogen`` logger to :py:const:`logging.INFO`, otherwise vast amounts of
143 output will be generated by default.
144
145
146 .. _logging-env-vars:
147
148 Logging Environment Variables
149 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
150
151 ``MITOGEN_LOG_LEVEL``
152 Overrides the :py:mod:`logging` package log level set by any call to
153 :py:func:`mitogen.utils.log_to_file`. Defaults to ``INFO``.
154
155 If set to ``IO``, equivalent to ``DEBUG`` but additionally enabled IO
156 logging for any call to :py:func:`mitogen.utils.log_to_file`. IO logging
157 produces verbose records of any IO interaction, which is useful for
158 debugging hangs and deadlocks.
159
160
161
162 Logging Records
163 ~~~~~~~~~~~~~~~
164
165 Messages received from a child context via :class:`mitogen.master.LogForwarder`
166 receive extra attributes:
167
168 * `mitogen_context`: :class:`mitogen.parent.Context` referring to the message
169 source.
170 * `mitogen_name`: original logger name in the source context.
171 * `mitogen_msg`: original message in the source context.
172
173
174 Creating A Context
175 ------------------
176
177 Contexts are simply external Python programs over which your program has
178 control, and can execute code within. They can be created as subprocesses on
179 the local machine, in another user account via `sudo`, on a remote machine via
180 `ssh`, or any recursive combination of the above.
181
182 Now a :py:class:`Router` exists, our first :py:class:`contexts <Context>` can
183 be created. To demonstrate basic functionality, we will start with some
184 :py:meth:`local() <Router.local>` contexts created as subprocesses::
185
186 >>> local = router.local()
187 >>> local_with_name = router.local(remote_name='i-have-a-name')
188
189 Examination of the system process list with the ``pstree`` utility reveals the
190 resulting process hierarchy::
191
192 | | \-+= 27660 dmw python
193 | | |--- 27661 dmw mitogen:[email protected]:27660
194 | | \--- 27663 dmw mitogen:i-have-a-name
195
196 Both contexts are visible as subprocesses of the interactive Python
197 interpreter, with their ``argv[0]`` including a description of their identity.
198 To aid systems administrators in identifying errant software running on their
199 machines, the default `remote_name` includes the location of the program that
200 started the context, however as shown, this can be overridden.
201
202 .. note::
203
204 Presently contexts are constructed in a blocking manner on the thread that
205 invoked the :ref:`context factory <context-factories>`. In a future
206 release, the factory will instead return immediately, and construction will
207 happen asynchronously on the broker thread.
208
209
210 Calling A Function
211 ------------------
212
213 .. currentmodule:: mitogen.parent
214
215 Now that some contexts exist, it is time to execute code in them. Any regular
216 function, static method, or class method reachable directly from module scope
217 may be used, including built-in functions such as :func:`time.time`.
218
219 The :py:meth:`Context.call` method is used to execute a function and block the
220 caller until the return value is available or an exception is raised::
221
222 >>> import time
223 >>> import os
224
225 >>> # Returns the current time.
226 >>> print('Time in remote context:', local.call(time.time))
227
228 >>> try:
229 ... # Raises OSError.
230 ... local.call(os.chdir, '/nonexistent')
231 ... except mitogen.core.CallError, e:
232 ... print('Call failed:', str(e))
233
234 It is a simple wrapper around the more flexible :meth:`Context.call_async`,
235 which immediately returns a :class:`Receiver <mitogen.core.Receiver>` wired up
236 to receive the return value instead. A receiver may simply be discarded, kept
237 around indefinitely without ever reading its result, or used to wait on the
238 results from several calls. Here :meth:`get() <mitogen.core.Receiver.get>`
239 is called to block the thread until the result arrives::
240
241 >>> call = local.call_async(time.time)
242 >>> msg = call.get()
243 >>> print(msg.unpickle())
244 1507292737.75547
245
246
247 Running User Functions
248 ----------------------
249
250 So far we have used the interactive interpreter to call some standard library
251 functions, but since the source code typed at the interpreter cannot be
252 recovered, Mitogen is unable to execute functions defined in this way.
253
254 We must therefore continue by writing our code as a script::
255
256 # first-script.py
257 import mitogen.utils
258
259 def my_first_function():
260 print('Hello from remote context!')
261 return 123
262
263 def main(router):
264 local = router.local()
265 print(local.call(my_first_function))
266
267 if __name__ == '__main__':
268 mitogen.utils.log_to_file("mitogen.log")
269 mitogen.utils.run_with_router(main)
270
271 Let's try running it:
272
273 .. code-block:: bash
274
275 $ python first-script.py
276 19:11:32 I mitogen.ctx.local.32466: stdout: Hello from remote context!
277 123
278
279
280 Waiting On Multiple Calls
281 -------------------------
282
283 Using :meth:`Context.call_async` it is possible to start multiple function
284 calls then sleep waiting for responses as they are available. This makes it
285 trivial to run tasks in parallel across processes (including remote processes)
286 without the need for writing asynchronous code::
287
288 hostnames = ['host1', 'host2', 'host3', 'host4']
289 contexts = [router.ssh(hostname=hn) for hn in hostnames]
290 calls = [context.call(my_func) for context in contexts]
291
292 for msg in mitogen.select.Select(calls):
293 print('Reply from %s: %s' % (recv.context, data))
294
295
296 Running Code That May Hang
297 --------------------------
298
299 When executing code that may hang due to, for example, talking to network peers
300 that may become unavailable, it is desirable to be able to recover control in
301 the case a remote call has hung.
302
303 By specifying the `timeout` parameter to :meth:`Receiver.get` on the receiver
304 returned by `Context.call_async`, it becomes possible to wait for a function to
305 complete, but time out if its result does not become available.
306
307 When a context has become hung like this, it is still possible to gracefully
308 terminate it using the :meth:`Context.shutdown` method. This method sends a
309 shutdown message to the target process, where its IO multiplexer thread can
310 still process it independently of the hung function running on on the target's
311 main thread.
312
313
314
315 Recovering Mitogen Object References In Children
316 ------------------------------------------------
317
318 ::
319
320 @mitogen.core.takes_econtext
321 def func1(a, b, econtext):
322 ...
323
324 @mitogen.core.takes_router
325 def func2(a, b, router):
326 ...
327
328
329 Recursion
330 ---------
331
332 Let's try something a little more complex:
333
334
335 .. _serialization-rules:
336
337 RPC Serialization Rules
338 -----------------------
339
340 The following built-in types may be used as parameters or return values in
341 remote procedure calls:
342
343 * :class:`bool`
344 * :class:`bytes` (:class:`str` on Python 2.x)
345 * :class:`dict`
346 * :class:`int`
347 * :class:`list`
348 * :class:`long`
349 * :class:`str` (:class:`unicode` on Python 2.x)
350 * :class:`tuple`
351
352 User-defined types may not be used, except for:
353
354 * :py:class:`mitogen.core.Blob`
355 * :py:class:`mitogen.core.Secret`
356 * :py:class:`mitogen.core.CallError`
357 * :py:class:`mitogen.core.Context`
358 * :py:class:`mitogen.core.Sender`
359
360 Subclasses of built-in types must be undecorated using
361 :py:func:`mitogen.utils.cast`.
362
363
364 Test Your Design
365 ----------------
366
367 ``tc qdisc add dev eth0 root netem delay 250ms``
368
369
370 .. _troubleshooting:
371
372 Troubleshooting
373 ---------------
374
375 .. warning::
376
377 This section is incomplete.
378
379 A typical example is a hang due to your application's main thread exitting
380 perhaps due to an unhandled exception, without first arranging for any
381 :py:class:`Broker <mitogen.master.Broker>` to be shut down gracefully.
382
383 Another example would be your main thread hanging indefinitely because a bug
384 in Mitogen fails to notice an event (such as RPC completion) your thread is
385 waiting for will never complete. Solving this kind of hang is a work in
386 progress.
387
388 router.enable_debug()