Debian Codegraph @

docs/index.rst

raw
1
2 Mitogen
3 =======
4
5 .. raw:: html
6
7 <style>
8 .warning code {
9 background-color: rgba(0, 0, 0, 0.1);
10 }
11 </style>
12
13 .. image:: images/mitogen.svg
14 :class: mitogen-right-180 mitogen-logo-wrap
15
16 Mitogen is a Python library for writing distributed self-replicating programs.
17
18 There is no requirement for installing packages, copying files around, writing
19 shell snippets, upfront configuration, or providing any secondary link to a
20 remote machine aside from an SSH connection. Due to its origins for use in
21 managing potentially damaged infrastructure, the **remote machine need not even
22 have free disk space or a writeable filesystem**.
23
24 It is not intended as a generic RPC framework; the goal is to provide a robust
25 and efficient low-level API on which tools like `Salt`_, `Ansible`_, or
26 `Fabric`_ can be built, and while the API is quite friendly and comparable to
27 `Fabric`_, ultimately it is not intended for direct use by consumer software.
28
29 .. _Salt: https://docs.saltproject.io/en/latest/
30 .. _Ansible: https://docs.ansible.com/
31 .. _Fabric: https://www.fabfile.org/
32
33 The focus is to centralize and perfect the intricate dance required to run
34 Python code safely and efficiently on a remote machine, while **avoiding
35 temporary files or large chunks of error-prone shell scripts**, and supporting
36 common privilege escalation techniques like `sudo`, potentially in combination
37 with exotic connection methods such as WMI, `telnet`, or console-over-IPMI.
38
39
40 Automatic Bootstrap
41 ###################
42
43 Mitogen's main feature is enabling your Python program to bootstrap and
44 communicate with new copies of itself under its control running on remote
45 machines, **using only an existing installed Python interpreter and SSH
46 client**, something that by default can be found on almost all contemporary
47 machines in the wild. To accomplish bootstrap, Mitogen uses a single 400 byte
48 SSH command line and 8KB of its own source code sent to stdin of the remote SSH
49 connection.
50
51 .. command-output:: python ../preamble_size.py
52
53 Once bootstrapped, the remote process is configured with a customizable
54 **argv[0]**, readily visible to system administrators of the remote machine
55 using the UNIX **ps** command:
56
57 .. code::
58
59 20051 ? Ss 0:00 \_ sshd: dmw [priv]
60 20053 ? S 0:00 | \_ sshd: dmw@notty
61 20054 ? Ssl 0:00 | \_ /usr/bin/python(mitogen:[email protected]:22476)
62 20103 ? S 0:00 | \_ tar zxvf myapp.tar.gz
63
64 The example context was started by UID ``dmw`` on host ``Eldil.home``, process
65 ID ``22476``.
66
67
68 IO Multiplexer
69 ##############
70
71 The bootstrap includes a compact IO multiplexer (like Twisted or asyncio) that
72 allows it to perform work in the background while executing your program's
73 code. For example, the remote context can be used to **connect to a new user on
74 the remote machine using sudo**, or as an intermediary for extending the
75 program's domain of control outward to other machines, enabling your program to
76 **manipulate machines behind a firewall**, or enable its **data plane to cohere
77 to your network topology**.
78
79 .. image:: images/billing.svg
80 :class: mitogen-right-150
81
82 .. code::
83
84 bastion_host = router.ssh(
85 hostname='jump-box.mycorp.com'
86 )
87
88 docker_host = router.ssh(
89 via=bastion_host,
90 hostname='docker-a.prod.mycorp.com'
91 )
92
93 sudo_account = router.sudo(
94 via=docker_host,
95 username='user_with_magic_ssh_key',
96 password='sudo password',
97 )
98
99 internal_box = router.docker(
100 via=sudo_account,
101 container='billing0',
102 )
103
104 internal_box.call(subprocess.check_call, ['./run-nightly-billing.py'])
105
106 The multiplexer also ensures the remote process is terminated if your Python
107 program crashes, communication is lost, or the application code running in the
108 context has hung.
109
110
111 Module Forwarder
112 ################
113
114 Slaves are configured with a custom `PEP-302 importer`_ that forwards requests
115 for unknown Python modules back to the host program. When your program asks a
116 context to execute code from an unknown module, all requisite modules are
117 transferred automatically and imported entirely in RAM without need for further
118 configuration.
119
120 .. _PEP-302 importer: https://www.python.org/dev/peps/pep-0302/
121
122 .. code-block:: python
123
124 import myapp.mypkg.mymodule
125
126 # myapp/__init__.py, myapp/mypkg/__init__.py, and myapp/mypkg/mymodule.py
127 # are transferred automatically.
128 print(context.call(myapp.mymodule.my_function))
129
130 As the forwarder reuses the import mechanism, it should integrate cleanly with
131 any tool such as `py2exe`_ that correctly implement the protocols in PEP-302,
132 allowing truly single file applications to run across multiple machines without
133 further effort.
134
135 .. _py2exe: https://www.py2exe.org/
136
137 Common sources of import latency and bandwidth consumption are mitigated:
138
139 * Modules need only be uploaded once per directly connected context. Subsequent
140 requests for modules from children of that context will be served by the
141 child itself.
142
143 * Imports by threads within a context triggering a load are deduplicated and
144 joined with any identical requests triggered by other threads in the same
145 context and children in the context's subtree.
146
147 * No roundtrip is required for negative responses due to Python 2's import
148 statement semantics: children have a list of submodules belonging to a
149 package, and ignore requests for submodules that did not exist on the master.
150
151 * Imports are extracted from each module, compared to those found in memory,
152 and recursively preloaded into children requesting that module, minimizing
153 round-trips to one per package nesting level. For example,
154 :py:mod:`django.db.models` only requires 3 round-trips to transfer 456KiB,
155 representing 1.7MiB of uncompressed source split across 148 modules.
156
157
158 Message Routing
159 ###############
160
161 .. image:: images/route.svg
162 :class: mitogen-full-width
163
164 Slaves may communicate autonomously without direct interaction with the master,
165 allowing a wide variety of complex data and control flows to be expressed using
166 the links between the processes.
167
168
169 Logging Forwarder
170 #################
171
172 The bootstrap configures the remote process's Python logging package to forward
173 all logs back to the local process, enabling management of program logs in one
174 location.
175
176 .. code::
177
178 18:15:29 D mitogen.ctx.k3: mitogen: Importer.find_module('mitogen.zlib')
179 18:15:29 D mitogen.ctx.k3: mitogen: _dispatch_calls((1002L, False, 'posix', None, 'system', ('ls -l /proc/self/fd',), {}))
180
181
182 Stdio Forwarder
183 ###############
184
185 To ease porting of crusty old infrastructure scripts to Python, the bootstrap
186 redirects stdio for itself and any child processes back into the logging
187 framework. This allows use of functions as basic as **os.system('hostname;
188 uptime')** without further need to capture or manage output.
189
190 .. code::
191
192 18:17:28 D mitogen.ctx.k3: mitogen: _dispatch_calls((1002L, False, 'posix', None, 'system', ('hostname; uptime',), {}))
193 18:17:56 I mitogen.ctx.k3: stdout: k3
194 18:17:56 I mitogen.ctx.k3: stdout: 17:37:10 up 562 days, 2:25, 5 users, load average: 1.24, 1.13, 1.14
195
196
197 Detached Subtrees
198 #################
199
200 .. image:: images/detached-subtree.svg
201 :class: mitogen-full-width
202
203 Contexts may detach from and outlive the running program, while maintaining
204 communication with descendents in their subtree. This enables persistent
205 background tasks that reuse Mitogen features.
206
207 .. code::
208
209 @mitogen.core.takes_econtext
210 def become_monitoring_master(children, econtext):
211 kill_old_process('/var/run/mydaemon.pid')
212 write_pid_file('/var/run/mydaemon.pid')
213 econtext.detach()
214
215 while True:
216 for child in children:
217 if child.call(get_cpu_load) > 0.9:
218 alert_operator('Child is too busy! ' + str(child))
219 time.sleep(1)
220
221 dc1.call_async(become_monitoring_master, children)
222
223
224 Blocking Code Friendly
225 ######################
226
227 Within each process, a private thread runs the I/O multiplexer, leaving the
228 main thread and any additional application threads free to perform useful work.
229
230 While Mitogen is internally asynchronous, it hides this asynchrony from
231 consumer code. This is since writing asynchronous code is mostly a foreign
232 concept to the target application of managing infrastructure. It should be
233 possible to rewrite a shell script in Python without significant restructuring,
234 or mind-bending feats of comprehension to understand control flow.
235
236 Before:
237
238 .. code-block:: sh
239
240 #!/bin/bash
241 # Install our application.
242
243 tar zxvf app.tar.gz
244
245 After:
246
247 .. code-block:: python
248
249 def install_app():
250 """
251 Install our application.
252 """
253 subprocess.check_call(['tar', 'zxvf', 'app.tar.gz'])
254
255 context.call(install_app)
256
257 Or even:
258
259 .. code-block:: python
260
261 context.call(subprocess.check_call, ['tar', 'zxvf', 'app.tar.gz'])
262
263 Exceptions raised by function calls are propagated back to the parent program,
264 and timeouts can be configured to ensure failed calls do not block progress of
265 the parent.
266
267
268 Scatter/Gather Calls
269 ####################
270
271 Functions may be invoked asynchronously, with results returned as they become
272 available.
273
274 .. code-block:: python
275
276 def usage(path):
277 return sum((os.path.getsize(os.path.join(dirpath, name))
278 for dirpath, dirnames, filenames in os.walk(path)
279 for name in dirnames + filenames), 0)
280
281 total = 0
282 for msg in Select(c.call_async(usage, '/tmp') for c in contexts):
283 usage = msg.unpickle()
284 print('Context %s /tmp usage: %d' % (recv.context, usage))
285 total += usage
286
287 print('Total /tmp usage across all contexts: %d' % (total,))
288
289
290 Single File Programs
291 ####################
292
293 Programs that are self-contained within a single Python script are supported.
294 External contexts are configured such that any attempt to execute a function
295 from the main Python script will correctly cause that script to be imported as
296 usual into the slave process.
297
298 .. literalinclude:: ../examples/install_app.py
299
300
301 Event-driven IO
302 ###############
303
304 Code running in a remote context can be connected to a *Channel*. Channels are
305 used to send data asynchronously back to the parent, without further need for
306 the parent to poll for changes. This is useful for monitoring systems managing
307 a large fleet of machines, or to alert the parent of unexpected state changes.
308
309 .. code-block:: python
310
311 def tail_log_file(channel, path='/var/log/messages'):
312 """
313 Forward new lines in a log file to the parent.
314 """
315 size = os.path.getsize(path)
316
317 while channel.open():
318 new_size = os.path.getsize(path)
319 if new_size == size:
320 time.sleep(1)
321 continue
322 elif new_size < size:
323 size = 0
324
325 fp = open(path, 'r')
326 fp.seek(size)
327 channel.send(fp.read(new_size - size))
328 fp.close()
329 size = new_size
330
331
332 Compatibility
333 #############
334
335 ``mitogen.*`` is compatible with Python 2.4 - 2.7 and 3.6 onward; making it
336 suitable for managing a fleet of potentially ancient corporate hardware, such
337 as Red Hat Enterprise Linux 5, released in 2007.
338
339 Every combination of Python 3.x/2.x parent and child should be possible.
340 Automated testing cannot cover every combination, automated testing tries to
341 cover the extemities (e.g. Python 3.14 parent -> Python 2.4 child).
342
343 ``ansible_mitogen.*`` is compatible with Python 2.7 and 3.6 onward; making it
344 suitable for Ansible 2.10 onward.
345
346
347 Zero Dependencies
348 #################
349
350 Mitogen is implemented entirely using the standard library functionality and
351 interfaces that were available in Python 2.4.