| 1 |
|
| 2 |
Mitogen |
| 3 |
======= |
| 4 |
|
| 5 |
.. raw:: html |
| 6 |
|
| 7 |
<style> |
| 8 |
.warning code { |
| 9 |
background-color: rgba(0, 0, 0, 0.1); |
| 10 |
} |
| 11 |
</style> |
| 12 |
|
| 13 |
.. image:: images/mitogen.svg |
| 14 |
:class: mitogen-right-180 mitogen-logo-wrap |
| 15 |
|
| 16 |
Mitogen is a Python library for writing distributed self-replicating programs. |
| 17 |
|
| 18 |
There is no requirement for installing packages, copying files around, writing |
| 19 |
shell snippets, upfront configuration, or providing any secondary link to a |
| 20 |
remote machine aside from an SSH connection. Due to its origins for use in |
| 21 |
managing potentially damaged infrastructure, the **remote machine need not even |
| 22 |
have free disk space or a writeable filesystem**. |
| 23 |
|
| 24 |
It is not intended as a generic RPC framework; the goal is to provide a robust |
| 25 |
and efficient low-level API on which tools like `Salt`_, `Ansible`_, or |
| 26 |
`Fabric`_ can be built, and while the API is quite friendly and comparable to |
| 27 |
`Fabric`_, ultimately it is not intended for direct use by consumer software. |
| 28 |
|
| 29 |
.. _Salt: https://docs.saltproject.io/en/latest/ |
| 30 |
.. _Ansible: https://docs.ansible.com/ |
| 31 |
.. _Fabric: https://www.fabfile.org/ |
| 32 |
|
| 33 |
The focus is to centralize and perfect the intricate dance required to run |
| 34 |
Python code safely and efficiently on a remote machine, while **avoiding |
| 35 |
temporary files or large chunks of error-prone shell scripts**, and supporting |
| 36 |
common privilege escalation techniques like `sudo`, potentially in combination |
| 37 |
with exotic connection methods such as WMI, `telnet`, or console-over-IPMI. |
| 38 |
|
| 39 |
|
| 40 |
Automatic Bootstrap |
| 41 |
################### |
| 42 |
|
| 43 |
Mitogen's main feature is enabling your Python program to bootstrap and |
| 44 |
communicate with new copies of itself under its control running on remote |
| 45 |
machines, **using only an existing installed Python interpreter and SSH |
| 46 |
client**, something that by default can be found on almost all contemporary |
| 47 |
machines in the wild. To accomplish bootstrap, Mitogen uses a single 400 byte |
| 48 |
SSH command line and 8KB of its own source code sent to stdin of the remote SSH |
| 49 |
connection. |
| 50 |
|
| 51 |
.. command-output:: python ../preamble_size.py |
| 52 |
|
| 53 |
Once bootstrapped, the remote process is configured with a customizable |
| 54 |
**argv[0]**, readily visible to system administrators of the remote machine |
| 55 |
using the UNIX **ps** command: |
| 56 |
|
| 57 |
.. code:: |
| 58 |
|
| 59 |
20051 ? Ss 0:00 \_ sshd: dmw [priv] |
| 60 |
20053 ? S 0:00 | \_ sshd: dmw@notty |
| 61 |
20054 ? Ssl 0:00 | \_ /usr/bin/python(mitogen:[email protected]:22476) |
| 62 |
20103 ? S 0:00 | \_ tar zxvf myapp.tar.gz |
| 63 |
|
| 64 |
The example context was started by UID ``dmw`` on host ``Eldil.home``, process |
| 65 |
ID ``22476``. |
| 66 |
|
| 67 |
|
| 68 |
IO Multiplexer |
| 69 |
############## |
| 70 |
|
| 71 |
The bootstrap includes a compact IO multiplexer (like Twisted or asyncio) that |
| 72 |
allows it to perform work in the background while executing your program's |
| 73 |
code. For example, the remote context can be used to **connect to a new user on |
| 74 |
the remote machine using sudo**, or as an intermediary for extending the |
| 75 |
program's domain of control outward to other machines, enabling your program to |
| 76 |
**manipulate machines behind a firewall**, or enable its **data plane to cohere |
| 77 |
to your network topology**. |
| 78 |
|
| 79 |
.. image:: images/billing.svg |
| 80 |
:class: mitogen-right-150 |
| 81 |
|
| 82 |
.. code:: |
| 83 |
|
| 84 |
bastion_host = router.ssh( |
| 85 |
hostname='jump-box.mycorp.com' |
| 86 |
) |
| 87 |
|
| 88 |
docker_host = router.ssh( |
| 89 |
via=bastion_host, |
| 90 |
hostname='docker-a.prod.mycorp.com' |
| 91 |
) |
| 92 |
|
| 93 |
sudo_account = router.sudo( |
| 94 |
via=docker_host, |
| 95 |
username='user_with_magic_ssh_key', |
| 96 |
password='sudo password', |
| 97 |
) |
| 98 |
|
| 99 |
internal_box = router.docker( |
| 100 |
via=sudo_account, |
| 101 |
container='billing0', |
| 102 |
) |
| 103 |
|
| 104 |
internal_box.call(subprocess.check_call, ['./run-nightly-billing.py']) |
| 105 |
|
| 106 |
The multiplexer also ensures the remote process is terminated if your Python |
| 107 |
program crashes, communication is lost, or the application code running in the |
| 108 |
context has hung. |
| 109 |
|
| 110 |
|
| 111 |
Module Forwarder |
| 112 |
################ |
| 113 |
|
| 114 |
Slaves are configured with a custom `PEP-302 importer`_ that forwards requests |
| 115 |
for unknown Python modules back to the host program. When your program asks a |
| 116 |
context to execute code from an unknown module, all requisite modules are |
| 117 |
transferred automatically and imported entirely in RAM without need for further |
| 118 |
configuration. |
| 119 |
|
| 120 |
.. _PEP-302 importer: https://www.python.org/dev/peps/pep-0302/ |
| 121 |
|
| 122 |
.. code-block:: python |
| 123 |
|
| 124 |
import myapp.mypkg.mymodule |
| 125 |
|
| 126 |
# myapp/__init__.py, myapp/mypkg/__init__.py, and myapp/mypkg/mymodule.py |
| 127 |
# are transferred automatically. |
| 128 |
print(context.call(myapp.mymodule.my_function)) |
| 129 |
|
| 130 |
As the forwarder reuses the import mechanism, it should integrate cleanly with |
| 131 |
any tool such as `py2exe`_ that correctly implement the protocols in PEP-302, |
| 132 |
allowing truly single file applications to run across multiple machines without |
| 133 |
further effort. |
| 134 |
|
| 135 |
.. _py2exe: https://www.py2exe.org/ |
| 136 |
|
| 137 |
Common sources of import latency and bandwidth consumption are mitigated: |
| 138 |
|
| 139 |
* Modules need only be uploaded once per directly connected context. Subsequent |
| 140 |
requests for modules from children of that context will be served by the |
| 141 |
child itself. |
| 142 |
|
| 143 |
* Imports by threads within a context triggering a load are deduplicated and |
| 144 |
joined with any identical requests triggered by other threads in the same |
| 145 |
context and children in the context's subtree. |
| 146 |
|
| 147 |
* No roundtrip is required for negative responses due to Python 2's import |
| 148 |
statement semantics: children have a list of submodules belonging to a |
| 149 |
package, and ignore requests for submodules that did not exist on the master. |
| 150 |
|
| 151 |
* Imports are extracted from each module, compared to those found in memory, |
| 152 |
and recursively preloaded into children requesting that module, minimizing |
| 153 |
round-trips to one per package nesting level. For example, |
| 154 |
:py:mod:`django.db.models` only requires 3 round-trips to transfer 456KiB, |
| 155 |
representing 1.7MiB of uncompressed source split across 148 modules. |
| 156 |
|
| 157 |
|
| 158 |
Message Routing |
| 159 |
############### |
| 160 |
|
| 161 |
.. image:: images/route.svg |
| 162 |
:class: mitogen-full-width |
| 163 |
|
| 164 |
Slaves may communicate autonomously without direct interaction with the master, |
| 165 |
allowing a wide variety of complex data and control flows to be expressed using |
| 166 |
the links between the processes. |
| 167 |
|
| 168 |
|
| 169 |
Logging Forwarder |
| 170 |
################# |
| 171 |
|
| 172 |
The bootstrap configures the remote process's Python logging package to forward |
| 173 |
all logs back to the local process, enabling management of program logs in one |
| 174 |
location. |
| 175 |
|
| 176 |
.. code:: |
| 177 |
|
| 178 |
18:15:29 D mitogen.ctx.k3: mitogen: Importer.find_module('mitogen.zlib') |
| 179 |
18:15:29 D mitogen.ctx.k3: mitogen: _dispatch_calls((1002L, False, 'posix', None, 'system', ('ls -l /proc/self/fd',), {})) |
| 180 |
|
| 181 |
|
| 182 |
Stdio Forwarder |
| 183 |
############### |
| 184 |
|
| 185 |
To ease porting of crusty old infrastructure scripts to Python, the bootstrap |
| 186 |
redirects stdio for itself and any child processes back into the logging |
| 187 |
framework. This allows use of functions as basic as **os.system('hostname; |
| 188 |
uptime')** without further need to capture or manage output. |
| 189 |
|
| 190 |
.. code:: |
| 191 |
|
| 192 |
18:17:28 D mitogen.ctx.k3: mitogen: _dispatch_calls((1002L, False, 'posix', None, 'system', ('hostname; uptime',), {})) |
| 193 |
18:17:56 I mitogen.ctx.k3: stdout: k3 |
| 194 |
18:17:56 I mitogen.ctx.k3: stdout: 17:37:10 up 562 days, 2:25, 5 users, load average: 1.24, 1.13, 1.14 |
| 195 |
|
| 196 |
|
| 197 |
Detached Subtrees |
| 198 |
################# |
| 199 |
|
| 200 |
.. image:: images/detached-subtree.svg |
| 201 |
:class: mitogen-full-width |
| 202 |
|
| 203 |
Contexts may detach from and outlive the running program, while maintaining |
| 204 |
communication with descendents in their subtree. This enables persistent |
| 205 |
background tasks that reuse Mitogen features. |
| 206 |
|
| 207 |
.. code:: |
| 208 |
|
| 209 |
@mitogen.core.takes_econtext |
| 210 |
def become_monitoring_master(children, econtext): |
| 211 |
kill_old_process('/var/run/mydaemon.pid') |
| 212 |
write_pid_file('/var/run/mydaemon.pid') |
| 213 |
econtext.detach() |
| 214 |
|
| 215 |
while True: |
| 216 |
for child in children: |
| 217 |
if child.call(get_cpu_load) > 0.9: |
| 218 |
alert_operator('Child is too busy! ' + str(child)) |
| 219 |
time.sleep(1) |
| 220 |
|
| 221 |
dc1.call_async(become_monitoring_master, children) |
| 222 |
|
| 223 |
|
| 224 |
Blocking Code Friendly |
| 225 |
###################### |
| 226 |
|
| 227 |
Within each process, a private thread runs the I/O multiplexer, leaving the |
| 228 |
main thread and any additional application threads free to perform useful work. |
| 229 |
|
| 230 |
While Mitogen is internally asynchronous, it hides this asynchrony from |
| 231 |
consumer code. This is since writing asynchronous code is mostly a foreign |
| 232 |
concept to the target application of managing infrastructure. It should be |
| 233 |
possible to rewrite a shell script in Python without significant restructuring, |
| 234 |
or mind-bending feats of comprehension to understand control flow. |
| 235 |
|
| 236 |
Before: |
| 237 |
|
| 238 |
.. code-block:: sh |
| 239 |
|
| 240 |
#!/bin/bash |
| 241 |
# Install our application. |
| 242 |
|
| 243 |
tar zxvf app.tar.gz |
| 244 |
|
| 245 |
After: |
| 246 |
|
| 247 |
.. code-block:: python |
| 248 |
|
| 249 |
def install_app(): |
| 250 |
""" |
| 251 |
Install our application. |
| 252 |
""" |
| 253 |
subprocess.check_call(['tar', 'zxvf', 'app.tar.gz']) |
| 254 |
|
| 255 |
context.call(install_app) |
| 256 |
|
| 257 |
Or even: |
| 258 |
|
| 259 |
.. code-block:: python |
| 260 |
|
| 261 |
context.call(subprocess.check_call, ['tar', 'zxvf', 'app.tar.gz']) |
| 262 |
|
| 263 |
Exceptions raised by function calls are propagated back to the parent program, |
| 264 |
and timeouts can be configured to ensure failed calls do not block progress of |
| 265 |
the parent. |
| 266 |
|
| 267 |
|
| 268 |
Scatter/Gather Calls |
| 269 |
#################### |
| 270 |
|
| 271 |
Functions may be invoked asynchronously, with results returned as they become |
| 272 |
available. |
| 273 |
|
| 274 |
.. code-block:: python |
| 275 |
|
| 276 |
def usage(path): |
| 277 |
return sum((os.path.getsize(os.path.join(dirpath, name)) |
| 278 |
for dirpath, dirnames, filenames in os.walk(path) |
| 279 |
for name in dirnames + filenames), 0) |
| 280 |
|
| 281 |
total = 0 |
| 282 |
for msg in Select(c.call_async(usage, '/tmp') for c in contexts): |
| 283 |
usage = msg.unpickle() |
| 284 |
print('Context %s /tmp usage: %d' % (recv.context, usage)) |
| 285 |
total += usage |
| 286 |
|
| 287 |
print('Total /tmp usage across all contexts: %d' % (total,)) |
| 288 |
|
| 289 |
|
| 290 |
Single File Programs |
| 291 |
#################### |
| 292 |
|
| 293 |
Programs that are self-contained within a single Python script are supported. |
| 294 |
External contexts are configured such that any attempt to execute a function |
| 295 |
from the main Python script will correctly cause that script to be imported as |
| 296 |
usual into the slave process. |
| 297 |
|
| 298 |
.. literalinclude:: ../examples/install_app.py |
| 299 |
|
| 300 |
|
| 301 |
Event-driven IO |
| 302 |
############### |
| 303 |
|
| 304 |
Code running in a remote context can be connected to a *Channel*. Channels are |
| 305 |
used to send data asynchronously back to the parent, without further need for |
| 306 |
the parent to poll for changes. This is useful for monitoring systems managing |
| 307 |
a large fleet of machines, or to alert the parent of unexpected state changes. |
| 308 |
|
| 309 |
.. code-block:: python |
| 310 |
|
| 311 |
def tail_log_file(channel, path='/var/log/messages'): |
| 312 |
""" |
| 313 |
Forward new lines in a log file to the parent. |
| 314 |
""" |
| 315 |
size = os.path.getsize(path) |
| 316 |
|
| 317 |
while channel.open(): |
| 318 |
new_size = os.path.getsize(path) |
| 319 |
if new_size == size: |
| 320 |
time.sleep(1) |
| 321 |
continue |
| 322 |
elif new_size < size: |
| 323 |
size = 0 |
| 324 |
|
| 325 |
fp = open(path, 'r') |
| 326 |
fp.seek(size) |
| 327 |
channel.send(fp.read(new_size - size)) |
| 328 |
fp.close() |
| 329 |
size = new_size |
| 330 |
|
| 331 |
|
| 332 |
Compatibility |
| 333 |
############# |
| 334 |
|
| 335 |
``mitogen.*`` is compatible with Python 2.4 - 2.7 and 3.6 onward; making it |
| 336 |
suitable for managing a fleet of potentially ancient corporate hardware, such |
| 337 |
as Red Hat Enterprise Linux 5, released in 2007. |
| 338 |
|
| 339 |
Every combination of Python 3.x/2.x parent and child should be possible. |
| 340 |
Automated testing cannot cover every combination, automated testing tries to |
| 341 |
cover the extemities (e.g. Python 3.14 parent -> Python 2.4 child). |
| 342 |
|
| 343 |
``ansible_mitogen.*`` is compatible with Python 2.7 and 3.6 onward; making it |
| 344 |
suitable for Ansible 2.10 onward. |
| 345 |
|
| 346 |
|
| 347 |
Zero Dependencies |
| 348 |
################# |
| 349 |
|
| 350 |
Mitogen is implemented entirely using the standard library functionality and |
| 351 |
interfaces that were available in Python 2.4. |