Debian Codegraph @

docs/api.rst

raw
1
2 API Reference
3 *************
4
5
6 Package Layout
7 ==============
8
9
10 mitogen Package
11 ---------------
12
13 .. automodule:: mitogen
14
15 .. autodata:: mitogen.__version__
16 .. autodata:: mitogen.is_master
17 .. autodata:: mitogen.context_id
18 .. autodata:: mitogen.parent_id
19 .. autodata:: mitogen.parent_ids
20 .. autofunction:: mitogen.main
21
22
23 mitogen.core
24 ------------
25
26 .. automodule:: mitogen.core
27
28 .. currentmodule:: mitogen.core
29 .. autodecorator:: takes_econtext
30
31 .. currentmodule:: mitogen.core
32 .. autodecorator:: takes_router
33
34
35 mitogen.master
36 --------------
37
38 .. automodule:: mitogen.master
39
40
41 mitogen.parent
42 --------------
43
44 .. automodule:: mitogen.parent
45
46
47 mitogen.fakessh
48 ---------------
49
50 .. image:: images/fakessh.svg
51 :class: mitogen-right-300
52
53 .. automodule:: mitogen.fakessh
54 .. currentmodule:: mitogen.fakessh
55 .. autofunction:: run (dest, router, args, daedline=None, econtext=None)
56
57
58
59 Message Class
60 =============
61
62 .. currentmodule:: mitogen.core
63 .. autoclass:: Message
64 :members:
65
66
67 Router Class
68 ============
69
70 .. currentmodule:: mitogen.core
71 .. autoclass:: Router
72 :members:
73
74
75 .. currentmodule:: mitogen.parent
76 .. autoclass:: Router
77 :members:
78
79
80 .. currentmodule:: mitogen.master
81 .. autoclass:: Router (broker=None)
82 :members:
83
84
85 .. _context-factories:
86
87 Connection Methods
88 ==================
89
90 .. currentmodule:: mitogen.parent
91 .. method:: Router.buildah (container=None, buildah_path=None, username=None, \**kwargs)
92
93 Construct a context on the local machine over a ``buildah`` invocation.
94 Accepts all parameters accepted by :meth:`local`, in addition to:
95
96 :param str container:
97 The name of the Buildah container to connect to.
98 :param str buildah_path:
99 Filename or complete path to the ``buildah`` binary. ``PATH`` will be
100 searched if given as a filename. Defaults to ``buildah``.
101 :param str username:
102 Username to use, defaults to unset.
103
104 .. currentmodule:: mitogen.parent
105 .. method:: Router.fork (on_fork=None, on_start=None, debug=False, profiling=False, via=None)
106
107 Construct a context on the local machine by forking the current
108 process. The forked child receives a new identity, sets up a new broker
109 and router, and responds to function calls identically to children
110 created using other methods.
111
112 The use of this method is strongly discouraged. It requires Python 2.6 or
113 newer, as older Pythons made no effort to reset threading state upon fork.
114
115 For long-lived processes, :meth:`local` is always better as it
116 guarantees a pristine interpreter state that inherited little from the
117 parent. Forking should only be used in performance-sensitive scenarios
118 where short-lived children must be spawned to isolate potentially buggy
119 code, and only after accounting for all the bad things possible as a
120 result of, at a minimum:
121
122 * Files open in the parent remaining open in the child,
123 causing the lifetime of the underlying object to be extended
124 indefinitely.
125
126 * From the perspective of external components, this is observable
127 in the form of pipes and sockets that are never closed, which may
128 break anything relying on closure to signal protocol termination.
129
130 * Descriptors that reference temporary files will not have their disk
131 space reclaimed until the child exits.
132
133 * Third party package state, such as urllib3's HTTP connection pool,
134 attempting to write to file descriptors shared with the parent,
135 causing random failures in both parent and child.
136
137 * UNIX signal handlers installed in the parent process remaining active
138 in the child, despite associated resources, such as service threads,
139 child processes, resource usage counters or process timers becoming
140 absent or reset in the child.
141
142 * Library code that makes assumptions about the process ID remaining
143 unchanged, for example to implement inter-process locking, or to
144 generate file names.
145
146 * Anonymous ``MAP_PRIVATE`` memory mappings whose storage requirement
147 doubles as either parent or child dirties their pages.
148
149 * File-backed memory mappings that cannot have their space freed on
150 disk due to the mapping living on in the child.
151
152 * Difficult to diagnose memory usage and latency spikes due to object
153 graphs becoming unreferenced in either parent or child, causing
154 immediate copy-on-write to large portions of the process heap.
155
156 * Locks held in the parent causing random deadlocks in the child, such
157 as when another thread emits a log entry via the :mod:`logging`
158 package concurrent to another thread calling :meth:`fork`, or when a C
159 extension module calls the C library allocator, or when a thread is using
160 the C library DNS resolver, for example via :func:`socket.gethostbyname`.
161
162 * Objects existing in Thread-Local Storage of every non-:meth:`fork`
163 thread becoming permanently inaccessible, and never having their
164 object destructors called, including TLS usage by native extension
165 code, triggering many new variants of all the issues above.
166
167 * Pseudo-Random Number Generator state that is easily observable by
168 network peers to be duplicate, violating requirements of
169 cryptographic protocols through one-time state reuse. In the worst
170 case, children continually reuse the same state due to repeatedly
171 forking from a static parent.
172
173 :meth:`fork` cleans up Mitogen-internal objects, in addition to
174 locks held by the :mod:`logging` package, reseeds
175 :func:`random.random`, and the OpenSSL PRNG via
176 :func:`ssl.RAND_add`, but only if the :mod:`ssl` module is
177 already loaded. You must arrange for your program's state, including
178 any third party packages in use, to be cleaned up by specifying an
179 `on_fork` function.
180
181 The associated stream implementation is
182 :class:`mitogen.fork.Stream`.
183
184 :param function on_fork:
185 Function invoked as `on_fork()` from within the child process. This
186 permits supplying a program-specific cleanup function to break
187 locks and close file descriptors belonging to the parent from
188 within the child.
189
190 :param function on_start:
191 Invoked as `on_start(econtext)` from within the child process after
192 it has been set up, but before the function dispatch loop starts.
193 This permits supplying a custom child main function that inherits
194 rich data structures that cannot normally be passed via a
195 serialization.
196
197 :param mitogen.core.Context via:
198 Same as the `via` parameter for :meth:`local`.
199
200 :param bool debug:
201 Same as the `debug` parameter for :meth:`local`.
202
203 :param bool profiling:
204 Same as the `profiling` parameter for :meth:`local`.
205
206 .. method:: Router.local (remote_name=None, python_path=None, debug=False, connect_timeout=None, profiling=False, via=None)
207
208 Construct a context on the local machine as a subprocess of the current
209 process. The associated stream implementation is
210 :class:`mitogen.master.Stream`.
211
212 :param str remote_name:
213 The ``argv[0]`` suffix for the new process. If `remote_name` is
214 ``test``, the new process ``argv[0]`` will be ``mitogen:test``.
215
216 If unspecified, defaults to ``<username>@<hostname>:<pid>``.
217
218 This variable cannot contain slash characters, as the resulting
219 ``argv[0]`` must be presented in such a way as to allow Python to
220 determine its installation prefix. This is required to support
221 virtualenv.
222
223 :param str|list python_path:
224 String or list path to the Python interpreter to use for bootstrap.
225 Defaults to :data:`sys.executable` for local connections, and
226 ``python`` for remote connections.
227
228 It is possible to pass a list to invoke Python wrapped using
229 another tool, such as ``["/usr/bin/env", "python"]``.
230
231 :param bool debug:
232 If :data:`True`, arrange for debug logging (:meth:`enable_debug`) to
233 be enabled in the new context. Automatically :data:`True` when
234 :meth:`enable_debug` has been called, but may be used
235 selectively otherwise.
236
237 :param bool unidirectional:
238 If :data:`True`, arrange for the child's router to be constructed
239 with :attr:`unidirectional routing
240 <mitogen.core.Router.unidirectional>` enabled. Automatically
241 :data:`True` when it was enabled for this router, but may still be
242 explicitly set to :data:`False`.
243
244 :param float connect_timeout:
245 Fractional seconds to wait for the subprocess to indicate it is
246 healthy. Defaults to 30 seconds.
247
248 :param bool profiling:
249 If :data:`True`, arrange for profiling (:data:`profiling`) to be
250 enabled in the new context. Automatically :data:`True` when
251 :data:`profiling` is :data:`True`, but may be used selectively
252 otherwise.
253
254 :param mitogen.core.Context via:
255 If not :data:`None`, arrange for construction to occur via RPCs
256 made to the context `via`, and for :data:`ADD_ROUTE
257 <mitogen.core.ADD_ROUTE>` messages to be generated as appropriate.
258
259 .. code-block:: python
260
261 # SSH to the remote machine.
262 remote_machine = router.ssh(hostname='mybox.com')
263
264 # Use the SSH connection to create a sudo connection.
265 remote_root = router.sudo(username='root', via=remote_machine)
266
267 .. method:: Router.doas (username=None, password=None, doas_path=None, password_prompt=None, incorrect_prompts=None, \**kwargs)
268
269 Construct a context on the local machine over a ``doas`` invocation.
270 The ``doas`` process is started in a newly allocated pseudo-terminal,
271 and supports typing interactive passwords.
272
273 Accepts all parameters accepted by :meth:`local`, in addition to:
274
275 :param str username:
276 Username to use, defaults to ``root``.
277 :param str password:
278 The account password to use if requested.
279 :param str doas_path:
280 Filename or complete path to the ``doas`` binary. ``PATH`` will be
281 searched if given as a filename. Defaults to ``doas``.
282 :param bytes password_prompt:
283 A string that indicates ``doas`` is requesting a password. Defaults
284 to ``Password:``.
285 :param list incorrect_prompts:
286 List of bytestrings indicating the password is incorrect. Defaults
287 to `(b"doas: authentication failed")`.
288 :raises mitogen.doas.PasswordError:
289 A password was requested but none was provided, the supplied
290 password was incorrect, or the target account did not exist.
291
292 .. method:: Router.docker (container=None, image=None, docker_path=None, \**kwargs)
293
294 Construct a context on the local machine within an existing or
295 temporary new Docker container using the ``docker`` program. One of
296 `container` or `image` must be specified.
297
298 Accepts all parameters accepted by :meth:`local`, in addition to:
299
300 :param str container:
301 Existing container to connect to. Defaults to :data:`None`.
302 :param str username:
303 Username within the container to :func:`setuid` to. Defaults to
304 :data:`None`, which Docker interprets as ``root``.
305 :param str image:
306 Image tag to use to construct a temporary container. Defaults to
307 :data:`None`.
308 :param str docker_path:
309 Filename or complete path to the Docker binary. ``PATH`` will be
310 searched if given as a filename. Defaults to ``docker``.
311
312 .. method:: Router.jail (container, jexec_path=None, \**kwargs)
313
314 Construct a context on the local machine within a FreeBSD jail using
315 the ``jexec`` program.
316
317 Accepts all parameters accepted by :meth:`local`, in addition to:
318
319 :param str container:
320 Existing container to connect to. Defaults to :data:`None`.
321 :param str username:
322 Username within the container to :func:`setuid` to. Defaults to
323 :data:`None`, which ``jexec`` interprets as ``root``.
324 :param str jexec_path:
325 Filename or complete path to the ``jexec`` binary. ``PATH`` will be
326 searched if given as a filename. Defaults to ``/usr/sbin/jexec``.
327
328 .. method:: Router.kubectl (pod, kubectl_path=None, kubectl_args=None, \**kwargs)
329
330 Construct a context in a container via the Kubernetes ``kubectl``
331 program.
332
333 Accepts all parameters accepted by :meth:`local`, in addition to:
334
335 :param str pod:
336 Kubernetes pod to connect to.
337 :param str kubectl_path:
338 Filename or complete path to the ``kubectl`` binary. ``PATH`` will
339 be searched if given as a filename. Defaults to ``kubectl``.
340 :param list kubectl_args:
341 Additional arguments to pass to the ``kubectl`` command.
342
343 .. method:: Router.lxc (container, lxc_attach_path=None, \**kwargs)
344
345 Construct a context on the local machine within an LXC classic
346 container using the ``lxc-attach`` program.
347
348 Accepts all parameters accepted by :meth:`local`, in addition to:
349
350 :param str container:
351 Existing container to connect to. Defaults to :data:`None`.
352 :param str lxc_attach_path:
353 Filename or complete path to the ``lxc-attach`` binary. ``PATH``
354 will be searched if given as a filename. Defaults to
355 ``lxc-attach``.
356
357 .. method:: Router.lxd (container, lxc_path=None, \**kwargs)
358
359 Construct a context on the local machine within a LXD container using
360 the ``lxc`` program.
361
362 Accepts all parameters accepted by :meth:`local`, in addition to:
363
364 :param str container:
365 Existing container to connect to. Defaults to :data:`None`.
366 :param str lxc_path:
367 Filename or complete path to the ``lxc`` binary. ``PATH`` will be
368 searched if given as a filename. Defaults to ``lxc``.
369
370 .. currentmodule:: mitogen.parent
371 .. method:: Router.podman (container=None, podman_path=None, username=None, \**kwargs)
372
373 Construct a context on the local machine over a ``podman`` invocation.
374 Accepts all parameters accepted by :meth:`local`, in addition to:
375
376 :param str container:
377 The name of the Podman container to connect to.
378 :param str podman_path:
379 Filename or complete path to the ``podman`` binary. ``PATH`` will be
380 searched if given as a filename. Defaults to ``podman``.
381 :param str username:
382 Username to use, defaults to unset.
383
384 .. method:: Router.setns (container, kind, username=None, docker_path=None, lxc_info_path=None, machinectl_path=None, \**kwargs)
385
386 Construct a context in the style of :meth:`local`, but change the
387 active Linux process namespaces via calls to `setns(2)` before
388 executing Python.
389
390 The namespaces to use, and the active root file system are taken from
391 the root PID of a running Docker, LXC, LXD, or systemd-nspawn
392 container.
393
394 The setns method depends on the built-in :mod:`ctypes` module, and thus
395 does not support Python 2.4.
396
397 A program is required only to find the root PID, after which management
398 of the child Python interpreter is handled directly.
399
400 :param str container:
401 Container to connect to.
402 :param str kind:
403 One of ``docker``, ``lxc``, ``lxd`` or ``machinectl``.
404 :param str username:
405 Username within the container to :func:`setuid` to. Defaults to
406 ``root``.
407 :param str docker_path:
408 Filename or complete path to the Docker binary. ``PATH`` will be
409 searched if given as a filename. Defaults to ``docker``.
410 :param str lxc_path:
411 Filename or complete path to the LXD ``lxc`` binary. ``PATH`` will
412 be searched if given as a filename. Defaults to ``lxc``.
413 :param str lxc_info_path:
414 Filename or complete path to the LXC ``lxc-info`` binary. ``PATH``
415 will be searched if given as a filename. Defaults to ``lxc-info``.
416 :param str machinectl_path:
417 Filename or complete path to the ``machinectl`` binary. ``PATH``
418 will be searched if given as a filename. Defaults to
419 ``machinectl``.
420
421 .. method:: Router.su (username=None, password=None, su_path=None, password_prompt=None, incorrect_prompts=None, \**kwargs)
422
423 Construct a context on the local machine over a ``su`` invocation. The
424 ``su`` process is started in a newly allocated pseudo-terminal, and
425 supports typing interactive passwords.
426
427 Accepts all parameters accepted by :meth:`local`, in addition to:
428
429 :param str username:
430 Username to pass to ``su``, defaults to ``root``.
431 :param str password:
432 The account password to use if requested.
433 :param str su_path:
434 Filename or complete path to the ``su`` binary. ``PATH`` will be
435 searched if given as a filename. Defaults to ``su``.
436 :param bytes password_prompt:
437 The string that indicates ``su`` is requesting a password. Defaults
438 to ``Password:``.
439 :param str incorrect_prompts:
440 Strings that signal the password is incorrect. Defaults to `("su:
441 sorry", "su: authentication failure")`.
442
443 :raises mitogen.su.PasswordError:
444 A password was requested but none was provided, the supplied
445 password was incorrect, or (on BSD) the target account did not
446 exist.
447
448 .. method:: Router.sudo (username=None, sudo_path=None, password=None, \**kwargs)
449
450 Construct a context on the local machine over a ``sudo`` invocation.
451 The ``sudo`` process is started in a newly allocated pseudo-terminal,
452 and supports typing interactive passwords.
453
454 Accepts all parameters accepted by :meth:`local`, in addition to:
455
456 :param str username:
457 Username to pass to sudo as the ``-u`` parameter, defaults to
458 ``root``.
459 :param str sudo_path:
460 Filename or complete path to the sudo binary. ``PATH`` will be
461 searched if given as a filename. Defaults to ``sudo``.
462 :param str password:
463 The password to use if/when sudo requests it. Depending on the sudo
464 configuration, this is either the current account password or the
465 target account password. :class:`mitogen.sudo.PasswordError`
466 will be raised if sudo requests a password but none is provided.
467 :param bool set_home:
468 If :data:`True`, request ``sudo`` set the ``HOME`` environment
469 variable to match the target UNIX account.
470 :param bool preserve_env:
471 If :data:`True`, request ``sudo`` to preserve the environment of
472 the parent process.
473 :param str selinux_type:
474 If not :data:`None`, the SELinux security context to use.
475 :param str selinux_role:
476 If not :data:`None`, the SELinux role to use.
477 :param list sudo_args:
478 Arguments in the style of :data:`sys.argv` that would normally
479 be passed to ``sudo``. The arguments are parsed in-process to set
480 equivalent parameters. Re-parsing ensures unsupported options cause
481 :class:`mitogen.core.StreamError` to be raised, and that
482 attributes of the stream match the actual behaviour of ``sudo``.
483
484 .. method:: Router.ssh (hostname, username=None, ssh_path=None, ssh_args=None, port=None, check_host_keys='enforce', password=None, identity_file=None, identities_only=True, compression=True, \**kwargs)
485
486 Construct a remote context over an OpenSSH ``ssh`` invocation.
487
488 The ``ssh`` process is started in a newly allocated pseudo-terminal to
489 support typing interactive passwords and responding to prompts, if a
490 password is specified, or `check_host_keys=accept`. In other scenarios,
491 ``BatchMode`` is enabled and no PTY is allocated. For many-target
492 configurations, both options should be avoided as most systems have a
493 conservative limit on the number of pseudo-terminals that may exist.
494
495 Accepts all parameters accepted by :meth:`local`, in addition to:
496
497 :param str username:
498 The SSH username; default is unspecified, which causes SSH to pick
499 the username to use.
500 :param str ssh_path:
501 Absolute or relative path to ``ssh``. Defaults to ``ssh``.
502 :param list ssh_args:
503 Additional arguments to pass to the SSH command.
504 :param int port:
505 Port number to connect to; default is unspecified, which causes SSH
506 to pick the port number.
507 :param str check_host_keys:
508 Specifies the SSH host key checking mode. Defaults to ``enforce``.
509
510 * ``ignore``: no host key checking is performed. Connections never
511 fail due to an unknown or changed host key.
512 * ``accept``: known hosts keys are checked to ensure they match,
513 new host keys are automatically accepted and verified in future
514 connections.
515 * ``enforce``: known host keys are checked to ensure they match,
516 unknown hosts cause a connection failure.
517 :param str password:
518 Password to type if/when ``ssh`` requests it. If not specified and
519 a password is requested, :class:`mitogen.ssh.PasswordError` is
520 raised.
521 :param str identity_file:
522 Path to an SSH private key file to use for authentication. Default
523 is unspecified, which causes SSH to pick the identity file.
524
525 When this option is specified, only `identity_file` will be used by
526 the SSH client to perform authenticaion; agent authentication is
527 automatically disabled, as is reading the default private key from
528 ``~/.ssh/id_rsa``, or ``~/.ssh/id_dsa``.
529 :param bool identities_only:
530 If :data:`True` and a password or explicit identity file is
531 specified, instruct the SSH client to disable any authentication
532 identities inherited from the surrounding environment, such as
533 those loaded in any running ``ssh-agent``, or default key files
534 present in ``~/.ssh``. This ensures authentication attempts only
535 occur using the supplied password or SSH key.
536 :param bool compression:
537 If :data:`True`, enable ``ssh`` compression support. Compression
538 has a minimal effect on the size of modules transmitted, as they
539 are already compressed, however it has a large effect on every
540 remaining message in the otherwise uncompressed stream protocol,
541 such as function call arguments and return values.
542 :param int ssh_debug_level:
543 Optional integer `0..3` indicating the SSH client debug level.
544 :raises mitogen.ssh.PasswordError:
545 A password was requested but none was specified, or the specified
546 password was incorrect.
547
548 :raises mitogen.ssh.HostKeyError:
549 When `check_host_keys` is set to either ``accept``, indicates a
550 previously recorded key no longer matches the remote machine. When
551 set to ``enforce``, as above, but additionally indicates no
552 previously recorded key exists for the remote machine.
553
554
555 Context Class
556 =============
557
558 .. currentmodule:: mitogen.core
559 .. autoclass:: Context
560 :members:
561
562
563 .. currentmodule:: mitogen.parent
564 .. autoclass:: Context
565 :members:
566
567 .. currentmodule:: mitogen.parent
568 .. autoclass:: CallChain
569 :members:
570
571
572 Receiver Class
573 ==============
574
575 .. currentmodule:: mitogen.core
576 .. autoclass:: Receiver
577 :members:
578
579
580 Sender Class
581 ============
582
583 .. currentmodule:: mitogen.core
584 .. autoclass:: Sender
585 :members:
586
587
588 Select Class
589 ============
590
591 .. module:: mitogen.select
592 .. currentmodule:: mitogen.select
593
594 .. autoclass:: Event
595 :members:
596
597 .. autoclass:: Select
598 :members:
599
600
601 Channel Class
602 =============
603
604 .. currentmodule:: mitogen.core
605 .. autoclass:: Channel
606 :members:
607
608
609
610 Broker Class
611 ============
612
613 .. currentmodule:: mitogen.core
614 .. autoclass:: Broker
615 :members:
616
617
618 .. currentmodule:: mitogen.master
619 .. autoclass:: Broker
620 :members:
621
622
623 Fork Safety
624 ===========
625
626 .. currentmodule:: mitogen.os_fork
627 .. autoclass:: Corker
628 :members:
629
630
631 Utility Functions
632 =================
633
634 .. currentmodule:: mitogen.core
635 .. function:: now
636
637 A reference to :func:`time.time` on Python 2, or :func:`time.monotonic` on
638 Python >3.3. We prefer :func:`time.monotonic` when available to ensure
639 timers are not impacted by system clock changes.
640
641
642 .. module:: mitogen.utils
643
644 A random assortment of utility functions useful on masters and children.
645
646 .. currentmodule:: mitogen.utils
647 .. autofunction:: cast
648
649
650 .. currentmodule:: mitogen.utils
651 .. autofunction:: setup_gil
652 .. autofunction:: disable_site_packages
653 .. autofunction:: log_to_file
654 .. autofunction:: run_with_router(func, \*args, \**kwargs)
655
656 .. currentmodule:: mitogen.utils
657 .. decorator:: with_router
658
659 Decorator version of :func:`run_with_router`. Example:
660
661 .. code-block:: python
662
663 @with_router
664 def do_stuff(router, arg):
665 pass
666
667 do_stuff(blah, 123)
668
669
670 Exceptions
671 ==========
672
673 .. currentmodule:: mitogen.core
674
675 .. autoclass:: Error
676 .. autoclass:: CallError
677 .. autoclass:: ChannelError
678 .. autoclass:: LatchError
679 .. autoclass:: StreamError
680 .. autoclass:: TimeoutError
681
682 .. currentmodule:: mitogen.parent
683 .. autoclass:: EofError
684 .. autoclass:: CancelledError