Debian Codegraph @

docs/ansible_detailed.rst

raw
1
2 .. _ansible_detailed:
3
4 Mitogen for Ansible
5 ===================
6
7 .. image:: images/mitogen.svg
8 :class: mitogen-right-180 mitogen-logo-wrap
9
10 **Mitogen for Ansible** is a completely redesigned UNIX connection layer and
11 module runtime for `Ansible`_. Requiring minimal configuration changes, it
12 updates Ansible's slow and wasteful shell-centric implementation with
13 pure-Python equivalents, invoked via highly efficient remote procedure calls to
14 persistent interpreters tunnelled over SSH. No changes are required to target
15 hosts.
16
17 The extension is considered stable and real-world use is encouraged.
18
19 .. _Ansible: https://www.ansible.com/
20
21 .. _Bug reports: https://github.com/mitogen-hq/mitogen/issues/new/choose
22
23
24 Overview
25 --------
26
27 **Expect a 1.25x - 7x speedup** and a **CPU usage reduction of at least 2x**,
28 depending on network conditions, modules executed, and time already spent by
29 targets on useful work. Mitogen cannot improve a module once it is executing,
30 it can only ensure the module executes as quickly as possible.
31
32 * **One connection is used per target**, in addition to one sudo invocation per
33 user account. This is much better than SSH multiplexing combined with
34 pipelining, as significant state can be maintained in RAM between steps, and
35 system logs aren't spammed with repeat authentication events.
36
37 * **A single network roundtrip is used** to execute a step whose code already
38 exists in RAM on the target. Eliminating multiplexed SSH channel creation
39 saves 4 ms runtime per 1 ms of network latency for every playbook step.
40
41 * **Processes are aggressively reused**, avoiding the cost of invoking Python
42 and recompiling imports, saving 300-800 ms for every playbook step.
43
44 * Code is ephemerally cached in RAM, **reducing bandwidth usage by an order
45 of magnitude** compared to SSH pipelining, with around 5x fewer frames
46 traversing the network in a typical run.
47
48 * **Fewer writes to the target filesystem occur**. In typical configurations,
49 Ansible repeatedly rewrites and extracts ZIP files to multiple temporary
50 directories on the target. Security issues relating to temporary files in
51 cross-account scenarios are entirely avoided.
52
53 The effect is most potent on playbooks that execute many **short-lived
54 actions**, where Ansible's overhead dominates the cost of the operation, for
55 example when executing large ``with_items`` loops to run simple commands or
56 write files.
57
58
59 Installation
60 ------------
61
62 1. Review :ref:`noteworthy_differences`.
63 2. Download and extract |mitogen_url|.
64 3. Modify ``ansible.cfg``:
65
66 .. parsed-literal::
67
68 [defaults]
69 strategy_plugins = /path/to/mitogen-|mitogen_version|/ansible_mitogen/plugins/strategy
70 strategy = mitogen_linear
71
72 The ``strategy`` key is optional. If omitted, the
73 ``ANSIBLE_STRATEGY=mitogen_linear`` environment variable can be set on a
74 per-run basis. Like ``mitogen_linear``, the ``mitogen_free`` and
75 ``mitogen_host_pinned`` strategies exists to mimic the ``free`` and
76 ``host_pinned`` strategies.
77
78
79 Demo
80 ~~~~
81
82 This demonstrates Ansible running a subset of the Mitogen integration tests
83 concurrent to an equivalent run using the extension.
84
85 .. raw:: html
86
87 <iframe src="https://player.vimeo.com/video/283272293?title=0&byline=0&portrait=0" width="720" height="439" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>
88
89
90 Testimonials
91 ~~~~~~~~~~~~
92
93 * "With mitogen **my playbook runtime went from 45 minutes to just under 3
94 minutes**. Awesome work!"
95
96 * "The runtime was reduced from **1.5 hours on 4 servers to just under 3
97 minutes**. Thanks!"
98
99 * "Oh, performance improvement using Mitogen is *huge*. As mentioned before,
100 running with Mitogen enables takes 7m36 (give or take a few seconds). Without
101 Mitogen, the same run takes 19m49! **I'm not even deploying without Mitogen
102 anymore** :)"
103
104 * "**Works like a charm**, thank you for your quick response"
105
106 * "I tried it out. **He is not kidding about the speed increase**."
107
108 * "I don't know what kind of dark magic @dmw_83 has done, but his Mitogen
109 strategy took Clojars' Ansible runs from **14 minutes to 2 minutes**. I still
110 can't quite believe it."
111
112 * "Enabling the mitogen plugin in ansible feels like switching from floppy to SSD"
113
114
115 .. _noteworthy_differences:
116
117 Noteworthy Differences
118 ----------------------
119
120 * Mitogen 0.2.x supports Ansible 2.3-2.9; with Python 2.6, 2.7, or 3.6.
121 Mitogen 0.3.1+ supports
122
123 +-----------------+-----------------+
124 | Ansible version | Python versions |
125 +=================+=================+
126 | 2.10 | |
127 +-----------------+ |
128 | 3 | 2.7, 3.6 - 3.11 |
129 +-----------------+ |
130 | 4 | |
131 +-----------------+-----------------+
132 | 5 | 3.8 - 3.11 |
133 +-----------------+-----------------+
134 | 6 | |
135 +-----------------+ 3.8 - 3.13 |
136 | 7 | |
137 +-----------------+-----------------+
138 | 8 | 3.9 - 3.13 |
139 +-----------------+-----------------+
140 | 9 | |
141 +-----------------+ 3.10 - 3.14 |
142 | 10 | |
143 +-----------------+-----------------+
144 | 11 | |
145 +-----------------+ 3.11 - 3.14 |
146 | 12 | |
147 +-----------------+-----------------+
148 | 13 | |
149 +-----------------+ 3.12 - 3.14 |
150 | 14 | |
151 +-----------------+-----------------+
152
153 Verify your installation is running one of these versions by checking
154 ``ansible --version`` output.
155
156 * The ``raw`` action executes as a regular Mitogen connection, which requires
157 Python on the target, precluding its use for installing Python. This will be
158 addressed in a future release. For now, simply mix Mitogen and vanilla
159 Ansible strategies:
160
161 .. code-block:: yaml
162
163 - hosts: web-servers
164 strategy: linear
165 tasks:
166 - name: Install Python if necessary.
167 raw: test -e /usr/bin/python || apt install -y python-minimal
168
169 - hosts: web-servers
170 strategy: mitogen_linear
171 roles:
172 - nginx
173 - initech_app
174 - y2k_fix
175
176 * Ansible `become plugins
177 <https://docs.ansible.com/ansible/latest/plugins/become.html>`_ are not yet
178 supported.
179
180 * The ``doas``, ``su`` and ``sudo`` become methods are available. File bugs to
181 register interest in more.
182
183 * The ``sudo`` comands executed differ slightly compared to Ansible. In some
184 cases where the target has a ``sudo`` configuration that restricts the exact
185 commands allowed to run, it may be necessary to add a ``sudoers`` rule like:
186
187 ::
188
189 your_ssh_username = (ALL) NOPASSWD:/usr/bin/python -c*
190
191 * The :ans:conn:`~buildah`, :ans:conn:`~docker`, :ans:conn:`~jail`,
192 :ans:conn:`~kubectl`, :ans:conn:`~local`, :ans:conn:`~lxd`,
193 :ans:conn:`~podman`, & :ans:conn:`~ssh` connection types are supported; also
194 Mitogen-specific :ref:`mitogen_doas <doas>`, :ref:`machinectl <machinectl>`,
195 :ref:`mitogen_su <su>`, :ref:`mitogen_sudo <sudo>`, and :ref:`setns <setns>`
196 types. File bugs to register interest in others.
197
198 * Actions are single-threaded for each `(host, user account)` combination,
199 including actions that execute on the local machine. Playbooks may experience
200 slowdown compared to vanilla Ansible if they employ long-running
201 ``local_action`` or ``delegate_to`` tasks delegating many target hosts to a
202 single machine and user account.
203
204 Ansible usually permits up to ``forks`` simultaneous local actions. Any
205 long-running local actions that execute for every target will experience
206 artificial serialization, causing slowdown equivalent to `task_duration *
207 num_targets`. This will be addressed soon.
208
209 * The Ansible 2.7 :ans:mod:`reboot` may require a ``pre_reboot_delay`` on
210 systemd hosts, as insufficient time exists for the reboot command's exit
211 status to be reported before necessary processes are torn down.
212
213 * On OS X when a SSH password is specified and the default connection type of
214 :ans:conn:`~smart` is used, Ansible may select the :ans:conn:`paramiko_ssh`
215 rather than Mitogen. If you specify a password on OS X, ensure ``connection:
216 ssh`` appears in your playbook, ``ansible.cfg``, or as ``-c ssh`` on the
217 command-line.
218
219 * Ansible permits up to ``forks`` connections to be setup in parallel, whereas
220 in Mitogen this is handled by a fixed-size thread pool. Up to 32 connections
221 may be established in parallel by default, this can be modified by setting
222 the ``MITOGEN_POOL_SIZE`` environment variable.
223
224 * Performance does not scale cleanly with target count. This will improve over
225 time.
226
227 * Performance on Python 3 is significantly worse than on Python 2. While this
228 has not yet been investigated, at least some of the regression appears to be
229 part of the core library, and should therefore be straightforward to fix as
230 part of 0.2.x.
231
232 * Connection and become timeouts are applied differently. Mitogen may consider
233 a connection to have timed out, when Ansible would have waited longer or
234 indefinately. For example if SSH authentication completes within the
235 timeout, but execution of login scripts exceeds it - then Mitogen will
236 consider the task to have timed out and that host to have failed.
237
238 ..
239 tests/ansible/integration/ssh/timeouts.yml covers (some of) this behaviour.
240
241 ..
242 * SSH and ``become`` are treated distinctly when applying timeouts, and
243 timeouts apply up to the point when the new interpreter is ready to accept
244 messages. Ansible has two timeouts: ``ConnectTimeout`` for SSH, applying up
245 to when authentication completes, and a separate parallel timeout up to
246 when ``become`` authentication completes.
247 For busy targets, Ansible may successfully execute a module where Mitogen
248 would fail without increasing the timeout. For sick targets, Ansible may
249 hang indefinitely after authentication without executing a command, for
250 example due to a stuck filesystem IO appearing in ``$HOME/.profile``.
251
252 ..
253 * "Module Replacer" style modules are not supported. These rarely appear in
254 practice, and light web searches failed to reveal many examples of them.
255
256 * The ``ansible_python_interpreter`` variable is parsed using a restrictive
257 :mod:`shell-like <shlex>` syntax, permitting values such as ``/usr/bin/env
258 FOO=bar python`` or ``source /opt/rh/rh-python36/enable && python``.
259 Jinja2 templating is also supported for complex task-level
260 interpreter settings. Ansible documents `ansible_python_interpreter
261 <https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html#ansible-python-interpreter>`_
262 as an absolute path and releases since June 2024 (e.g. Ansible 10.1)
263 reflect this. Older Ansible releases passed it to the shell unquoted.
264
265 ..
266 * Configurations will break that rely on the `hashbang argument splitting
267 behaviour <https://github.com/ansible/ansible/issues/15635>`_ of the
268 ``ansible_python_interpreter`` setting, contrary to the Ansible
269 documentation. This will be addressed in a future 0.2 release.
270
271
272 New Features & Notes
273 --------------------
274
275
276 Connection Delegation
277 ~~~~~~~~~~~~~~~~~~~~~
278
279 .. image:: images/jumpbox.svg
280 :class: mitogen-right-275
281
282 Included is a preview of **Connection Delegation**, a Mitogen-specific
283 implementation of `stackable connection plug-ins`_. This enables connections
284 via a bastion, or container connections delegated via their host machine, where
285 reaching the host may entail further delegation.
286
287 .. _Stackable connection plug-ins: https://github.com/ansible/proposals/issues/25
288
289 Unlike with SSH forwarding Ansible has complete visibility of the final
290 topology, declarative configuration via static/dynamic inventory is possible,
291 and data can be cached and re-served, and code executed on every intermediary.
292
293 For example when targeting Docker containers on a remote machine, each module
294 need only be uploaded once for the first task and container that requires it,
295 then cached and served from the SSH account for every future task in any
296 container.
297
298 .. raw:: html
299
300 <div style="clear: both;"></div>
301
302
303 .. caution::
304
305 Connection delegation is a work in progress, bug reports are welcome.
306
307 * Delegated connection setup is single-threaded; only one connection can be
308 constructed in parallel per intermediary.
309
310 * Inferring the configuration of intermediaries may be buggy, manifesting
311 as duplicate connections between hops, due to not perfectly replicating
312 the configuration Ansible would normally use for the intermediary.
313
314 * Intermediary machines cannot use login and become passwords that were
315 supplied to Ansible interactively. If an intermediary requires a
316 password, it must be supplied via ``ansible_ssh_pass``,
317 ``ansible_ssh_password``, ``ansible_password``, or
318 ``ansible_become_pass`` inventory variables.
319
320 * Automatic tunnelling of SSH-dependent actions, such as the
321 ``synchronize`` module, is not yet supported. This will be addressed in a
322 future release.
323
324 To enable connection delegation, set ``mitogen_via=<inventory name>`` on the
325 command line, or as host and group variables.
326
327 .. code-block:: ini
328
329 # Docker container on web1.dc1 is reachable via web1.dc1.
330 [app-containers.web1.dc1]
331 app1.web1.dc1 ansible_host=app1 ansible_connection=docker mitogen_via=web1.dc1
332
333 # Web servers in DC1 are reachable via bastion.dc1
334 [dc1]
335 web1.dc1
336 web2.dc1
337 web3.dc1
338
339 [dc1:vars]
340 mitogen_via = bastion.dc1
341
342 # Web servers in DC2 are reachable via bastion.dc2
343 [dc2]
344 web1.dc2
345 web2.dc2
346 web3.dc2
347
348 [dc2:vars]
349 mitogen_via = bastion.dc2
350
351 # Prod bastions are reachable via a magic account on a
352 # corporate network gateway.
353 [bastions]
354 bastion.dc1 [email protected]
355 bastion.dc2 [email protected]
356
357 [corp-gateway]
358 corp-gateway.internal
359
360
361 File Transfer
362 ~~~~~~~~~~~~~
363
364 Normally :linux:man1:`sftp` or :linux:man1:`scp` are used to copy files by the
365 :ans:mod:`~assemble`, :ans:mod:`~aws_s3`, :ans:mod:`~copy`, :ans:mod:`~patch`,
366 :ans:mod:`~script`, :ans:mod:`~template`, :ans:mod:`~unarchive`, and
367 :ans:mod:`~uri` actions, or when uploading modules with pipelining disabled.
368 With Mitogen copies are implemented natively using the same interpreters,
369 connection tree, and routed message bus that carries RPCs.
370
371 This permits direct streaming between endpoints regardless of execution
372 environment, without necessitating temporary copies in intermediary accounts or
373 machines, for example when ``become`` is active, or in the presence of
374 connection delegation. It also avoids the need to securely share temporary
375 files between accounts and machines.
376
377 As the implementation is self-contained, it is simple to make improvements like
378 prioritizing transfers, supporting resume, or displaying progress bars.
379
380
381 Safety
382 ^^^^^^
383
384 Transfers proceed to a hidden file in the destination directory, with content
385 and metadata synced using :linux:man2:`fsync` prior to rename over any existing
386 file. This ensures the file remains consistent at all times, in the event of a
387 crash, or when overlapping `ansible-playbook` runs deploy differing file
388 contents.
389
390 The :linux:man1:`sftp` and :linux:man1:`scp` tools may cause undetected data
391 corruption in the form of truncated files, or files containing intermingled
392 data segments from overlapping runs. As part of normal operation, both tools
393 expose a window where readers may observe inconsistent file contents.
394
395
396 Performance
397 ^^^^^^^^^^^
398
399 One roundtrip initiates a transfer larger than 124 KiB, while smaller transfers
400 are embedded in a 0-roundtrip pipelined call. For tools operating via SSH
401 multiplexing, 4 roundtrips are required to configure the IO channel, followed
402 by 6 roundtrips to transfer the file in the case of ``sftp``, in addition to
403 the time to start the local and remote processes.
404
405 An invocation of ``scp`` with an empty ``.profile`` over a 30 ms link takes
406 ~140 ms, wasting 110 ms per invocation, rising to ~2,000 ms over a 400 ms
407 UK-India link, wasting 1,600 ms per invocation.
408
409
410 Interpreter Reuse
411 ~~~~~~~~~~~~~~~~~
412
413 Python interpreters are aggressively reused to execute modules. While this
414 works well, it violates an unwritten assumption, and so it is possible an
415 earlier module execution could cause a subsequent module to fail, or for
416 unrelated modules to interact poorly due to bad hygiene, such as
417 monkey-patching that becomes stacked over repeat invocations.
418
419 Before reporting a bug relating to a misbehaving module, please re-run with
420 ``-e mitogen_task_isolation=fork`` to see if the problem abates. This may be
421 set per-task, paying attention to the possibility an earlier task may be the
422 true cause of a failure.
423
424 .. code-block:: yaml
425
426 - name: My task.
427 broken_module:
428 some_option: true
429 vars:
430 mitogen_task_isolation: fork
431
432 If forking solves your problem, **please report a bug regardless**, as an
433 internal list can be updated to prevent others bumping into the same problem.
434
435
436 Interpreter Recycling
437 ~~~~~~~~~~~~~~~~~~~~~
438
439 There is a per-target limit on the number of interpreters. Once 20 exist, the
440 youngest is terminated before starting any new interpreter, preventing
441 situations like below from triggering memory exhaustion.
442
443 .. code-block:: yaml
444
445 - hosts: corp_boxes
446 vars:
447 user_directory: [
448 # 10,000 corporate user accounts
449 ]
450 tasks:
451 - name: Create user bashrc
452 become: true
453 vars:
454 ansible_become_user: "{{item}}"
455 copy:
456 src: bashrc
457 dest: "~{{item}}/.bashrc"
458 with_items: "{{user_directory}}"
459
460 The youngest is chosen to preserve useful accounts like ``root`` and
461 ``postgresql`` that often appear early in a run, however it is simple to
462 construct a playbook that defeats this strategy. A future version will key
463 interpreters on the identity of their creating task, avoiding useful account
464 recycling in every scenario.
465
466 To modify the limit, set the ``MITOGEN_MAX_INTERPRETERS`` environment variable.
467
468
469 Standard IO
470 ~~~~~~~~~~~
471
472 Ansible uses pseudo TTYs for most invocations to allow it to type interactive
473 passwords, however pseudo TTYs are disabled where standard input is required or
474 ``sudo`` is not in use. Additionally when SSH multiplexing is enabled, a string
475 like ``Shared connection to localhost closed\r\n`` appears in ``stderr`` of
476 every invocation.
477
478 Mitogen does not naturally require either of these, as command output is always
479 embedded within framed messages, and it can simply call :py:func:`pty.openpty`
480 in any location an interactive password must be typed.
481
482 A major downside to Ansible's behaviour is that ``stdout`` and ``stderr`` are
483 merged together into a single ``stdout`` variable, with carriage returns
484 inserted in the output by the TTY layer. However ugly, the extension emulates
485 this precisely, to avoid breaking playbooks that expect text to appear in
486 specific variables with a particular linefeed style.
487
488
489 .. _ansible_tempfiles:
490
491 Temporary Files
492 ~~~~~~~~~~~~~~~
493
494 Temporary file handling in Ansible is tricky, and the precise behaviour varies
495 across major versions. A variety of temporary files and directories are
496 created, depending on the operating mode.
497
498 In the best case when pipelining is enabled and no temporary uploads are
499 required, for each task Ansible will create one directory below a
500 system-supplied temporary directory returned by :func:`tempfile.mkdtemp`, owned
501 by the target account a new-style module will execute in.
502
503 In other cases depending on the task type, whether become is active, whether
504 the target become user is privileged, whether the associated action plugin
505 needs to upload files, and whether the associated module needs to store files,
506 Ansible may:
507
508 * Create a directory owned by the SSH user either under ``remote_tmp``, or a
509 system-default directory,
510 * Upload action dependencies such as non-new style modules or rendered
511 templates to that directory via :linux:man1:`sftp` or :linux:man1:`scp`.
512 * Attempt to modify the directory's access control list to grant access to the
513 target user using :linux:man1:`setfacl`, requiring that tool to be installed
514 and a supported filesystem to be in use, or for the
515 ``allow_world_readable_tmpfiles`` setting to be :data:`True`.
516 * Create a directory owned by the target user either under ``remote_tmp``, or
517 a system-default directory, if a new-style module needs a temporary directory
518 and one was not previously created for a supporting file earlier in the
519 invocation.
520
521 In summary, for each task Ansible may create one or more of:
522
523 * ``~ssh_user/<remote_tmp>/...`` owned by the login user,
524 * ``$TMPDIR/ansible-tmp-...`` owned by the login user,
525 * ``$TMPDIR/ansible-tmp-...`` owned by the login user with ACLs permitting
526 write access by the become user,
527 * ``~become_user/<remote_tmp>/...`` owned by the become user,
528 * ``$TMPDIR/ansible_<modname>_payload_.../`` owned by the become user,
529 * ``$TMPDIR/ansible-module-tmp-.../`` owned by the become user.
530
531
532 Mitogen for Ansible
533 ^^^^^^^^^^^^^^^^^^^
534
535 As Mitogen can execute new-style modules from RAM, and transfer files to target
536 user accounts without first writing an intermediary file in any separate login
537 account, handling is relatively simplified.
538
539 Temporary directories must exist to maintain compatibility with Ansible, as
540 many modules introspect :data:`sys.argv` to find a directory where they may
541 write files, however only one directory exists for the lifetime of each
542 interpreter, its location is consistent for each account, and it is always
543 privately owned by that account.
544
545 During startup, the persistent remote interpreter tries the paths below until
546 one is found that is writeable:
547
548 1. ``$variable`` and tilde-expanded ``remote_tmp`` setting from
549 ``ansible.cfg``
550 2. ``$variable`` and tilde-expanded ``system_tmpdirs`` setting from
551 ``ansible.cfg``
552 3. ``TMPDIR`` environment variable
553 4. ``TEMP`` environment variable
554 5. ``TMP`` environment variable
555 6. ``/tmp``
556 7. ``/var/tmp``
557 8. ``/usr/tmp``
558 9. Current working directory
559
560 The directory is created at startup and recursively destroyed during interpeter
561 shutdown. Subdirectories are automatically created and destroyed by the
562 controller for each task that requires them.
563
564
565 Round-trip Avoidance
566 ^^^^^^^^^^^^^^^^^^^^
567
568 Mitogen avoids many round-trips due to temporary file handling that are present
569 in regular Ansible:
570
571 * During task startup, it is not necessary to wait until the target has
572 succeeded in creating a temporary directory. Instead, any failed attempt to
573 create the directory will cause any subsequent RPC belonging to the same task
574 to fail with the error that occurred.
575
576 * As temporary directories are privately owned by the target user account,
577 operations relating to modifying the directory to support cross-account
578 access are avoided.
579
580 * An explicit work-around is included to avoid the :ans:mod:`~copy` and
581 :ans:mod:`~template` actions needlessly triggering a round-trip to set their
582 temporary file as executable.
583
584 * During task shutdown, it is not necessary to wait to learn if the target has
585 succeeded in deleting a temporary directory, since any error that may occur
586 is logged asynchronously via the logging framework, and the persistent
587 remote interpreter arranges for all subdirectories to be destroyed during
588 interpreter shutdown.
589
590
591 .. _ansible_process_env:
592
593 Process Environment Emulation
594 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
595
596 Since Ansible discards processes after each module invocation, follow-up tasks
597 often (but not always) receive a new environment that will usually include
598 changes made by previous tasks. As such modifications are common, for
599 compatibility the extension emulates the existing behaviour as closely as
600 possible.
601
602 Some scenarios exist where emulation is impossible, for example, applying
603 ``nsswitch.conf`` changes when ``nscd`` is not in use. If future scenarios
604 appear that cannot be solved through emulation, the extension will be updated
605 to automatically restart affected interpreters instead.
606
607
608 DNS Resolution
609 ^^^^^^^^^^^^^^
610
611 Modifications to ``/etc/resolv.conf`` cause the glibc resolver configuration to
612 be reloaded via :linux:man3:`res_init`. This isn't necessary on some Linux
613 distributions carrying glibc patches to automatically check
614 ``/etc/resolv.conf`` periodically, however it is necessary on at least Debian
615 and BSD derivatives.
616
617
618 ``/etc/environment``
619 ^^^^^^^^^^^^^^^^^^^^
620
621 When ``become: true`` is active or SSH multiplexing is disabled, modifications
622 by previous tasks to ``/etc/environment`` and ``$HOME/.pam_environment`` are
623 normally reflected, since the content of those files is reapplied by `PAM
624 <https://en.wikipedia.org/wiki/Pluggable_authentication_module>`_ via `pam_env`
625 on each authentication of ``sudo`` or ``sshd``.
626
627 Both files are monitored for changes, and changes are applied where it appears
628 safe to do so:
629
630 * New keys are added if they did not otherwise exist in the inherited
631 environment, or previously had the same value as found in the file before it
632 changed.
633
634 * Given a key (such as ``http_proxy``) added to the file where no such key
635 exists in the environment, the key will be added.
636
637 * Given a key (such as ``PATH``) where an existing environment key exists with
638 a different value, the update or deletion will be ignored, as it is likely
639 the key was overridden elsewhere after `pam_env` ran, such as by
640 ``/etc/profile``.
641
642 * Given a key removed from the file that had the same value as the existing
643 environment key, the key will be removed.
644
645
646 How Modules Execute
647 ~~~~~~~~~~~~~~~~~~~
648
649 Ansible usually modifies, recompresses and reuploads modules every time they
650 run on a target, work that must be repeated by the controller for every
651 playbook step.
652
653 With the extension any modifications are done on the target, allowing pristine
654 copies of modules to be cached, reducing the necessity to re-transfer modules
655 for each invocation. Unmodified modules are uploaded once on first use and
656 cached in RAM for the remainder of the run.
657
658 **Binary**
659 Native executables detected using a complex heuristic. Arguments are
660 supplied as a JSON file whose path is the sole script parameter.
661
662 **Module Replacer**
663 Python scripts detected by the presence of
664 ``#<<INCLUDE_ANSIBLE_MODULE_COMMON>>`` appearing in their source. This type
665 is not yet supported.
666
667 **New-Style**
668 Python scripts detected by the presence of ``from ansible.module_utils.``
669 appearing in their source. Arguments are supplied as JSON written to
670 ``sys.stdin`` of the target interpreter.
671
672 **JSON_ARGS**
673 Detected by the presence of ``INCLUDE_ANSIBLE_MODULE_JSON_ARGS`` appearing
674 in the script source. The interpreter directive (``#!interpreter``) is
675 adjusted to match the corresponding value of ``{{ansible_*_interpreter}}``
676 if one is set. Arguments are supplied as JSON mixed into the script as a
677 replacement for ``INCLUDE_ANSIBLE_MODULE_JSON_ARGS``.
678
679 **WANT_JSON**
680 Detected by the presence of ``WANT_JSON`` appearing in the script source.
681 The interpreter directive is adjusted as above. Arguments are supplied as a
682 JSON file whose path is the sole script parameter.
683
684 **Old Style**
685 Files not matching any of the above tests. The interpreter directive is
686 adjusted as above. Arguments are supplied as a file whose path is the sole
687 script parameter. The format of the file is ``"key=repr(value)[
688 key2=repr(value2)[ ..]] "``.
689
690
691 Runtime Patches
692 ~~~~~~~~~~~~~~~
693
694 Three small runtime patches are employed in ``strategy.py`` to hook into
695 desirable locations, in order to override uses of shell, the module executor,
696 and the mechanism for selecting a connection plug-in. While it is hoped the
697 patches can be avoided in future, for interesting versions of Ansible deployed
698 today this simply is not possible, and so they continue to be required.
699
700 The patches are concise and behave conservatively, including by disabling
701 themselves when non-Mitogen connections are in use. Additional third party
702 plug-ins are unlikely to attempt similar patches, so the risk to an established
703 configuration should be minimal.
704
705
706 Flag Emulation
707 ~~~~~~~~~~~~~~
708
709 Mitogen re-parses ``sudo_flags``, ``become_flags``, and ``ssh_flags`` using
710 option parsers extracted from `sudo(1)` and `ssh(1)` in order to emulate their
711 equivalent semantics. This allows:
712
713 * robust support for common ``ansible.cfg`` tricks without reconfiguration,
714 such as forwarding SSH agents across ``sudo`` invocations,
715 * reporting on conflicting flag combinations,
716 * reporting on unsupported flag combinations,
717 * internally special-casing certain behaviour (like recursive agent forwarding)
718 without boring the user with the details,
719 * avoiding opening the extension up to untestable scenarios where users can
720 insert arbitrary garbage between Mitogen and the components it integrates
721 with,
722 * precise emulation by an alternative implementation, for example if Mitogen
723 grew support for Paramiko.
724
725
726 Connection Types
727 ----------------
728
729 Matching Ansible, connection variables are treated on a per-task basis, causing
730 establishment of additional reuseable interpreters as necessary to match the
731 configuration of each task.
732
733
734 .. _method-buildah:
735
736 Buildah
737 ~~~~~~~
738
739 Like the :ans:conn:`buildah` except connection delegation is supported.
740
741 * ``ansible_host``: Name of Buildah container (default: inventory hostname).
742 * ``ansible_user``: Name of user within the container to execute as.
743
744
745 .. _doas:
746
747 Doas
748 ~~~~
749
750 ``doas`` can be used as a connection method that supports connection delegation, or
751 as a become method.
752
753 When used as a become method:
754
755 * ``ansible_python_interpreter``
756 * ``ansible_become_exe`` / ``ansible_doas_exe``: path to ``doas`` binary.
757 * ``ansible_become_user`` (default: ``root``)
758 * ``ansible_become_pass`` (default: assume passwordless)
759 * ``mitogen_mask_remote_name``: if :data:`True`, mask the identity of the
760 Ansible controller process on remote machines. To simplify diagnostics,
761 Mitogen produces remote processes named like
762 `"mitogen:[email protected]:1234"`, however this may be a privacy issue in
763 some circumstances.
764 * ansible.cfg: ``timeout``
765
766 When used as the ``mitogen_doas`` connection method:
767
768 * The inventory hostname has no special meaning.
769 * ``ansible_user``: username to use.
770 * ``ansible_password``: password to use.
771 * ``ansible_doas_exe``: path to ``doas`` binary.
772 * ``ansible_python_interpreter``
773
774
775 .. _method-docker:
776
777 Docker
778 ~~~~~~
779
780 Like the :ans:conn:`docker` except connection delegation is supported.
781
782 * ``ansible_host``: Name of Docker container (default: inventory hostname).
783 * ``ansible_user``: Name of user within the container to execute as.
784 * ``mitogen_mask_remote_name``: if :data:`True`, mask the identity of the
785 Ansible controller process on remote machines. To simplify diagnostics,
786 Mitogen produces remote processes named like
787 `"mitogen:[email protected]:1234"`, however this may be a privacy issue in
788 some circumstances.
789
790
791 .. _method-jail:
792
793 FreeBSD Jail
794 ~~~~~~~~~~~~
795
796 Like the :ans:conn:`jail` except connection delegation is supported.
797
798 * ``ansible_host``: Name of jail (default: inventory hostname).
799 * ``ansible_user``: Name of user within the jail to execute as.
800 * ``mitogen_mask_remote_name``: if :data:`True`, mask the identity of the
801 Ansible controller process on remote machines. To simplify diagnostics,
802 Mitogen produces remote processes named like
803 `"mitogen:[email protected]:1234"`, however this may be a privacy issue in
804 some circumstances.
805
806
807 .. _method-kubectl:
808
809 Kubernetes Pod
810 ~~~~~~~~~~~~~~
811
812 Like the :ans:conn:`kubectl` except connection delegation is supported.
813
814 * ``ansible_host``: Name of pod (default: inventory hostname).
815 * ``ansible_user``: Name of user to authenticate to API as.
816 * ``mitogen_mask_remote_name``: if :data:`True`, mask the identity of the
817 Ansible controller process on remote machines. To simplify diagnostics,
818 Mitogen produces remote processes named like
819 `"mitogen:[email protected]:1234"`, however this may be a privacy issue in
820 some circumstances.
821
822
823 Local
824 ~~~~~
825
826 Like the :ans:conn:`local` except connection delegation is supported.
827
828 * ``ansible_python_interpreter``
829
830
831 Podman
832 ~~~~~~
833
834 Like :ans:conn:`podman` except connection delegation is supported.
835
836 * ``ansible_host``: Name of container (default: inventory hostname).
837 * ``ansible_user``: Name of user within the container to execute as.
838 * ``mitogen_mask_remote_name``: if :data:`True`, mask the identity of the
839 Ansible controller process on remote machines. To simplify diagnostics,
840 Mitogen produces remote processes named like
841 `"mitogen:[email protected]:1234"`, however this may be a privacy issue in
842 some circumstances.
843
844
845 Process Model
846 ^^^^^^^^^^^^^
847
848 Ansible usually executes local connection commands as a transient subprocess of
849 the forked worker executing a task. With the extension, the local connection
850 exists as a persistent subprocess of the connection multiplexer.
851
852 This means that global state mutations made to the top-level Ansible process
853 that are normally visible to newly forked subprocesses, such as vars plug-ins
854 that modify the environment, will not be reflected when executing local
855 commands without additional effort.
856
857 During execution the extension presently mimics the working directory and
858 process environment inheritence of regular Ansible, however it is possible some
859 additional differences exist that may break existing playbooks.
860
861
862 .. _method-lxc:
863
864 LXC
865 ~~~
866
867 Connect to classic LXC containers, like the :ans:conn:`lxc` except connection
868 delegation is supported, and ``lxc-attach`` is always used rather than the LXC
869 Python bindings, as is usual with ``lxc``.
870
871 * ``ansible_python_interpreter``
872 * ``ansible_host``: Name of LXC container (default: inventory hostname).
873 * ``mitogen_lxc_attach_path``: path to ``lxc-attach`` command if not available
874 on the system path.
875 * ``mitogen_mask_remote_name``: if :data:`True`, mask the identity of the
876 Ansible controller process on remote machines. To simplify diagnostics,
877 Mitogen produces remote processes named like
878 `"mitogen:[email protected]:1234"`, however this may be a privacy issue in
879 some circumstances.
880
881
882 .. _method-lxd:
883
884 LXD
885 ~~~
886
887 Connect to modern LXD containers, like the :ans:conn:`lxd` except connection
888 delegation is supported. The ``lxc`` command must be available on the host
889 machine.
890
891 * ``ansible_python_interpreter``
892 * ``ansible_host``: Name of LXC container (default: inventory hostname).
893 * ``mitogen_lxc_path``: path to ``lxc`` command if not available on the system
894 path.
895 * ``mitogen_mask_remote_name``: if :data:`True`, mask the identity of the
896 Ansible controller process on remote machines. To simplify diagnostics,
897 Mitogen produces remote processes named like
898 `"mitogen:[email protected]:1234"`, however this may be a privacy issue in
899 some circumstances.
900
901
902 .. _machinectl:
903
904 Machinectl
905 ~~~~~~~~~~
906
907 Like the `machinectl third party plugin
908 <https://github.com/BaxterStockman/ansible-connection-machinectl>`_ except
909 connection delegation is supported. This is a light wrapper around the
910 :ref:`setns <setns>` method.
911
912 * ``ansible_host``: Name of Docker container (default: inventory hostname).
913 * ``ansible_user``: Name of user within the container to execute as.
914 * ``mitogen_machinectl_path``: path to ``machinectl`` command if not available
915 as ``/bin/machinectl``.
916 * ``mitogen_mask_remote_name``: if :data:`True`, mask the identity of the
917 Ansible controller process on remote machines. To simplify diagnostics,
918 Mitogen produces remote processes named like
919 `"mitogen:[email protected]:1234"`, however this may be a privacy issue in
920 some circumstances.
921
922
923 .. _setns:
924
925 Setns
926 ~~~~~
927
928 The ``setns`` method connects to Linux containers via `setns(2)
929 <https://linux.die.net/man/2/setns>`_. Unlike :ref:`method-docker`,
930 :ref:`method-lxc`, and :ref:`method-lxd` the namespace transition is handled
931 internally, ensuring optimal throughput to the child. This is necessary for
932 :ref:`machinectl` where only PTY channels are supported.
933
934 A utility program must be installed to discover the PID of the container's root
935 process.
936
937 * ``mitogen_kind``: one of ``docker``, ``lxc``, ``lxd`` or ``machinectl``.
938 * ``ansible_host``: Name of container as it is known to the corresponding tool
939 (default: inventory hostname).
940 * ``ansible_user``: Name of user within the container to execute as.
941 * ``mitogen_docker_path``: path to Docker if not available on the system path.
942 * ``mitogen_lxc_path``: path to LXD's ``lxc`` command if not available as
943 ``lxc-info``.
944 * ``mitogen_lxc_info_path``: path to LXC classic's ``lxc-info`` command if not
945 available as ``lxc-info``.
946 * ``mitogen_machinectl_path``: path to ``machinectl`` command if not available
947 as ``/bin/machinectl``.
948
949
950 .. _su:
951
952 Su
953 ~~
954
955 Su can be used as a connection method that supports connection delegation, or
956 as a become method.
957
958 When used as a become method:
959
960 * ``ansible_python_interpreter``
961 * ``ansible_su_exe``, ``ansible_become_exe``
962 * ``ansible_su_user``, ``ansible_become_user`` (default: ``root``)
963 * ``ansible_su_pass``, ``ansible_become_pass`` (default: assume passwordless)
964 * ``su_flags``, ``become_flags``
965 * ``mitogen_mask_remote_name``: if :data:`True`, mask the identity of the
966 Ansible controller process on remote machines. To simplify diagnostics,
967 Mitogen produces remote processes named like
968 `"mitogen:[email protected]:1234"`, however this may be a privacy issue in
969 some circumstances.
970 * ansible.cfg: ``timeout``
971
972 When used as the ``mitogen_su`` connection method:
973
974 * The inventory hostname has no special meaning.
975 * ``ansible_user``: username to su as.
976 * ``ansible_password``: password to su as.
977 * ``ansible_python_interpreter``
978
979
980 .. _sudo:
981
982 Sudo
983 ~~~~
984
985 Sudo can be used as a connection method that supports connection delegation, or
986 as a become method.
987
988 When used as a become method:
989
990 * ``ansible_python_interpreter``
991 * ``ansible_sudo_exe``, ``ansible_become_exe``
992 * ``ansible_sudo_user``, ``ansible_become_user`` (default: ``root``)
993 * ``ansible_sudo_pass``, ``ansible_become_pass`` (default: assume passwordless)
994 * ``sudo_flags``, ``become_flags``
995 * ``mitogen_mask_remote_name``: if :data:`True`, mask the identity of the
996 Ansible controller process on remote machines. To simplify diagnostics,
997 Mitogen produces remote processes named like
998 `"mitogen:[email protected]:1234"`, however this may be a privacy issue in
999 some circumstances.
1000 * ansible.cfg: ``timeout``
1001
1002 When used as the ``mitogen_sudo`` connection method:
1003
1004 * The inventory hostname has no special meaning.
1005 * ``ansible_user``: username to sudo as.
1006 * ``ansible_password``: password to sudo as.
1007 * ``sudo_flags``, ``become_flags``
1008 * ``ansible_python_interpreter``
1009
1010
1011 SSH
1012 ~~~
1013
1014 Like the :ans:conn:`ssh` except connection delegation is supported.
1015
1016 * ``ansible_ssh_timeout``
1017 * ``ansible_host``, ``ansible_ssh_host``
1018 * ``ansible_user``, ``ansible_ssh_user``
1019 * ``ansible_port``, ``ssh_port``
1020 * ``ansible_ssh_executable``, ``ssh_executable``
1021 * ``ansible_ssh_private_key_file``
1022 * ``ansible_ssh_pass``, ``ansible_ssh_password``, ``ansible_password``
1023 (default: assume passwordless)
1024 * ``ansible_ssh_host_key_checking``, ``ansible_host_key_checking`` (default:
1025 :data:`True`)
1026 * ``ssh_args``, ``ssh_common_args``, ``ssh_extra_args``
1027 * ``mitogen_mask_remote_name``: if :data:`True`, mask the identity of the
1028 Ansible controller process on remote machines. To simplify diagnostics,
1029 Mitogen produces remote processes named like
1030 `"mitogen:[email protected]:1234"`, however this may be a privacy issue in
1031 some circumstances.
1032 * ``mitogen_ssh_debug_level``: integer between `0..3` indicating the SSH client
1033 debug level. Ansible must also be run with '-vvv' to view the output.
1034 * ``mitogen_ssh_compression``: :data:`True` to enable SSH compression,
1035 otherwise :data:`False`. This will change to off by default in a future
1036 release. If you are targetting many hosts on a fast network, please consider
1037 disabling SSH compression.
1038 * ``mitogen_ssh_keepalive_count``: integer count of server keepalive messages to
1039 which no reply is received before considering the SSH server dead. Defaults
1040 to 10.
1041 * ``mitogen_ssh_keepalive_interval``: integer seconds delay between keepalive
1042 messages. Defaults to 30.
1043
1044
1045 Debugging
1046 ---------
1047
1048 Diagnostics and :py:mod:`logging` package output on targets are usually
1049 discarded. Mitogen can capture and forward them to the controller.
1050
1051 Mitogen >= 0.3.45 uses `Ansible verbosity`_ (e.g. ``ansible -vvv ...``) and an
1052 environment variable ``MITOGEN_LOG_LEVEL`` (e.g. ``MITOGEN_LOG_LEVEL=debug``)
1053 to configure this. Both are required for more verbose Mitogen logging output.
1054
1055 Ansible verbosity is mapped to Mitogen logging as follows
1056
1057 ================= ===========================================================
1058 Ansible verbosity Mitogen output
1059 ================= ===========================================================
1060 0-2 ``ERROR``, ``WARNING``
1061 3 ``ERROR``, ``WARNING``, ``INFO``, ``DEBUG`` except IO
1062 4-5 ``ERROR``, ``WARNING``, ``INFO``, ``DEBUG`` including IO
1063 ================= ===========================================================
1064
1065 ``MITOGEN_LOG_LEVEL`` accepts a Python `logging level`_ name, or ``IO`` for
1066 extra ``DEBUG`` output of Mitogen's network traffic.
1067
1068 Mitogen <= 0.3.44 only uses Ansible verbosity to configure the output.
1069 ``MITOGEN_LOG_LEVEL`` is ignored.
1070
1071 While uncaptured standard IO and the logging package on targets is forwarded,
1072 it is not possible to receive IO activity logs, as the forwarding process would
1073 would itself generate additional IO.
1074
1075 To receive a complete trace of every process on every machine, file-based
1076 logging is necessary. File-based logging can be enabled by setting
1077 ``MITOGEN_ROUTER_DEBUG=1`` in your environment. When file-based logging is
1078 enabled, one file per context will be created on the local machine and every
1079 target machine, as ``/tmp/mitogen.<pid>.log``.
1080
1081 .. _ansible verbosity: https://docs.ansible.com/projects/ansible/latest/reference_appendices/config.html#default-verbosity
1082 .. _logging level: https://docs.python.org/3/library/logging.html#logging-levels
1083
1084
1085 Common Problems
1086 ~~~~~~~~~~~~~~~
1087
1088 The most common bug reports fall into the following categories, so it is worth
1089 checking whether you can categorize a problem using the tools provided before
1090 reporting it:
1091
1092 **Missed/Incorrect Configuration Variables**
1093 In some cases Ansible may support a configuration variable that Mitogen
1094 does not yet support, or Mitogen supports, but the support is broken. For
1095 example, Mitogen may pick the wrong username or SSH parameters.
1096
1097 To detect this, use the special ``mitogen_get_stack`` action described
1098 below to verify the settings Mitogen has chosen for the connection make
1099 sense.
1100
1101 **Process Environment Differences**
1102 Mitogen's process model differs significantly to Ansible's in many places.
1103 In the past, bugs have been reported because Ansible plug-ins modify an
1104 environment variable after Mitogen processes are started.
1105
1106 If your task's failure may relate to the process environment in some way,
1107 for example, ``SSH_AUTH_SOCK``, ``LC_ALL`` or ``PATH``, then an environment
1108 difference may explain it. Environment differences are always considered
1109 bugs in the extension, and are very easy to repair, so even if you find a
1110 workaround, please report them to avoid someone else encountering the same
1111 problem.
1112
1113 **Variable Expansion Differences**
1114 To avoid many classes of bugs, Mitogen avoids shell wherever possible.
1115 Ansible however is traditionally built on shell, and it is often difficult
1116 to tell just how many times a configuration parameter will pass through
1117 shell expansion and quoting, and in what context before it is used.
1118
1119 Due to this, in some circumstances Mitogen may parse some expanded
1120 variables differently, for example, in the wrong user account. Careful
1121 review of ``-vvv`` and ``mitogen_ssh_debug_level`` logs can reveal this.
1122 For example in the past, Mitogen used a different method of expanding
1123 ``~/.ssh/id_rsa``, causing authentication to fail when ``ansible-playbook``
1124 was run via ``sudo -E``.
1125
1126 **External Tool Integration Differences**
1127 Mitogen reimplements any aspect of Ansible that involves integrating with
1128 SSH, sudo, Docker, or related tools. For this reason, sometimes its support
1129 for those tools differs or is less mature than in Ansible.
1130
1131 In the past Mitogen has had bug reports due to failing to recognize a
1132 particular variation of a login or password prompt on an exotic or
1133 non-English operating system, or confusing a login banner for a password
1134 prompt. Careful review of ``-vvv`` logs help identify these cases, as
1135 Mitogen logs all strings it receives during connection, and how it
1136 interprets them.
1137
1138
1139 .. _mitogen-get-stack:
1140
1141 The `mitogen_get_stack` Action
1142 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1143
1144 When a Mitogen strategy is loaded, a special ``mitogen_get_stack`` action is
1145 available that returns a concise description of the connection configuration as
1146 extracted from Ansible and passed to the core library. Using it, you can learn
1147 whether a problem lies in the Ansible extension or deeper in library code.
1148
1149 The action may be used in a playbook as ``mitogen_get_stack:`` just like a
1150 regular module, or directly from the command-line::
1151
1152 $ ANSIBLE_STRATEGY=mitogen_linear ansible -m mitogen_get_stack -b -k k3
1153 SSH password:
1154 k3 | SUCCESS => {
1155 "changed": true,
1156 "result": [
1157 {
1158 "kwargs": {
1159 "check_host_keys": "enforce",
1160 "connect_timeout": 10,
1161 "hostname": "k3",
1162 "identities_only": false,
1163 "identity_file": null,
1164 "password": "mysecretpassword",
1165 "port": null,
1166 "python_path": null,
1167 "ssh_args": [
1168 "-C",
1169 "-o",
1170 "ControlMaster=auto",
1171 "-o",
1172 "ControlPersist=60s"
1173 ],
1174 "ssh_debug_level": null,
1175 "ssh_path": "ssh",
1176 "username": null
1177 },
1178 "method": "ssh"
1179 },
1180 {
1181 "enable_lru": true,
1182 "kwargs": {
1183 "connect_timeout": 10,
1184 "password": null,
1185 "python_path": null,
1186 "sudo_args": [
1187 "-H",
1188 "-S",
1189 "-n"
1190 ],
1191 "sudo_path": null,
1192 "username": "root"
1193 },
1194 "method": "sudo"
1195 }
1196 ]
1197 }
1198
1199 Each object in the list represents a single 'hop' in the connection, from
1200 nearest to furthest. Unlike in Ansible, the core library treats ``become``
1201 steps and SSH steps identically, so they are represented distinctly in the
1202 output.
1203
1204 The presence of ``null`` means no explicit value was extracted from Ansible,
1205 and either the Mitogen library or SSH will choose a value for the parameter. In
1206 the example above, Mitogen will choose ``/usr/bin/python`` for ``python_path``,
1207 and SSH will choose ``22`` for ``port``, or whatever ``Port`` it parses from
1208 ``~/.ssh/config``. Note the presence of ``null`` may indicate the extension
1209 failed to extract the correct value.
1210
1211 When using ``mitogen_get_stack`` to diagnose a problem, pay special attention
1212 to ensuring the invocation exactly matches the problematic task. For example,
1213 if the failing task has ``delegate_to:`` or ``become:`` enabled, the
1214 ``mitogen_get_stack`` invocation must include those statements in order for the
1215 output to be accurate.
1216
1217 If a playbook cannot start at all, you may need to temporarily use
1218 ``gather_facts: no`` to allow the first task to proceed. This action does not
1219 create connections, so if it is the first task, it is still possible to review
1220 its output.
1221
1222
1223 The `mitogen_ssh_debug_level` Variable
1224 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1225
1226 Mitogen has support for capturing SSH diagnostic logs, and integrating them
1227 into the regular debug log output produced when ``-vvv`` is active. This
1228 provides a single audit trail of every component active during SSH
1229 authentication.
1230
1231 Particularly for authentication failures, setting this variable to 3, in
1232 combination with ``-vvv``, allows review of every parameter passed to SSH, and
1233 review of every action SSH attempted during authentication.
1234
1235 For example, this method can be used to ascertain whether SSH attempted agent
1236 authentication, or what private key files it was able to access and which it tried.
1237
1238
1239 Post-authentication Bootstrap Failure
1240 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1241
1242 If logging indicates Mitogen was able to authenticate, but some error occurred
1243 after authentication preventing the Python bootstrap from completing, it can be
1244 immensely useful to temporarily replace ``ansible_python_interpreter`` with a
1245 wrapper that runs Python under ``strace``::
1246
1247 $ ssh badbox
1248
1249 badbox$ cat > strace-python.sh
1250 #!/bin/sh
1251 strace -o /tmp/strace-python.$$ -ff -s 100 python "$@"
1252 ^D
1253
1254 badbox$ chmod +x strace-python.sh
1255 badbox$ logout
1256
1257 $ ansible-playbook site.yml \
1258 -e ansible_python_interpreter=./strace-python.sh \
1259 -l badbox
1260
1261 This will produce a potentially large number of log files under ``/tmp/``. The
1262 lowest-numbered traced PID is generally the main Python interpreter. The most
1263 intricate bootstrap steps happen there, any error should be visible near the
1264 end of the trace.
1265
1266 It is also possible the first stage bootstrap failed. That is usually the next
1267 lowest-numbered PID and tends to be the smallest file. Even if you can't
1268 ascertain the problem with your configuration from these logs, including them
1269 in a bug report can save days of detective effort.
1270
1271
1272 .. _diagnosing-hangs:
1273
1274 Diagnosing Hangs
1275 ~~~~~~~~~~~~~~~~
1276
1277 If you encounter a hang, the ``MITOGEN_DUMP_THREAD_STACKS=<secs>`` environment
1278 variable arranges for each process on each machine to dump each thread stack
1279 into the logging framework every `secs` seconds, which is visible when running
1280 with ``-vvv``.
1281
1282 However, certain controller hangs may render ``MITOGEN_DUMP_THREAD_STACKS``
1283 ineffective, or occur too infrequently for interactive reproduction. In these
1284 cases :py:mod:`faulthandler` may be used with Python >= 3.3:
1285
1286 1. Once the hang occurs, observe the process tree using ``pstree`` or ``ps
1287 --forest``.
1288 2. The most likely process to be hung is the connection multiplexer, which can
1289 easily be identified as the parent of all SSH client processes.
1290 3. Send ``kill -SEGV <pid>`` to the multiplexer PID, causing it to print all
1291 thread stacks.
1292 4. `File a bug <https://github.com/mitogen-hq/mitogen/issues/new/>`_
1293 including a copy of the stacks and a description of the last task executing
1294 before the hang
1295
1296 It is possible the hang occurred in a process on a target. If ``strace`` is
1297 available, look for the host name not listed in Ansible output as reporting a
1298 result for the most recent task, log into it, and use ``strace -ff -p <pid>``
1299 on each process whose name begins with ``mitogen:``::
1300
1301 $ strace -ff -p 29858
1302 strace: Process 29858 attached with 3 threads
1303 [pid 29864] futex(0x55ea9be52f60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff <unfinished ...>
1304 [pid 29860] restart_syscall(<... resuming interrupted poll ...> <unfinished ...>
1305 [pid 29858] futex(0x55ea9be52f60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff
1306 ^C
1307
1308 $
1309
1310 This shows one thread waiting on IO (``poll``) and two more waiting on the same
1311 lock. It is taken from a real example of a deadlock due to a forking bug.
1312 Please include any such information for all processes that you are able to
1313 collect in any bug report.
1314
1315
1316 Getting Help
1317 ~~~~~~~~~~~~
1318 Some users and developers hang out on the
1319 `#mitogen <https://webchat.freenode.net/?channels=mitogen>`_ channel on the
1320 FreeNode IRC network.
1321
1322
1323 Sample Profiles
1324 ---------------
1325
1326 The summaries below may be reproduced using data and scripts maintained in the
1327 `pcaps branch <https://github.com/mitogen-hq/mitogen/tree/pcaps/>`_. Traces were
1328 recorded using Ansible 2.5.14.
1329
1330
1331 Trivial Loop: Local Host
1332 ~~~~~~~~~~~~~~~~~~~~~~~~
1333
1334 This demonstrates Mitogen vs. SSH pipelining to the local machine running
1335 `bench/loop-100-items.yml
1336 <https://github.com/mitogen-hq/mitogen/blob/master/tests/ansible/bench/loop-100-items.yml>`_,
1337 executing a simple command 100 times. Most Ansible controller overhead is
1338 isolated, characterizing just module executor and connection layer performance.
1339 Mitogen requires **63x less bandwidth and 5.9x less time**.
1340
1341 .. image:: images/ansible/pcaps/loop-100-items-local.svg
1342
1343 Unlike in SSH pipelining where payloads are sent as a single compressed block,
1344 by default Mitogen enables SSH compression for its uncompressed RPC data. In
1345 many-host scenarios it may be desirable to disable compression. This has
1346 negligible impact on footprint, since program code is separately compressed and
1347 sent only once. Compression also benefits SSH pipelining, but the presence of
1348 large precompressed per-task payloads may present a more significant CPU burden
1349 during many-host runs.
1350
1351 .. image:: images/ansible/pcaps/loop-100-items-local-detail.svg
1352
1353 In a detailed trace, improved interaction with the host machine is visible. In
1354 this playbook because no forks were required to start SSH clients from the
1355 worker process executing the loop, the worker's memory was never marked
1356 read-only, thus avoiding a major hidden performance problem - the page fault
1357 rate is more than halved.
1358
1359
1360 File Transfer: UK to France
1361 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
1362
1363 `This playbook
1364 <https://github.com/mitogen-hq/mitogen/blob/master/tests/ansible/regression/issue_140__thread_pileup.yml>`_
1365 was used to compare file transfer performance over a ~26 ms link. It uses the
1366 ``with_filetree`` loop syntax to copy a directory of 1,000 0-byte files to the
1367 target.
1368
1369 .. raw:: html
1370
1371 <style>
1372 .nojunk td,
1373 .nojunk th { padding: 4px; font-size: 90%; text-align: right !important; }
1374
1375 table.docutils col {
1376 width: auto !important;
1377 }
1378 </style>
1379
1380 .. csv-table::
1381 :header: , Secs, CPU Secs, Sent, Received, Roundtrips
1382 :class: nojunk
1383 :align: right
1384
1385 Mitogen, 98.54, 43.04, "815 KiB", "447 KiB", 3.79
1386 SSH Pipelining, "1,483.54", 329.37, "99,539 KiB", "6,870 KiB", 57.01
1387
1388 *Roundtrips* is the approximate number of network roundtrips required to
1389 describe the runtime that was consumed. Due to Mitogen's built-in file transfer
1390 support, continuous reinitialization of an external `scp`/`sftp` client is
1391 avoided, permitting large ``with_filetree`` copies to become practical without
1392 any special casing within the playbook or the Ansible implementation.
1393
1394
1395 DebOps: UK to India
1396 ~~~~~~~~~~~~~~~~~~~
1397
1398 This is an all-green run of 246 tasks from the `DebOps
1399 <https://docs.debops.org/en/master/>`_ 0.7.2 `common.yml
1400 <https://github.com/debops/debops-playbooks/blob/master/playbooks/common.yml>`_
1401 playbook over a ~370 ms link between the UK and India. The playbook touches a
1402 wide variety of modules, many featuring unavoidable waits for slow computation
1403 on the target.
1404
1405 More tasks of a wider variety are featured than previously, placing strain on
1406 Mitogen's module loading and in-memory caching. By running over a long-distance
1407 connection, it highlights behaviour of the connection layer in the presence of
1408 high latency.
1409
1410 Mitogen requires **14.5x less bandwidth and 4x less time**.
1411
1412 .. image:: images/ansible/pcaps/debops-uk-india.svg
1413
1414
1415 Django App: UK to India
1416 ~~~~~~~~~~~~~~~~~~~~~~~
1417
1418 This short playbook features only 23 steps executed over the same ~370 ms link
1419 as previously, with many steps running unavoidably expensive tasks like
1420 building C++ code, and compiling static web site assets.
1421
1422 Despite the small margin for optimization, Mitogen still manages **6.2x less
1423 bandwidth and 1.8x less time**.
1424
1425 .. image:: images/ansible/pcaps/costapp-uk-india.svg