| 1 |
|
| 2 |
.. _ansible_detailed: |
| 3 |
|
| 4 |
Mitogen for Ansible |
| 5 |
=================== |
| 6 |
|
| 7 |
.. image:: images/mitogen.svg |
| 8 |
:class: mitogen-right-180 mitogen-logo-wrap |
| 9 |
|
| 10 |
**Mitogen for Ansible** is a completely redesigned UNIX connection layer and |
| 11 |
module runtime for `Ansible`_. Requiring minimal configuration changes, it |
| 12 |
updates Ansible's slow and wasteful shell-centric implementation with |
| 13 |
pure-Python equivalents, invoked via highly efficient remote procedure calls to |
| 14 |
persistent interpreters tunnelled over SSH. No changes are required to target |
| 15 |
hosts. |
| 16 |
|
| 17 |
The extension is considered stable and real-world use is encouraged. |
| 18 |
|
| 19 |
.. _Ansible: https://www.ansible.com/ |
| 20 |
|
| 21 |
.. _Bug reports: https://github.com/mitogen-hq/mitogen/issues/new/choose |
| 22 |
|
| 23 |
|
| 24 |
Overview |
| 25 |
-------- |
| 26 |
|
| 27 |
**Expect a 1.25x - 7x speedup** and a **CPU usage reduction of at least 2x**, |
| 28 |
depending on network conditions, modules executed, and time already spent by |
| 29 |
targets on useful work. Mitogen cannot improve a module once it is executing, |
| 30 |
it can only ensure the module executes as quickly as possible. |
| 31 |
|
| 32 |
* **One connection is used per target**, in addition to one sudo invocation per |
| 33 |
user account. This is much better than SSH multiplexing combined with |
| 34 |
pipelining, as significant state can be maintained in RAM between steps, and |
| 35 |
system logs aren't spammed with repeat authentication events. |
| 36 |
|
| 37 |
* **A single network roundtrip is used** to execute a step whose code already |
| 38 |
exists in RAM on the target. Eliminating multiplexed SSH channel creation |
| 39 |
saves 4 ms runtime per 1 ms of network latency for every playbook step. |
| 40 |
|
| 41 |
* **Processes are aggressively reused**, avoiding the cost of invoking Python |
| 42 |
and recompiling imports, saving 300-800 ms for every playbook step. |
| 43 |
|
| 44 |
* Code is ephemerally cached in RAM, **reducing bandwidth usage by an order |
| 45 |
of magnitude** compared to SSH pipelining, with around 5x fewer frames |
| 46 |
traversing the network in a typical run. |
| 47 |
|
| 48 |
* **Fewer writes to the target filesystem occur**. In typical configurations, |
| 49 |
Ansible repeatedly rewrites and extracts ZIP files to multiple temporary |
| 50 |
directories on the target. Security issues relating to temporary files in |
| 51 |
cross-account scenarios are entirely avoided. |
| 52 |
|
| 53 |
The effect is most potent on playbooks that execute many **short-lived |
| 54 |
actions**, where Ansible's overhead dominates the cost of the operation, for |
| 55 |
example when executing large ``with_items`` loops to run simple commands or |
| 56 |
write files. |
| 57 |
|
| 58 |
|
| 59 |
Installation |
| 60 |
------------ |
| 61 |
|
| 62 |
1. Review :ref:`noteworthy_differences`. |
| 63 |
2. Download and extract |mitogen_url|. |
| 64 |
3. Modify ``ansible.cfg``: |
| 65 |
|
| 66 |
.. parsed-literal:: |
| 67 |
|
| 68 |
[defaults] |
| 69 |
strategy_plugins = /path/to/mitogen-|mitogen_version|/ansible_mitogen/plugins/strategy |
| 70 |
strategy = mitogen_linear |
| 71 |
|
| 72 |
The ``strategy`` key is optional. If omitted, the |
| 73 |
``ANSIBLE_STRATEGY=mitogen_linear`` environment variable can be set on a |
| 74 |
per-run basis. Like ``mitogen_linear``, the ``mitogen_free`` and |
| 75 |
``mitogen_host_pinned`` strategies exists to mimic the ``free`` and |
| 76 |
``host_pinned`` strategies. |
| 77 |
|
| 78 |
|
| 79 |
Demo |
| 80 |
~~~~ |
| 81 |
|
| 82 |
This demonstrates Ansible running a subset of the Mitogen integration tests |
| 83 |
concurrent to an equivalent run using the extension. |
| 84 |
|
| 85 |
.. raw:: html |
| 86 |
|
| 87 |
<iframe src="https://player.vimeo.com/video/283272293?title=0&byline=0&portrait=0" width="720" height="439" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe> |
| 88 |
|
| 89 |
|
| 90 |
Testimonials |
| 91 |
~~~~~~~~~~~~ |
| 92 |
|
| 93 |
* "With mitogen **my playbook runtime went from 45 minutes to just under 3 |
| 94 |
minutes**. Awesome work!" |
| 95 |
|
| 96 |
* "The runtime was reduced from **1.5 hours on 4 servers to just under 3 |
| 97 |
minutes**. Thanks!" |
| 98 |
|
| 99 |
* "Oh, performance improvement using Mitogen is *huge*. As mentioned before, |
| 100 |
running with Mitogen enables takes 7m36 (give or take a few seconds). Without |
| 101 |
Mitogen, the same run takes 19m49! **I'm not even deploying without Mitogen |
| 102 |
anymore** :)" |
| 103 |
|
| 104 |
* "**Works like a charm**, thank you for your quick response" |
| 105 |
|
| 106 |
* "I tried it out. **He is not kidding about the speed increase**." |
| 107 |
|
| 108 |
* "I don't know what kind of dark magic @dmw_83 has done, but his Mitogen |
| 109 |
strategy took Clojars' Ansible runs from **14 minutes to 2 minutes**. I still |
| 110 |
can't quite believe it." |
| 111 |
|
| 112 |
* "Enabling the mitogen plugin in ansible feels like switching from floppy to SSD" |
| 113 |
|
| 114 |
|
| 115 |
.. _noteworthy_differences: |
| 116 |
|
| 117 |
Noteworthy Differences |
| 118 |
---------------------- |
| 119 |
|
| 120 |
* Mitogen 0.2.x supports Ansible 2.3-2.9; with Python 2.6, 2.7, or 3.6. |
| 121 |
Mitogen 0.3.1+ supports |
| 122 |
|
| 123 |
+-----------------+-----------------+ |
| 124 |
| Ansible version | Python versions | |
| 125 |
+=================+=================+ |
| 126 |
| 2.10 | | |
| 127 |
+-----------------+ | |
| 128 |
| 3 | 2.7, 3.6 - 3.11 | |
| 129 |
+-----------------+ | |
| 130 |
| 4 | | |
| 131 |
+-----------------+-----------------+ |
| 132 |
| 5 | 3.8 - 3.11 | |
| 133 |
+-----------------+-----------------+ |
| 134 |
| 6 | | |
| 135 |
+-----------------+ 3.8 - 3.13 | |
| 136 |
| 7 | | |
| 137 |
+-----------------+-----------------+ |
| 138 |
| 8 | 3.9 - 3.13 | |
| 139 |
+-----------------+-----------------+ |
| 140 |
| 9 | | |
| 141 |
+-----------------+ 3.10 - 3.14 | |
| 142 |
| 10 | | |
| 143 |
+-----------------+-----------------+ |
| 144 |
| 11 | | |
| 145 |
+-----------------+ 3.11 - 3.14 | |
| 146 |
| 12 | | |
| 147 |
+-----------------+-----------------+ |
| 148 |
| 13 | | |
| 149 |
+-----------------+ 3.12 - 3.14 | |
| 150 |
| 14 | | |
| 151 |
+-----------------+-----------------+ |
| 152 |
|
| 153 |
Verify your installation is running one of these versions by checking |
| 154 |
``ansible --version`` output. |
| 155 |
|
| 156 |
* The ``raw`` action executes as a regular Mitogen connection, which requires |
| 157 |
Python on the target, precluding its use for installing Python. This will be |
| 158 |
addressed in a future release. For now, simply mix Mitogen and vanilla |
| 159 |
Ansible strategies: |
| 160 |
|
| 161 |
.. code-block:: yaml |
| 162 |
|
| 163 |
- hosts: web-servers |
| 164 |
strategy: linear |
| 165 |
tasks: |
| 166 |
- name: Install Python if necessary. |
| 167 |
raw: test -e /usr/bin/python || apt install -y python-minimal |
| 168 |
|
| 169 |
- hosts: web-servers |
| 170 |
strategy: mitogen_linear |
| 171 |
roles: |
| 172 |
- nginx |
| 173 |
- initech_app |
| 174 |
- y2k_fix |
| 175 |
|
| 176 |
* Ansible `become plugins |
| 177 |
<https://docs.ansible.com/ansible/latest/plugins/become.html>`_ are not yet |
| 178 |
supported. |
| 179 |
|
| 180 |
* The ``doas``, ``su`` and ``sudo`` become methods are available. File bugs to |
| 181 |
register interest in more. |
| 182 |
|
| 183 |
* The ``sudo`` comands executed differ slightly compared to Ansible. In some |
| 184 |
cases where the target has a ``sudo`` configuration that restricts the exact |
| 185 |
commands allowed to run, it may be necessary to add a ``sudoers`` rule like: |
| 186 |
|
| 187 |
:: |
| 188 |
|
| 189 |
your_ssh_username = (ALL) NOPASSWD:/usr/bin/python -c* |
| 190 |
|
| 191 |
* The :ans:conn:`~buildah`, :ans:conn:`~docker`, :ans:conn:`~jail`, |
| 192 |
:ans:conn:`~kubectl`, :ans:conn:`~local`, :ans:conn:`~lxd`, |
| 193 |
:ans:conn:`~podman`, & :ans:conn:`~ssh` connection types are supported; also |
| 194 |
Mitogen-specific :ref:`mitogen_doas <doas>`, :ref:`machinectl <machinectl>`, |
| 195 |
:ref:`mitogen_su <su>`, :ref:`mitogen_sudo <sudo>`, and :ref:`setns <setns>` |
| 196 |
types. File bugs to register interest in others. |
| 197 |
|
| 198 |
* Actions are single-threaded for each `(host, user account)` combination, |
| 199 |
including actions that execute on the local machine. Playbooks may experience |
| 200 |
slowdown compared to vanilla Ansible if they employ long-running |
| 201 |
``local_action`` or ``delegate_to`` tasks delegating many target hosts to a |
| 202 |
single machine and user account. |
| 203 |
|
| 204 |
Ansible usually permits up to ``forks`` simultaneous local actions. Any |
| 205 |
long-running local actions that execute for every target will experience |
| 206 |
artificial serialization, causing slowdown equivalent to `task_duration * |
| 207 |
num_targets`. This will be addressed soon. |
| 208 |
|
| 209 |
* The Ansible 2.7 :ans:mod:`reboot` may require a ``pre_reboot_delay`` on |
| 210 |
systemd hosts, as insufficient time exists for the reboot command's exit |
| 211 |
status to be reported before necessary processes are torn down. |
| 212 |
|
| 213 |
* On OS X when a SSH password is specified and the default connection type of |
| 214 |
:ans:conn:`~smart` is used, Ansible may select the :ans:conn:`paramiko_ssh` |
| 215 |
rather than Mitogen. If you specify a password on OS X, ensure ``connection: |
| 216 |
ssh`` appears in your playbook, ``ansible.cfg``, or as ``-c ssh`` on the |
| 217 |
command-line. |
| 218 |
|
| 219 |
* Ansible permits up to ``forks`` connections to be setup in parallel, whereas |
| 220 |
in Mitogen this is handled by a fixed-size thread pool. Up to 32 connections |
| 221 |
may be established in parallel by default, this can be modified by setting |
| 222 |
the ``MITOGEN_POOL_SIZE`` environment variable. |
| 223 |
|
| 224 |
* Performance does not scale cleanly with target count. This will improve over |
| 225 |
time. |
| 226 |
|
| 227 |
* Performance on Python 3 is significantly worse than on Python 2. While this |
| 228 |
has not yet been investigated, at least some of the regression appears to be |
| 229 |
part of the core library, and should therefore be straightforward to fix as |
| 230 |
part of 0.2.x. |
| 231 |
|
| 232 |
* Connection and become timeouts are applied differently. Mitogen may consider |
| 233 |
a connection to have timed out, when Ansible would have waited longer or |
| 234 |
indefinately. For example if SSH authentication completes within the |
| 235 |
timeout, but execution of login scripts exceeds it - then Mitogen will |
| 236 |
consider the task to have timed out and that host to have failed. |
| 237 |
|
| 238 |
.. |
| 239 |
tests/ansible/integration/ssh/timeouts.yml covers (some of) this behaviour. |
| 240 |
|
| 241 |
.. |
| 242 |
* SSH and ``become`` are treated distinctly when applying timeouts, and |
| 243 |
timeouts apply up to the point when the new interpreter is ready to accept |
| 244 |
messages. Ansible has two timeouts: ``ConnectTimeout`` for SSH, applying up |
| 245 |
to when authentication completes, and a separate parallel timeout up to |
| 246 |
when ``become`` authentication completes. |
| 247 |
For busy targets, Ansible may successfully execute a module where Mitogen |
| 248 |
would fail without increasing the timeout. For sick targets, Ansible may |
| 249 |
hang indefinitely after authentication without executing a command, for |
| 250 |
example due to a stuck filesystem IO appearing in ``$HOME/.profile``. |
| 251 |
|
| 252 |
.. |
| 253 |
* "Module Replacer" style modules are not supported. These rarely appear in |
| 254 |
practice, and light web searches failed to reveal many examples of them. |
| 255 |
|
| 256 |
* The ``ansible_python_interpreter`` variable is parsed using a restrictive |
| 257 |
:mod:`shell-like <shlex>` syntax, permitting values such as ``/usr/bin/env |
| 258 |
FOO=bar python`` or ``source /opt/rh/rh-python36/enable && python``. |
| 259 |
Jinja2 templating is also supported for complex task-level |
| 260 |
interpreter settings. Ansible documents `ansible_python_interpreter |
| 261 |
<https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html#ansible-python-interpreter>`_ |
| 262 |
as an absolute path and releases since June 2024 (e.g. Ansible 10.1) |
| 263 |
reflect this. Older Ansible releases passed it to the shell unquoted. |
| 264 |
|
| 265 |
.. |
| 266 |
* Configurations will break that rely on the `hashbang argument splitting |
| 267 |
behaviour <https://github.com/ansible/ansible/issues/15635>`_ of the |
| 268 |
``ansible_python_interpreter`` setting, contrary to the Ansible |
| 269 |
documentation. This will be addressed in a future 0.2 release. |
| 270 |
|
| 271 |
|
| 272 |
New Features & Notes |
| 273 |
-------------------- |
| 274 |
|
| 275 |
|
| 276 |
Connection Delegation |
| 277 |
~~~~~~~~~~~~~~~~~~~~~ |
| 278 |
|
| 279 |
.. image:: images/jumpbox.svg |
| 280 |
:class: mitogen-right-275 |
| 281 |
|
| 282 |
Included is a preview of **Connection Delegation**, a Mitogen-specific |
| 283 |
implementation of `stackable connection plug-ins`_. This enables connections |
| 284 |
via a bastion, or container connections delegated via their host machine, where |
| 285 |
reaching the host may entail further delegation. |
| 286 |
|
| 287 |
.. _Stackable connection plug-ins: https://github.com/ansible/proposals/issues/25 |
| 288 |
|
| 289 |
Unlike with SSH forwarding Ansible has complete visibility of the final |
| 290 |
topology, declarative configuration via static/dynamic inventory is possible, |
| 291 |
and data can be cached and re-served, and code executed on every intermediary. |
| 292 |
|
| 293 |
For example when targeting Docker containers on a remote machine, each module |
| 294 |
need only be uploaded once for the first task and container that requires it, |
| 295 |
then cached and served from the SSH account for every future task in any |
| 296 |
container. |
| 297 |
|
| 298 |
.. raw:: html |
| 299 |
|
| 300 |
<div style="clear: both;"></div> |
| 301 |
|
| 302 |
|
| 303 |
.. caution:: |
| 304 |
|
| 305 |
Connection delegation is a work in progress, bug reports are welcome. |
| 306 |
|
| 307 |
* Delegated connection setup is single-threaded; only one connection can be |
| 308 |
constructed in parallel per intermediary. |
| 309 |
|
| 310 |
* Inferring the configuration of intermediaries may be buggy, manifesting |
| 311 |
as duplicate connections between hops, due to not perfectly replicating |
| 312 |
the configuration Ansible would normally use for the intermediary. |
| 313 |
|
| 314 |
* Intermediary machines cannot use login and become passwords that were |
| 315 |
supplied to Ansible interactively. If an intermediary requires a |
| 316 |
password, it must be supplied via ``ansible_ssh_pass``, |
| 317 |
``ansible_ssh_password``, ``ansible_password``, or |
| 318 |
``ansible_become_pass`` inventory variables. |
| 319 |
|
| 320 |
* Automatic tunnelling of SSH-dependent actions, such as the |
| 321 |
``synchronize`` module, is not yet supported. This will be addressed in a |
| 322 |
future release. |
| 323 |
|
| 324 |
To enable connection delegation, set ``mitogen_via=<inventory name>`` on the |
| 325 |
command line, or as host and group variables. |
| 326 |
|
| 327 |
.. code-block:: ini |
| 328 |
|
| 329 |
# Docker container on web1.dc1 is reachable via web1.dc1. |
| 330 |
[app-containers.web1.dc1] |
| 331 |
app1.web1.dc1 ansible_host=app1 ansible_connection=docker mitogen_via=web1.dc1 |
| 332 |
|
| 333 |
# Web servers in DC1 are reachable via bastion.dc1 |
| 334 |
[dc1] |
| 335 |
web1.dc1 |
| 336 |
web2.dc1 |
| 337 |
web3.dc1 |
| 338 |
|
| 339 |
[dc1:vars] |
| 340 |
mitogen_via = bastion.dc1 |
| 341 |
|
| 342 |
# Web servers in DC2 are reachable via bastion.dc2 |
| 343 |
[dc2] |
| 344 |
web1.dc2 |
| 345 |
web2.dc2 |
| 346 |
web3.dc2 |
| 347 |
|
| 348 |
[dc2:vars] |
| 349 |
mitogen_via = bastion.dc2 |
| 350 |
|
| 351 |
# Prod bastions are reachable via a magic account on a |
| 352 |
# corporate network gateway. |
| 353 |
[bastions] |
| 354 |
bastion.dc1 [email protected] |
| 355 |
bastion.dc2 [email protected] |
| 356 |
|
| 357 |
[corp-gateway] |
| 358 |
corp-gateway.internal |
| 359 |
|
| 360 |
|
| 361 |
File Transfer |
| 362 |
~~~~~~~~~~~~~ |
| 363 |
|
| 364 |
Normally :linux:man1:`sftp` or :linux:man1:`scp` are used to copy files by the |
| 365 |
:ans:mod:`~assemble`, :ans:mod:`~aws_s3`, :ans:mod:`~copy`, :ans:mod:`~patch`, |
| 366 |
:ans:mod:`~script`, :ans:mod:`~template`, :ans:mod:`~unarchive`, and |
| 367 |
:ans:mod:`~uri` actions, or when uploading modules with pipelining disabled. |
| 368 |
With Mitogen copies are implemented natively using the same interpreters, |
| 369 |
connection tree, and routed message bus that carries RPCs. |
| 370 |
|
| 371 |
This permits direct streaming between endpoints regardless of execution |
| 372 |
environment, without necessitating temporary copies in intermediary accounts or |
| 373 |
machines, for example when ``become`` is active, or in the presence of |
| 374 |
connection delegation. It also avoids the need to securely share temporary |
| 375 |
files between accounts and machines. |
| 376 |
|
| 377 |
As the implementation is self-contained, it is simple to make improvements like |
| 378 |
prioritizing transfers, supporting resume, or displaying progress bars. |
| 379 |
|
| 380 |
|
| 381 |
Safety |
| 382 |
^^^^^^ |
| 383 |
|
| 384 |
Transfers proceed to a hidden file in the destination directory, with content |
| 385 |
and metadata synced using :linux:man2:`fsync` prior to rename over any existing |
| 386 |
file. This ensures the file remains consistent at all times, in the event of a |
| 387 |
crash, or when overlapping `ansible-playbook` runs deploy differing file |
| 388 |
contents. |
| 389 |
|
| 390 |
The :linux:man1:`sftp` and :linux:man1:`scp` tools may cause undetected data |
| 391 |
corruption in the form of truncated files, or files containing intermingled |
| 392 |
data segments from overlapping runs. As part of normal operation, both tools |
| 393 |
expose a window where readers may observe inconsistent file contents. |
| 394 |
|
| 395 |
|
| 396 |
Performance |
| 397 |
^^^^^^^^^^^ |
| 398 |
|
| 399 |
One roundtrip initiates a transfer larger than 124 KiB, while smaller transfers |
| 400 |
are embedded in a 0-roundtrip pipelined call. For tools operating via SSH |
| 401 |
multiplexing, 4 roundtrips are required to configure the IO channel, followed |
| 402 |
by 6 roundtrips to transfer the file in the case of ``sftp``, in addition to |
| 403 |
the time to start the local and remote processes. |
| 404 |
|
| 405 |
An invocation of ``scp`` with an empty ``.profile`` over a 30 ms link takes |
| 406 |
~140 ms, wasting 110 ms per invocation, rising to ~2,000 ms over a 400 ms |
| 407 |
UK-India link, wasting 1,600 ms per invocation. |
| 408 |
|
| 409 |
|
| 410 |
Interpreter Reuse |
| 411 |
~~~~~~~~~~~~~~~~~ |
| 412 |
|
| 413 |
Python interpreters are aggressively reused to execute modules. While this |
| 414 |
works well, it violates an unwritten assumption, and so it is possible an |
| 415 |
earlier module execution could cause a subsequent module to fail, or for |
| 416 |
unrelated modules to interact poorly due to bad hygiene, such as |
| 417 |
monkey-patching that becomes stacked over repeat invocations. |
| 418 |
|
| 419 |
Before reporting a bug relating to a misbehaving module, please re-run with |
| 420 |
``-e mitogen_task_isolation=fork`` to see if the problem abates. This may be |
| 421 |
set per-task, paying attention to the possibility an earlier task may be the |
| 422 |
true cause of a failure. |
| 423 |
|
| 424 |
.. code-block:: yaml |
| 425 |
|
| 426 |
- name: My task. |
| 427 |
broken_module: |
| 428 |
some_option: true |
| 429 |
vars: |
| 430 |
mitogen_task_isolation: fork |
| 431 |
|
| 432 |
If forking solves your problem, **please report a bug regardless**, as an |
| 433 |
internal list can be updated to prevent others bumping into the same problem. |
| 434 |
|
| 435 |
|
| 436 |
Interpreter Recycling |
| 437 |
~~~~~~~~~~~~~~~~~~~~~ |
| 438 |
|
| 439 |
There is a per-target limit on the number of interpreters. Once 20 exist, the |
| 440 |
youngest is terminated before starting any new interpreter, preventing |
| 441 |
situations like below from triggering memory exhaustion. |
| 442 |
|
| 443 |
.. code-block:: yaml |
| 444 |
|
| 445 |
- hosts: corp_boxes |
| 446 |
vars: |
| 447 |
user_directory: [ |
| 448 |
# 10,000 corporate user accounts |
| 449 |
] |
| 450 |
tasks: |
| 451 |
- name: Create user bashrc |
| 452 |
become: true |
| 453 |
vars: |
| 454 |
ansible_become_user: "{{item}}" |
| 455 |
copy: |
| 456 |
src: bashrc |
| 457 |
dest: "~{{item}}/.bashrc" |
| 458 |
with_items: "{{user_directory}}" |
| 459 |
|
| 460 |
The youngest is chosen to preserve useful accounts like ``root`` and |
| 461 |
``postgresql`` that often appear early in a run, however it is simple to |
| 462 |
construct a playbook that defeats this strategy. A future version will key |
| 463 |
interpreters on the identity of their creating task, avoiding useful account |
| 464 |
recycling in every scenario. |
| 465 |
|
| 466 |
To modify the limit, set the ``MITOGEN_MAX_INTERPRETERS`` environment variable. |
| 467 |
|
| 468 |
|
| 469 |
Standard IO |
| 470 |
~~~~~~~~~~~ |
| 471 |
|
| 472 |
Ansible uses pseudo TTYs for most invocations to allow it to type interactive |
| 473 |
passwords, however pseudo TTYs are disabled where standard input is required or |
| 474 |
``sudo`` is not in use. Additionally when SSH multiplexing is enabled, a string |
| 475 |
like ``Shared connection to localhost closed\r\n`` appears in ``stderr`` of |
| 476 |
every invocation. |
| 477 |
|
| 478 |
Mitogen does not naturally require either of these, as command output is always |
| 479 |
embedded within framed messages, and it can simply call :py:func:`pty.openpty` |
| 480 |
in any location an interactive password must be typed. |
| 481 |
|
| 482 |
A major downside to Ansible's behaviour is that ``stdout`` and ``stderr`` are |
| 483 |
merged together into a single ``stdout`` variable, with carriage returns |
| 484 |
inserted in the output by the TTY layer. However ugly, the extension emulates |
| 485 |
this precisely, to avoid breaking playbooks that expect text to appear in |
| 486 |
specific variables with a particular linefeed style. |
| 487 |
|
| 488 |
|
| 489 |
.. _ansible_tempfiles: |
| 490 |
|
| 491 |
Temporary Files |
| 492 |
~~~~~~~~~~~~~~~ |
| 493 |
|
| 494 |
Temporary file handling in Ansible is tricky, and the precise behaviour varies |
| 495 |
across major versions. A variety of temporary files and directories are |
| 496 |
created, depending on the operating mode. |
| 497 |
|
| 498 |
In the best case when pipelining is enabled and no temporary uploads are |
| 499 |
required, for each task Ansible will create one directory below a |
| 500 |
system-supplied temporary directory returned by :func:`tempfile.mkdtemp`, owned |
| 501 |
by the target account a new-style module will execute in. |
| 502 |
|
| 503 |
In other cases depending on the task type, whether become is active, whether |
| 504 |
the target become user is privileged, whether the associated action plugin |
| 505 |
needs to upload files, and whether the associated module needs to store files, |
| 506 |
Ansible may: |
| 507 |
|
| 508 |
* Create a directory owned by the SSH user either under ``remote_tmp``, or a |
| 509 |
system-default directory, |
| 510 |
* Upload action dependencies such as non-new style modules or rendered |
| 511 |
templates to that directory via :linux:man1:`sftp` or :linux:man1:`scp`. |
| 512 |
* Attempt to modify the directory's access control list to grant access to the |
| 513 |
target user using :linux:man1:`setfacl`, requiring that tool to be installed |
| 514 |
and a supported filesystem to be in use, or for the |
| 515 |
``allow_world_readable_tmpfiles`` setting to be :data:`True`. |
| 516 |
* Create a directory owned by the target user either under ``remote_tmp``, or |
| 517 |
a system-default directory, if a new-style module needs a temporary directory |
| 518 |
and one was not previously created for a supporting file earlier in the |
| 519 |
invocation. |
| 520 |
|
| 521 |
In summary, for each task Ansible may create one or more of: |
| 522 |
|
| 523 |
* ``~ssh_user/<remote_tmp>/...`` owned by the login user, |
| 524 |
* ``$TMPDIR/ansible-tmp-...`` owned by the login user, |
| 525 |
* ``$TMPDIR/ansible-tmp-...`` owned by the login user with ACLs permitting |
| 526 |
write access by the become user, |
| 527 |
* ``~become_user/<remote_tmp>/...`` owned by the become user, |
| 528 |
* ``$TMPDIR/ansible_<modname>_payload_.../`` owned by the become user, |
| 529 |
* ``$TMPDIR/ansible-module-tmp-.../`` owned by the become user. |
| 530 |
|
| 531 |
|
| 532 |
Mitogen for Ansible |
| 533 |
^^^^^^^^^^^^^^^^^^^ |
| 534 |
|
| 535 |
As Mitogen can execute new-style modules from RAM, and transfer files to target |
| 536 |
user accounts without first writing an intermediary file in any separate login |
| 537 |
account, handling is relatively simplified. |
| 538 |
|
| 539 |
Temporary directories must exist to maintain compatibility with Ansible, as |
| 540 |
many modules introspect :data:`sys.argv` to find a directory where they may |
| 541 |
write files, however only one directory exists for the lifetime of each |
| 542 |
interpreter, its location is consistent for each account, and it is always |
| 543 |
privately owned by that account. |
| 544 |
|
| 545 |
During startup, the persistent remote interpreter tries the paths below until |
| 546 |
one is found that is writeable: |
| 547 |
|
| 548 |
1. ``$variable`` and tilde-expanded ``remote_tmp`` setting from |
| 549 |
``ansible.cfg`` |
| 550 |
2. ``$variable`` and tilde-expanded ``system_tmpdirs`` setting from |
| 551 |
``ansible.cfg`` |
| 552 |
3. ``TMPDIR`` environment variable |
| 553 |
4. ``TEMP`` environment variable |
| 554 |
5. ``TMP`` environment variable |
| 555 |
6. ``/tmp`` |
| 556 |
7. ``/var/tmp`` |
| 557 |
8. ``/usr/tmp`` |
| 558 |
9. Current working directory |
| 559 |
|
| 560 |
The directory is created at startup and recursively destroyed during interpeter |
| 561 |
shutdown. Subdirectories are automatically created and destroyed by the |
| 562 |
controller for each task that requires them. |
| 563 |
|
| 564 |
|
| 565 |
Round-trip Avoidance |
| 566 |
^^^^^^^^^^^^^^^^^^^^ |
| 567 |
|
| 568 |
Mitogen avoids many round-trips due to temporary file handling that are present |
| 569 |
in regular Ansible: |
| 570 |
|
| 571 |
* During task startup, it is not necessary to wait until the target has |
| 572 |
succeeded in creating a temporary directory. Instead, any failed attempt to |
| 573 |
create the directory will cause any subsequent RPC belonging to the same task |
| 574 |
to fail with the error that occurred. |
| 575 |
|
| 576 |
* As temporary directories are privately owned by the target user account, |
| 577 |
operations relating to modifying the directory to support cross-account |
| 578 |
access are avoided. |
| 579 |
|
| 580 |
* An explicit work-around is included to avoid the :ans:mod:`~copy` and |
| 581 |
:ans:mod:`~template` actions needlessly triggering a round-trip to set their |
| 582 |
temporary file as executable. |
| 583 |
|
| 584 |
* During task shutdown, it is not necessary to wait to learn if the target has |
| 585 |
succeeded in deleting a temporary directory, since any error that may occur |
| 586 |
is logged asynchronously via the logging framework, and the persistent |
| 587 |
remote interpreter arranges for all subdirectories to be destroyed during |
| 588 |
interpreter shutdown. |
| 589 |
|
| 590 |
|
| 591 |
.. _ansible_process_env: |
| 592 |
|
| 593 |
Process Environment Emulation |
| 594 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 595 |
|
| 596 |
Since Ansible discards processes after each module invocation, follow-up tasks |
| 597 |
often (but not always) receive a new environment that will usually include |
| 598 |
changes made by previous tasks. As such modifications are common, for |
| 599 |
compatibility the extension emulates the existing behaviour as closely as |
| 600 |
possible. |
| 601 |
|
| 602 |
Some scenarios exist where emulation is impossible, for example, applying |
| 603 |
``nsswitch.conf`` changes when ``nscd`` is not in use. If future scenarios |
| 604 |
appear that cannot be solved through emulation, the extension will be updated |
| 605 |
to automatically restart affected interpreters instead. |
| 606 |
|
| 607 |
|
| 608 |
DNS Resolution |
| 609 |
^^^^^^^^^^^^^^ |
| 610 |
|
| 611 |
Modifications to ``/etc/resolv.conf`` cause the glibc resolver configuration to |
| 612 |
be reloaded via :linux:man3:`res_init`. This isn't necessary on some Linux |
| 613 |
distributions carrying glibc patches to automatically check |
| 614 |
``/etc/resolv.conf`` periodically, however it is necessary on at least Debian |
| 615 |
and BSD derivatives. |
| 616 |
|
| 617 |
|
| 618 |
``/etc/environment`` |
| 619 |
^^^^^^^^^^^^^^^^^^^^ |
| 620 |
|
| 621 |
When ``become: true`` is active or SSH multiplexing is disabled, modifications |
| 622 |
by previous tasks to ``/etc/environment`` and ``$HOME/.pam_environment`` are |
| 623 |
normally reflected, since the content of those files is reapplied by `PAM |
| 624 |
<https://en.wikipedia.org/wiki/Pluggable_authentication_module>`_ via `pam_env` |
| 625 |
on each authentication of ``sudo`` or ``sshd``. |
| 626 |
|
| 627 |
Both files are monitored for changes, and changes are applied where it appears |
| 628 |
safe to do so: |
| 629 |
|
| 630 |
* New keys are added if they did not otherwise exist in the inherited |
| 631 |
environment, or previously had the same value as found in the file before it |
| 632 |
changed. |
| 633 |
|
| 634 |
* Given a key (such as ``http_proxy``) added to the file where no such key |
| 635 |
exists in the environment, the key will be added. |
| 636 |
|
| 637 |
* Given a key (such as ``PATH``) where an existing environment key exists with |
| 638 |
a different value, the update or deletion will be ignored, as it is likely |
| 639 |
the key was overridden elsewhere after `pam_env` ran, such as by |
| 640 |
``/etc/profile``. |
| 641 |
|
| 642 |
* Given a key removed from the file that had the same value as the existing |
| 643 |
environment key, the key will be removed. |
| 644 |
|
| 645 |
|
| 646 |
How Modules Execute |
| 647 |
~~~~~~~~~~~~~~~~~~~ |
| 648 |
|
| 649 |
Ansible usually modifies, recompresses and reuploads modules every time they |
| 650 |
run on a target, work that must be repeated by the controller for every |
| 651 |
playbook step. |
| 652 |
|
| 653 |
With the extension any modifications are done on the target, allowing pristine |
| 654 |
copies of modules to be cached, reducing the necessity to re-transfer modules |
| 655 |
for each invocation. Unmodified modules are uploaded once on first use and |
| 656 |
cached in RAM for the remainder of the run. |
| 657 |
|
| 658 |
**Binary** |
| 659 |
Native executables detected using a complex heuristic. Arguments are |
| 660 |
supplied as a JSON file whose path is the sole script parameter. |
| 661 |
|
| 662 |
**Module Replacer** |
| 663 |
Python scripts detected by the presence of |
| 664 |
``#<<INCLUDE_ANSIBLE_MODULE_COMMON>>`` appearing in their source. This type |
| 665 |
is not yet supported. |
| 666 |
|
| 667 |
**New-Style** |
| 668 |
Python scripts detected by the presence of ``from ansible.module_utils.`` |
| 669 |
appearing in their source. Arguments are supplied as JSON written to |
| 670 |
``sys.stdin`` of the target interpreter. |
| 671 |
|
| 672 |
**JSON_ARGS** |
| 673 |
Detected by the presence of ``INCLUDE_ANSIBLE_MODULE_JSON_ARGS`` appearing |
| 674 |
in the script source. The interpreter directive (``#!interpreter``) is |
| 675 |
adjusted to match the corresponding value of ``{{ansible_*_interpreter}}`` |
| 676 |
if one is set. Arguments are supplied as JSON mixed into the script as a |
| 677 |
replacement for ``INCLUDE_ANSIBLE_MODULE_JSON_ARGS``. |
| 678 |
|
| 679 |
**WANT_JSON** |
| 680 |
Detected by the presence of ``WANT_JSON`` appearing in the script source. |
| 681 |
The interpreter directive is adjusted as above. Arguments are supplied as a |
| 682 |
JSON file whose path is the sole script parameter. |
| 683 |
|
| 684 |
**Old Style** |
| 685 |
Files not matching any of the above tests. The interpreter directive is |
| 686 |
adjusted as above. Arguments are supplied as a file whose path is the sole |
| 687 |
script parameter. The format of the file is ``"key=repr(value)[ |
| 688 |
key2=repr(value2)[ ..]]Â "``. |
| 689 |
|
| 690 |
|
| 691 |
Runtime Patches |
| 692 |
~~~~~~~~~~~~~~~ |
| 693 |
|
| 694 |
Three small runtime patches are employed in ``strategy.py`` to hook into |
| 695 |
desirable locations, in order to override uses of shell, the module executor, |
| 696 |
and the mechanism for selecting a connection plug-in. While it is hoped the |
| 697 |
patches can be avoided in future, for interesting versions of Ansible deployed |
| 698 |
today this simply is not possible, and so they continue to be required. |
| 699 |
|
| 700 |
The patches are concise and behave conservatively, including by disabling |
| 701 |
themselves when non-Mitogen connections are in use. Additional third party |
| 702 |
plug-ins are unlikely to attempt similar patches, so the risk to an established |
| 703 |
configuration should be minimal. |
| 704 |
|
| 705 |
|
| 706 |
Flag Emulation |
| 707 |
~~~~~~~~~~~~~~ |
| 708 |
|
| 709 |
Mitogen re-parses ``sudo_flags``, ``become_flags``, and ``ssh_flags`` using |
| 710 |
option parsers extracted from `sudo(1)` and `ssh(1)` in order to emulate their |
| 711 |
equivalent semantics. This allows: |
| 712 |
|
| 713 |
* robust support for common ``ansible.cfg`` tricks without reconfiguration, |
| 714 |
such as forwarding SSH agents across ``sudo`` invocations, |
| 715 |
* reporting on conflicting flag combinations, |
| 716 |
* reporting on unsupported flag combinations, |
| 717 |
* internally special-casing certain behaviour (like recursive agent forwarding) |
| 718 |
without boring the user with the details, |
| 719 |
* avoiding opening the extension up to untestable scenarios where users can |
| 720 |
insert arbitrary garbage between Mitogen and the components it integrates |
| 721 |
with, |
| 722 |
* precise emulation by an alternative implementation, for example if Mitogen |
| 723 |
grew support for Paramiko. |
| 724 |
|
| 725 |
|
| 726 |
Connection Types |
| 727 |
---------------- |
| 728 |
|
| 729 |
Matching Ansible, connection variables are treated on a per-task basis, causing |
| 730 |
establishment of additional reuseable interpreters as necessary to match the |
| 731 |
configuration of each task. |
| 732 |
|
| 733 |
|
| 734 |
.. _method-buildah: |
| 735 |
|
| 736 |
Buildah |
| 737 |
~~~~~~~ |
| 738 |
|
| 739 |
Like the :ans:conn:`buildah` except connection delegation is supported. |
| 740 |
|
| 741 |
* ``ansible_host``: Name of Buildah container (default: inventory hostname). |
| 742 |
* ``ansible_user``: Name of user within the container to execute as. |
| 743 |
|
| 744 |
|
| 745 |
.. _doas: |
| 746 |
|
| 747 |
Doas |
| 748 |
~~~~ |
| 749 |
|
| 750 |
``doas`` can be used as a connection method that supports connection delegation, or |
| 751 |
as a become method. |
| 752 |
|
| 753 |
When used as a become method: |
| 754 |
|
| 755 |
* ``ansible_python_interpreter`` |
| 756 |
* ``ansible_become_exe`` / ``ansible_doas_exe``: path to ``doas`` binary. |
| 757 |
* ``ansible_become_user`` (default: ``root``) |
| 758 |
* ``ansible_become_pass`` (default: assume passwordless) |
| 759 |
* ``mitogen_mask_remote_name``: if :data:`True`, mask the identity of the |
| 760 |
Ansible controller process on remote machines. To simplify diagnostics, |
| 761 |
Mitogen produces remote processes named like |
| 762 |
`"mitogen:[email protected]:1234"`, however this may be a privacy issue in |
| 763 |
some circumstances. |
| 764 |
* ansible.cfg: ``timeout`` |
| 765 |
|
| 766 |
When used as the ``mitogen_doas`` connection method: |
| 767 |
|
| 768 |
* The inventory hostname has no special meaning. |
| 769 |
* ``ansible_user``: username to use. |
| 770 |
* ``ansible_password``: password to use. |
| 771 |
* ``ansible_doas_exe``: path to ``doas`` binary. |
| 772 |
* ``ansible_python_interpreter`` |
| 773 |
|
| 774 |
|
| 775 |
.. _method-docker: |
| 776 |
|
| 777 |
Docker |
| 778 |
~~~~~~ |
| 779 |
|
| 780 |
Like the :ans:conn:`docker` except connection delegation is supported. |
| 781 |
|
| 782 |
* ``ansible_host``: Name of Docker container (default: inventory hostname). |
| 783 |
* ``ansible_user``: Name of user within the container to execute as. |
| 784 |
* ``mitogen_mask_remote_name``: if :data:`True`, mask the identity of the |
| 785 |
Ansible controller process on remote machines. To simplify diagnostics, |
| 786 |
Mitogen produces remote processes named like |
| 787 |
`"mitogen:[email protected]:1234"`, however this may be a privacy issue in |
| 788 |
some circumstances. |
| 789 |
|
| 790 |
|
| 791 |
.. _method-jail: |
| 792 |
|
| 793 |
FreeBSD Jail |
| 794 |
~~~~~~~~~~~~ |
| 795 |
|
| 796 |
Like the :ans:conn:`jail` except connection delegation is supported. |
| 797 |
|
| 798 |
* ``ansible_host``: Name of jail (default: inventory hostname). |
| 799 |
* ``ansible_user``: Name of user within the jail to execute as. |
| 800 |
* ``mitogen_mask_remote_name``: if :data:`True`, mask the identity of the |
| 801 |
Ansible controller process on remote machines. To simplify diagnostics, |
| 802 |
Mitogen produces remote processes named like |
| 803 |
`"mitogen:[email protected]:1234"`, however this may be a privacy issue in |
| 804 |
some circumstances. |
| 805 |
|
| 806 |
|
| 807 |
.. _method-kubectl: |
| 808 |
|
| 809 |
Kubernetes Pod |
| 810 |
~~~~~~~~~~~~~~ |
| 811 |
|
| 812 |
Like the :ans:conn:`kubectl` except connection delegation is supported. |
| 813 |
|
| 814 |
* ``ansible_host``: Name of pod (default: inventory hostname). |
| 815 |
* ``ansible_user``: Name of user to authenticate to API as. |
| 816 |
* ``mitogen_mask_remote_name``: if :data:`True`, mask the identity of the |
| 817 |
Ansible controller process on remote machines. To simplify diagnostics, |
| 818 |
Mitogen produces remote processes named like |
| 819 |
`"mitogen:[email protected]:1234"`, however this may be a privacy issue in |
| 820 |
some circumstances. |
| 821 |
|
| 822 |
|
| 823 |
Local |
| 824 |
~~~~~ |
| 825 |
|
| 826 |
Like the :ans:conn:`local` except connection delegation is supported. |
| 827 |
|
| 828 |
* ``ansible_python_interpreter`` |
| 829 |
|
| 830 |
|
| 831 |
Podman |
| 832 |
~~~~~~ |
| 833 |
|
| 834 |
Like :ans:conn:`podman` except connection delegation is supported. |
| 835 |
|
| 836 |
* ``ansible_host``: Name of container (default: inventory hostname). |
| 837 |
* ``ansible_user``: Name of user within the container to execute as. |
| 838 |
* ``mitogen_mask_remote_name``: if :data:`True`, mask the identity of the |
| 839 |
Ansible controller process on remote machines. To simplify diagnostics, |
| 840 |
Mitogen produces remote processes named like |
| 841 |
`"mitogen:[email protected]:1234"`, however this may be a privacy issue in |
| 842 |
some circumstances. |
| 843 |
|
| 844 |
|
| 845 |
Process Model |
| 846 |
^^^^^^^^^^^^^ |
| 847 |
|
| 848 |
Ansible usually executes local connection commands as a transient subprocess of |
| 849 |
the forked worker executing a task. With the extension, the local connection |
| 850 |
exists as a persistent subprocess of the connection multiplexer. |
| 851 |
|
| 852 |
This means that global state mutations made to the top-level Ansible process |
| 853 |
that are normally visible to newly forked subprocesses, such as vars plug-ins |
| 854 |
that modify the environment, will not be reflected when executing local |
| 855 |
commands without additional effort. |
| 856 |
|
| 857 |
During execution the extension presently mimics the working directory and |
| 858 |
process environment inheritence of regular Ansible, however it is possible some |
| 859 |
additional differences exist that may break existing playbooks. |
| 860 |
|
| 861 |
|
| 862 |
.. _method-lxc: |
| 863 |
|
| 864 |
LXC |
| 865 |
~~~ |
| 866 |
|
| 867 |
Connect to classic LXC containers, like the :ans:conn:`lxc` except connection |
| 868 |
delegation is supported, and ``lxc-attach`` is always used rather than the LXC |
| 869 |
Python bindings, as is usual with ``lxc``. |
| 870 |
|
| 871 |
* ``ansible_python_interpreter`` |
| 872 |
* ``ansible_host``: Name of LXC container (default: inventory hostname). |
| 873 |
* ``mitogen_lxc_attach_path``: path to ``lxc-attach`` command if not available |
| 874 |
on the system path. |
| 875 |
* ``mitogen_mask_remote_name``: if :data:`True`, mask the identity of the |
| 876 |
Ansible controller process on remote machines. To simplify diagnostics, |
| 877 |
Mitogen produces remote processes named like |
| 878 |
`"mitogen:[email protected]:1234"`, however this may be a privacy issue in |
| 879 |
some circumstances. |
| 880 |
|
| 881 |
|
| 882 |
.. _method-lxd: |
| 883 |
|
| 884 |
LXD |
| 885 |
~~~ |
| 886 |
|
| 887 |
Connect to modern LXD containers, like the :ans:conn:`lxd` except connection |
| 888 |
delegation is supported. The ``lxc`` command must be available on the host |
| 889 |
machine. |
| 890 |
|
| 891 |
* ``ansible_python_interpreter`` |
| 892 |
* ``ansible_host``: Name of LXC container (default: inventory hostname). |
| 893 |
* ``mitogen_lxc_path``: path to ``lxc`` command if not available on the system |
| 894 |
path. |
| 895 |
* ``mitogen_mask_remote_name``: if :data:`True`, mask the identity of the |
| 896 |
Ansible controller process on remote machines. To simplify diagnostics, |
| 897 |
Mitogen produces remote processes named like |
| 898 |
`"mitogen:[email protected]:1234"`, however this may be a privacy issue in |
| 899 |
some circumstances. |
| 900 |
|
| 901 |
|
| 902 |
.. _machinectl: |
| 903 |
|
| 904 |
Machinectl |
| 905 |
~~~~~~~~~~ |
| 906 |
|
| 907 |
Like the `machinectl third party plugin |
| 908 |
<https://github.com/BaxterStockman/ansible-connection-machinectl>`_ except |
| 909 |
connection delegation is supported. This is a light wrapper around the |
| 910 |
:ref:`setns <setns>` method. |
| 911 |
|
| 912 |
* ``ansible_host``: Name of Docker container (default: inventory hostname). |
| 913 |
* ``ansible_user``: Name of user within the container to execute as. |
| 914 |
* ``mitogen_machinectl_path``: path to ``machinectl`` command if not available |
| 915 |
as ``/bin/machinectl``. |
| 916 |
* ``mitogen_mask_remote_name``: if :data:`True`, mask the identity of the |
| 917 |
Ansible controller process on remote machines. To simplify diagnostics, |
| 918 |
Mitogen produces remote processes named like |
| 919 |
`"mitogen:[email protected]:1234"`, however this may be a privacy issue in |
| 920 |
some circumstances. |
| 921 |
|
| 922 |
|
| 923 |
.. _setns: |
| 924 |
|
| 925 |
Setns |
| 926 |
~~~~~ |
| 927 |
|
| 928 |
The ``setns`` method connects to Linux containers via `setns(2) |
| 929 |
<https://linux.die.net/man/2/setns>`_. Unlike :ref:`method-docker`, |
| 930 |
:ref:`method-lxc`, and :ref:`method-lxd` the namespace transition is handled |
| 931 |
internally, ensuring optimal throughput to the child. This is necessary for |
| 932 |
:ref:`machinectl` where only PTY channels are supported. |
| 933 |
|
| 934 |
A utility program must be installed to discover the PID of the container's root |
| 935 |
process. |
| 936 |
|
| 937 |
* ``mitogen_kind``: one of ``docker``, ``lxc``, ``lxd`` or ``machinectl``. |
| 938 |
* ``ansible_host``: Name of container as it is known to the corresponding tool |
| 939 |
(default: inventory hostname). |
| 940 |
* ``ansible_user``: Name of user within the container to execute as. |
| 941 |
* ``mitogen_docker_path``: path to Docker if not available on the system path. |
| 942 |
* ``mitogen_lxc_path``: path to LXD's ``lxc`` command if not available as |
| 943 |
``lxc-info``. |
| 944 |
* ``mitogen_lxc_info_path``: path to LXC classic's ``lxc-info`` command if not |
| 945 |
available as ``lxc-info``. |
| 946 |
* ``mitogen_machinectl_path``: path to ``machinectl`` command if not available |
| 947 |
as ``/bin/machinectl``. |
| 948 |
|
| 949 |
|
| 950 |
.. _su: |
| 951 |
|
| 952 |
Su |
| 953 |
~~ |
| 954 |
|
| 955 |
Su can be used as a connection method that supports connection delegation, or |
| 956 |
as a become method. |
| 957 |
|
| 958 |
When used as a become method: |
| 959 |
|
| 960 |
* ``ansible_python_interpreter`` |
| 961 |
* ``ansible_su_exe``, ``ansible_become_exe`` |
| 962 |
* ``ansible_su_user``, ``ansible_become_user`` (default: ``root``) |
| 963 |
* ``ansible_su_pass``, ``ansible_become_pass`` (default: assume passwordless) |
| 964 |
* ``su_flags``, ``become_flags`` |
| 965 |
* ``mitogen_mask_remote_name``: if :data:`True`, mask the identity of the |
| 966 |
Ansible controller process on remote machines. To simplify diagnostics, |
| 967 |
Mitogen produces remote processes named like |
| 968 |
`"mitogen:[email protected]:1234"`, however this may be a privacy issue in |
| 969 |
some circumstances. |
| 970 |
* ansible.cfg: ``timeout`` |
| 971 |
|
| 972 |
When used as the ``mitogen_su`` connection method: |
| 973 |
|
| 974 |
* The inventory hostname has no special meaning. |
| 975 |
* ``ansible_user``: username to su as. |
| 976 |
* ``ansible_password``: password to su as. |
| 977 |
* ``ansible_python_interpreter`` |
| 978 |
|
| 979 |
|
| 980 |
.. _sudo: |
| 981 |
|
| 982 |
Sudo |
| 983 |
~~~~ |
| 984 |
|
| 985 |
Sudo can be used as a connection method that supports connection delegation, or |
| 986 |
as a become method. |
| 987 |
|
| 988 |
When used as a become method: |
| 989 |
|
| 990 |
* ``ansible_python_interpreter`` |
| 991 |
* ``ansible_sudo_exe``, ``ansible_become_exe`` |
| 992 |
* ``ansible_sudo_user``, ``ansible_become_user`` (default: ``root``) |
| 993 |
* ``ansible_sudo_pass``, ``ansible_become_pass`` (default: assume passwordless) |
| 994 |
* ``sudo_flags``, ``become_flags`` |
| 995 |
* ``mitogen_mask_remote_name``: if :data:`True`, mask the identity of the |
| 996 |
Ansible controller process on remote machines. To simplify diagnostics, |
| 997 |
Mitogen produces remote processes named like |
| 998 |
`"mitogen:[email protected]:1234"`, however this may be a privacy issue in |
| 999 |
some circumstances. |
| 1000 |
* ansible.cfg: ``timeout`` |
| 1001 |
|
| 1002 |
When used as the ``mitogen_sudo`` connection method: |
| 1003 |
|
| 1004 |
* The inventory hostname has no special meaning. |
| 1005 |
* ``ansible_user``: username to sudo as. |
| 1006 |
* ``ansible_password``: password to sudo as. |
| 1007 |
* ``sudo_flags``, ``become_flags`` |
| 1008 |
* ``ansible_python_interpreter`` |
| 1009 |
|
| 1010 |
|
| 1011 |
SSH |
| 1012 |
~~~ |
| 1013 |
|
| 1014 |
Like the :ans:conn:`ssh` except connection delegation is supported. |
| 1015 |
|
| 1016 |
* ``ansible_ssh_timeout`` |
| 1017 |
* ``ansible_host``, ``ansible_ssh_host`` |
| 1018 |
* ``ansible_user``, ``ansible_ssh_user`` |
| 1019 |
* ``ansible_port``, ``ssh_port`` |
| 1020 |
* ``ansible_ssh_executable``, ``ssh_executable`` |
| 1021 |
* ``ansible_ssh_private_key_file`` |
| 1022 |
* ``ansible_ssh_pass``, ``ansible_ssh_password``, ``ansible_password`` |
| 1023 |
(default: assume passwordless) |
| 1024 |
* ``ansible_ssh_host_key_checking``, ``ansible_host_key_checking`` (default: |
| 1025 |
:data:`True`) |
| 1026 |
* ``ssh_args``, ``ssh_common_args``, ``ssh_extra_args`` |
| 1027 |
* ``mitogen_mask_remote_name``: if :data:`True`, mask the identity of the |
| 1028 |
Ansible controller process on remote machines. To simplify diagnostics, |
| 1029 |
Mitogen produces remote processes named like |
| 1030 |
`"mitogen:[email protected]:1234"`, however this may be a privacy issue in |
| 1031 |
some circumstances. |
| 1032 |
* ``mitogen_ssh_debug_level``: integer between `0..3` indicating the SSH client |
| 1033 |
debug level. Ansible must also be run with '-vvv' to view the output. |
| 1034 |
* ``mitogen_ssh_compression``: :data:`True` to enable SSH compression, |
| 1035 |
otherwise :data:`False`. This will change to off by default in a future |
| 1036 |
release. If you are targetting many hosts on a fast network, please consider |
| 1037 |
disabling SSH compression. |
| 1038 |
* ``mitogen_ssh_keepalive_count``: integer count of server keepalive messages to |
| 1039 |
which no reply is received before considering the SSH server dead. Defaults |
| 1040 |
to 10. |
| 1041 |
* ``mitogen_ssh_keepalive_interval``: integer seconds delay between keepalive |
| 1042 |
messages. Defaults to 30. |
| 1043 |
|
| 1044 |
|
| 1045 |
Debugging |
| 1046 |
--------- |
| 1047 |
|
| 1048 |
Diagnostics and :py:mod:`logging` package output on targets are usually |
| 1049 |
discarded. Mitogen can capture and forward them to the controller. |
| 1050 |
|
| 1051 |
Mitogen >= 0.3.45 uses `Ansible verbosity`_ (e.g. ``ansible -vvv ...``) and an |
| 1052 |
environment variable ``MITOGEN_LOG_LEVEL`` (e.g. ``MITOGEN_LOG_LEVEL=debug``) |
| 1053 |
to configure this. Both are required for more verbose Mitogen logging output. |
| 1054 |
|
| 1055 |
Ansible verbosity is mapped to Mitogen logging as follows |
| 1056 |
|
| 1057 |
================= =========================================================== |
| 1058 |
Ansible verbosity Mitogen output |
| 1059 |
================= =========================================================== |
| 1060 |
0-2 ``ERROR``, ``WARNING`` |
| 1061 |
3 ``ERROR``, ``WARNING``, ``INFO``, ``DEBUG`` except IO |
| 1062 |
4-5 ``ERROR``, ``WARNING``, ``INFO``, ``DEBUG`` including IO |
| 1063 |
================= =========================================================== |
| 1064 |
|
| 1065 |
``MITOGEN_LOG_LEVEL`` accepts a Python `logging level`_ name, or ``IO`` for |
| 1066 |
extra ``DEBUG`` output of Mitogen's network traffic. |
| 1067 |
|
| 1068 |
Mitogen <= 0.3.44 only uses Ansible verbosity to configure the output. |
| 1069 |
``MITOGEN_LOG_LEVEL`` is ignored. |
| 1070 |
|
| 1071 |
While uncaptured standard IO and the logging package on targets is forwarded, |
| 1072 |
it is not possible to receive IO activity logs, as the forwarding process would |
| 1073 |
would itself generate additional IO. |
| 1074 |
|
| 1075 |
To receive a complete trace of every process on every machine, file-based |
| 1076 |
logging is necessary. File-based logging can be enabled by setting |
| 1077 |
``MITOGEN_ROUTER_DEBUG=1`` in your environment. When file-based logging is |
| 1078 |
enabled, one file per context will be created on the local machine and every |
| 1079 |
target machine, as ``/tmp/mitogen.<pid>.log``. |
| 1080 |
|
| 1081 |
.. _ansible verbosity: https://docs.ansible.com/projects/ansible/latest/reference_appendices/config.html#default-verbosity |
| 1082 |
.. _logging level: https://docs.python.org/3/library/logging.html#logging-levels |
| 1083 |
|
| 1084 |
|
| 1085 |
Common Problems |
| 1086 |
~~~~~~~~~~~~~~~ |
| 1087 |
|
| 1088 |
The most common bug reports fall into the following categories, so it is worth |
| 1089 |
checking whether you can categorize a problem using the tools provided before |
| 1090 |
reporting it: |
| 1091 |
|
| 1092 |
**Missed/Incorrect Configuration Variables** |
| 1093 |
In some cases Ansible may support a configuration variable that Mitogen |
| 1094 |
does not yet support, or Mitogen supports, but the support is broken. For |
| 1095 |
example, Mitogen may pick the wrong username or SSH parameters. |
| 1096 |
|
| 1097 |
To detect this, use the special ``mitogen_get_stack`` action described |
| 1098 |
below to verify the settings Mitogen has chosen for the connection make |
| 1099 |
sense. |
| 1100 |
|
| 1101 |
**Process Environment Differences** |
| 1102 |
Mitogen's process model differs significantly to Ansible's in many places. |
| 1103 |
In the past, bugs have been reported because Ansible plug-ins modify an |
| 1104 |
environment variable after Mitogen processes are started. |
| 1105 |
|
| 1106 |
If your task's failure may relate to the process environment in some way, |
| 1107 |
for example, ``SSH_AUTH_SOCK``, ``LC_ALL`` or ``PATH``, then an environment |
| 1108 |
difference may explain it. Environment differences are always considered |
| 1109 |
bugs in the extension, and are very easy to repair, so even if you find a |
| 1110 |
workaround, please report them to avoid someone else encountering the same |
| 1111 |
problem. |
| 1112 |
|
| 1113 |
**Variable Expansion Differences** |
| 1114 |
To avoid many classes of bugs, Mitogen avoids shell wherever possible. |
| 1115 |
Ansible however is traditionally built on shell, and it is often difficult |
| 1116 |
to tell just how many times a configuration parameter will pass through |
| 1117 |
shell expansion and quoting, and in what context before it is used. |
| 1118 |
|
| 1119 |
Due to this, in some circumstances Mitogen may parse some expanded |
| 1120 |
variables differently, for example, in the wrong user account. Careful |
| 1121 |
review of ``-vvv`` and ``mitogen_ssh_debug_level`` logs can reveal this. |
| 1122 |
For example in the past, Mitogen used a different method of expanding |
| 1123 |
``~/.ssh/id_rsa``, causing authentication to fail when ``ansible-playbook`` |
| 1124 |
was run via ``sudo -E``. |
| 1125 |
|
| 1126 |
**External Tool Integration Differences** |
| 1127 |
Mitogen reimplements any aspect of Ansible that involves integrating with |
| 1128 |
SSH, sudo, Docker, or related tools. For this reason, sometimes its support |
| 1129 |
for those tools differs or is less mature than in Ansible. |
| 1130 |
|
| 1131 |
In the past Mitogen has had bug reports due to failing to recognize a |
| 1132 |
particular variation of a login or password prompt on an exotic or |
| 1133 |
non-English operating system, or confusing a login banner for a password |
| 1134 |
prompt. Careful review of ``-vvv`` logs help identify these cases, as |
| 1135 |
Mitogen logs all strings it receives during connection, and how it |
| 1136 |
interprets them. |
| 1137 |
|
| 1138 |
|
| 1139 |
.. _mitogen-get-stack: |
| 1140 |
|
| 1141 |
The `mitogen_get_stack` Action |
| 1142 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 1143 |
|
| 1144 |
When a Mitogen strategy is loaded, a special ``mitogen_get_stack`` action is |
| 1145 |
available that returns a concise description of the connection configuration as |
| 1146 |
extracted from Ansible and passed to the core library. Using it, you can learn |
| 1147 |
whether a problem lies in the Ansible extension or deeper in library code. |
| 1148 |
|
| 1149 |
The action may be used in a playbook as ``mitogen_get_stack:`` just like a |
| 1150 |
regular module, or directly from the command-line:: |
| 1151 |
|
| 1152 |
$ ANSIBLE_STRATEGY=mitogen_linear ansible -m mitogen_get_stack -b -k k3 |
| 1153 |
SSH password: |
| 1154 |
k3 | SUCCESS => { |
| 1155 |
"changed": true, |
| 1156 |
"result": [ |
| 1157 |
{ |
| 1158 |
"kwargs": { |
| 1159 |
"check_host_keys": "enforce", |
| 1160 |
"connect_timeout": 10, |
| 1161 |
"hostname": "k3", |
| 1162 |
"identities_only": false, |
| 1163 |
"identity_file": null, |
| 1164 |
"password": "mysecretpassword", |
| 1165 |
"port": null, |
| 1166 |
"python_path": null, |
| 1167 |
"ssh_args": [ |
| 1168 |
"-C", |
| 1169 |
"-o", |
| 1170 |
"ControlMaster=auto", |
| 1171 |
"-o", |
| 1172 |
"ControlPersist=60s" |
| 1173 |
], |
| 1174 |
"ssh_debug_level": null, |
| 1175 |
"ssh_path": "ssh", |
| 1176 |
"username": null |
| 1177 |
}, |
| 1178 |
"method": "ssh" |
| 1179 |
}, |
| 1180 |
{ |
| 1181 |
"enable_lru": true, |
| 1182 |
"kwargs": { |
| 1183 |
"connect_timeout": 10, |
| 1184 |
"password": null, |
| 1185 |
"python_path": null, |
| 1186 |
"sudo_args": [ |
| 1187 |
"-H", |
| 1188 |
"-S", |
| 1189 |
"-n" |
| 1190 |
], |
| 1191 |
"sudo_path": null, |
| 1192 |
"username": "root" |
| 1193 |
}, |
| 1194 |
"method": "sudo" |
| 1195 |
} |
| 1196 |
] |
| 1197 |
} |
| 1198 |
|
| 1199 |
Each object in the list represents a single 'hop' in the connection, from |
| 1200 |
nearest to furthest. Unlike in Ansible, the core library treats ``become`` |
| 1201 |
steps and SSH steps identically, so they are represented distinctly in the |
| 1202 |
output. |
| 1203 |
|
| 1204 |
The presence of ``null`` means no explicit value was extracted from Ansible, |
| 1205 |
and either the Mitogen library or SSH will choose a value for the parameter. In |
| 1206 |
the example above, Mitogen will choose ``/usr/bin/python`` for ``python_path``, |
| 1207 |
and SSH will choose ``22`` for ``port``, or whatever ``Port`` it parses from |
| 1208 |
``~/.ssh/config``. Note the presence of ``null`` may indicate the extension |
| 1209 |
failed to extract the correct value. |
| 1210 |
|
| 1211 |
When using ``mitogen_get_stack`` to diagnose a problem, pay special attention |
| 1212 |
to ensuring the invocation exactly matches the problematic task. For example, |
| 1213 |
if the failing task has ``delegate_to:`` or ``become:`` enabled, the |
| 1214 |
``mitogen_get_stack`` invocation must include those statements in order for the |
| 1215 |
output to be accurate. |
| 1216 |
|
| 1217 |
If a playbook cannot start at all, you may need to temporarily use |
| 1218 |
``gather_facts: no`` to allow the first task to proceed. This action does not |
| 1219 |
create connections, so if it is the first task, it is still possible to review |
| 1220 |
its output. |
| 1221 |
|
| 1222 |
|
| 1223 |
The `mitogen_ssh_debug_level` Variable |
| 1224 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 1225 |
|
| 1226 |
Mitogen has support for capturing SSH diagnostic logs, and integrating them |
| 1227 |
into the regular debug log output produced when ``-vvv`` is active. This |
| 1228 |
provides a single audit trail of every component active during SSH |
| 1229 |
authentication. |
| 1230 |
|
| 1231 |
Particularly for authentication failures, setting this variable to 3, in |
| 1232 |
combination with ``-vvv``, allows review of every parameter passed to SSH, and |
| 1233 |
review of every action SSH attempted during authentication. |
| 1234 |
|
| 1235 |
For example, this method can be used to ascertain whether SSH attempted agent |
| 1236 |
authentication, or what private key files it was able to access and which it tried. |
| 1237 |
|
| 1238 |
|
| 1239 |
Post-authentication Bootstrap Failure |
| 1240 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 1241 |
|
| 1242 |
If logging indicates Mitogen was able to authenticate, but some error occurred |
| 1243 |
after authentication preventing the Python bootstrap from completing, it can be |
| 1244 |
immensely useful to temporarily replace ``ansible_python_interpreter`` with a |
| 1245 |
wrapper that runs Python under ``strace``:: |
| 1246 |
|
| 1247 |
$ ssh badbox |
| 1248 |
|
| 1249 |
badbox$ cat > strace-python.sh |
| 1250 |
#!/bin/sh |
| 1251 |
strace -o /tmp/strace-python.$$ -ff -s 100 python "$@" |
| 1252 |
^D |
| 1253 |
|
| 1254 |
badbox$ chmod +x strace-python.sh |
| 1255 |
badbox$ logout |
| 1256 |
|
| 1257 |
$ ansible-playbook site.yml \ |
| 1258 |
-e ansible_python_interpreter=./strace-python.sh \ |
| 1259 |
-l badbox |
| 1260 |
|
| 1261 |
This will produce a potentially large number of log files under ``/tmp/``. The |
| 1262 |
lowest-numbered traced PID is generally the main Python interpreter. The most |
| 1263 |
intricate bootstrap steps happen there, any error should be visible near the |
| 1264 |
end of the trace. |
| 1265 |
|
| 1266 |
It is also possible the first stage bootstrap failed. That is usually the next |
| 1267 |
lowest-numbered PID and tends to be the smallest file. Even if you can't |
| 1268 |
ascertain the problem with your configuration from these logs, including them |
| 1269 |
in a bug report can save days of detective effort. |
| 1270 |
|
| 1271 |
|
| 1272 |
.. _diagnosing-hangs: |
| 1273 |
|
| 1274 |
Diagnosing Hangs |
| 1275 |
~~~~~~~~~~~~~~~~ |
| 1276 |
|
| 1277 |
If you encounter a hang, the ``MITOGEN_DUMP_THREAD_STACKS=<secs>`` environment |
| 1278 |
variable arranges for each process on each machine to dump each thread stack |
| 1279 |
into the logging framework every `secs` seconds, which is visible when running |
| 1280 |
with ``-vvv``. |
| 1281 |
|
| 1282 |
However, certain controller hangs may render ``MITOGEN_DUMP_THREAD_STACKS`` |
| 1283 |
ineffective, or occur too infrequently for interactive reproduction. In these |
| 1284 |
cases :py:mod:`faulthandler` may be used with Python >= 3.3: |
| 1285 |
|
| 1286 |
1. Once the hang occurs, observe the process tree using ``pstree`` or ``ps |
| 1287 |
--forest``. |
| 1288 |
2. The most likely process to be hung is the connection multiplexer, which can |
| 1289 |
easily be identified as the parent of all SSH client processes. |
| 1290 |
3. Send ``kill -SEGV <pid>`` to the multiplexer PID, causing it to print all |
| 1291 |
thread stacks. |
| 1292 |
4. `File a bug <https://github.com/mitogen-hq/mitogen/issues/new/>`_ |
| 1293 |
including a copy of the stacks and a description of the last task executing |
| 1294 |
before the hang |
| 1295 |
|
| 1296 |
It is possible the hang occurred in a process on a target. If ``strace`` is |
| 1297 |
available, look for the host name not listed in Ansible output as reporting a |
| 1298 |
result for the most recent task, log into it, and use ``strace -ff -p <pid>`` |
| 1299 |
on each process whose name begins with ``mitogen:``:: |
| 1300 |
|
| 1301 |
$ strace -ff -p 29858 |
| 1302 |
strace: Process 29858 attached with 3 threads |
| 1303 |
[pid 29864] futex(0x55ea9be52f60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff <unfinished ...> |
| 1304 |
[pid 29860] restart_syscall(<... resuming interrupted poll ...> <unfinished ...> |
| 1305 |
[pid 29858] futex(0x55ea9be52f60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff |
| 1306 |
^C |
| 1307 |
|
| 1308 |
$ |
| 1309 |
|
| 1310 |
This shows one thread waiting on IO (``poll``) and two more waiting on the same |
| 1311 |
lock. It is taken from a real example of a deadlock due to a forking bug. |
| 1312 |
Please include any such information for all processes that you are able to |
| 1313 |
collect in any bug report. |
| 1314 |
|
| 1315 |
|
| 1316 |
Getting Help |
| 1317 |
~~~~~~~~~~~~ |
| 1318 |
Some users and developers hang out on the |
| 1319 |
`#mitogen <https://webchat.freenode.net/?channels=mitogen>`_ channel on the |
| 1320 |
FreeNode IRC network. |
| 1321 |
|
| 1322 |
|
| 1323 |
Sample Profiles |
| 1324 |
--------------- |
| 1325 |
|
| 1326 |
The summaries below may be reproduced using data and scripts maintained in the |
| 1327 |
`pcaps branch <https://github.com/mitogen-hq/mitogen/tree/pcaps/>`_. Traces were |
| 1328 |
recorded using Ansible 2.5.14. |
| 1329 |
|
| 1330 |
|
| 1331 |
Trivial Loop: Local Host |
| 1332 |
~~~~~~~~~~~~~~~~~~~~~~~~ |
| 1333 |
|
| 1334 |
This demonstrates Mitogen vs. SSH pipelining to the local machine running |
| 1335 |
`bench/loop-100-items.yml |
| 1336 |
<https://github.com/mitogen-hq/mitogen/blob/master/tests/ansible/bench/loop-100-items.yml>`_, |
| 1337 |
executing a simple command 100 times. Most Ansible controller overhead is |
| 1338 |
isolated, characterizing just module executor and connection layer performance. |
| 1339 |
Mitogen requires **63x less bandwidth and 5.9x less time**. |
| 1340 |
|
| 1341 |
.. image:: images/ansible/pcaps/loop-100-items-local.svg |
| 1342 |
|
| 1343 |
Unlike in SSH pipelining where payloads are sent as a single compressed block, |
| 1344 |
by default Mitogen enables SSH compression for its uncompressed RPC data. In |
| 1345 |
many-host scenarios it may be desirable to disable compression. This has |
| 1346 |
negligible impact on footprint, since program code is separately compressed and |
| 1347 |
sent only once. Compression also benefits SSH pipelining, but the presence of |
| 1348 |
large precompressed per-task payloads may present a more significant CPU burden |
| 1349 |
during many-host runs. |
| 1350 |
|
| 1351 |
.. image:: images/ansible/pcaps/loop-100-items-local-detail.svg |
| 1352 |
|
| 1353 |
In a detailed trace, improved interaction with the host machine is visible. In |
| 1354 |
this playbook because no forks were required to start SSH clients from the |
| 1355 |
worker process executing the loop, the worker's memory was never marked |
| 1356 |
read-only, thus avoiding a major hidden performance problem - the page fault |
| 1357 |
rate is more than halved. |
| 1358 |
|
| 1359 |
|
| 1360 |
File Transfer: UK to France |
| 1361 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 1362 |
|
| 1363 |
`This playbook |
| 1364 |
<https://github.com/mitogen-hq/mitogen/blob/master/tests/ansible/regression/issue_140__thread_pileup.yml>`_ |
| 1365 |
was used to compare file transfer performance over a ~26 ms link. It uses the |
| 1366 |
``with_filetree`` loop syntax to copy a directory of 1,000 0-byte files to the |
| 1367 |
target. |
| 1368 |
|
| 1369 |
.. raw:: html |
| 1370 |
|
| 1371 |
<style> |
| 1372 |
.nojunk td, |
| 1373 |
.nojunk th { padding: 4px; font-size: 90%; text-align: right !important; } |
| 1374 |
|
| 1375 |
table.docutils col { |
| 1376 |
width: auto !important; |
| 1377 |
} |
| 1378 |
</style> |
| 1379 |
|
| 1380 |
.. csv-table:: |
| 1381 |
:header: , Secs, CPU Secs, Sent, Received, Roundtrips |
| 1382 |
:class: nojunk |
| 1383 |
:align: right |
| 1384 |
|
| 1385 |
Mitogen, 98.54, 43.04, "815 KiB", "447 KiB", 3.79 |
| 1386 |
SSH Pipelining, "1,483.54", 329.37, "99,539 KiB", "6,870 KiB", 57.01 |
| 1387 |
|
| 1388 |
*Roundtrips* is the approximate number of network roundtrips required to |
| 1389 |
describe the runtime that was consumed. Due to Mitogen's built-in file transfer |
| 1390 |
support, continuous reinitialization of an external `scp`/`sftp` client is |
| 1391 |
avoided, permitting large ``with_filetree`` copies to become practical without |
| 1392 |
any special casing within the playbook or the Ansible implementation. |
| 1393 |
|
| 1394 |
|
| 1395 |
DebOps: UK to India |
| 1396 |
~~~~~~~~~~~~~~~~~~~ |
| 1397 |
|
| 1398 |
This is an all-green run of 246 tasks from the `DebOps |
| 1399 |
<https://docs.debops.org/en/master/>`_ 0.7.2 `common.yml |
| 1400 |
<https://github.com/debops/debops-playbooks/blob/master/playbooks/common.yml>`_ |
| 1401 |
playbook over a ~370 ms link between the UK and India. The playbook touches a |
| 1402 |
wide variety of modules, many featuring unavoidable waits for slow computation |
| 1403 |
on the target. |
| 1404 |
|
| 1405 |
More tasks of a wider variety are featured than previously, placing strain on |
| 1406 |
Mitogen's module loading and in-memory caching. By running over a long-distance |
| 1407 |
connection, it highlights behaviour of the connection layer in the presence of |
| 1408 |
high latency. |
| 1409 |
|
| 1410 |
Mitogen requires **14.5x less bandwidth and 4x less time**. |
| 1411 |
|
| 1412 |
.. image:: images/ansible/pcaps/debops-uk-india.svg |
| 1413 |
|
| 1414 |
|
| 1415 |
Django App: UK to India |
| 1416 |
~~~~~~~~~~~~~~~~~~~~~~~ |
| 1417 |
|
| 1418 |
This short playbook features only 23 steps executed over the same ~370 ms link |
| 1419 |
as previously, with many steps running unavoidably expensive tasks like |
| 1420 |
building C++ code, and compiling static web site assets. |
| 1421 |
|
| 1422 |
Despite the small margin for optimization, Mitogen still manages **6.2x less |
| 1423 |
bandwidth and 1.8x less time**. |
| 1424 |
|
| 1425 |
.. image:: images/ansible/pcaps/costapp-uk-india.svg |