| 1 |
Parallelization |
| 2 |
=============== |
| 3 |
|
| 4 |
Description |
| 5 |
----------- |
| 6 |
cdist has two modes of parallel operation. |
| 7 |
|
| 8 |
One of them is to operate on each host in separate process. This is enabled |
| 9 |
with :strong:`-p/--parallel` option. |
| 10 |
|
| 11 |
The other way is to operate in parallel within one host where you specify |
| 12 |
the number of jobs. This is enabled with :strong:`-j/--jobs` option where you |
| 13 |
can specify the number of parallel jobs. By default, |
| 14 |
:strong:`multiprocessing.cpu_count()` is used. For this mode global explorers, |
| 15 |
object preparation and object run are supported. |
| 16 |
|
| 17 |
You can, of course, use those two options together. This means that each host |
| 18 |
will be processed by its own process. Within each process cdist will operate |
| 19 |
using specified number of parallel jobs. |
| 20 |
|
| 21 |
For more info on those options see :strong:`cdist`\ (1). |
| 22 |
|
| 23 |
|
| 24 |
Examples |
| 25 |
-------- |
| 26 |
|
| 27 |
.. code-block:: sh |
| 28 |
|
| 29 |
# Configure hosts read from file hosts.file in parallel |
| 30 |
$ cdist config -p -f hosts.file |
| 31 |
|
| 32 |
# Configure hosts read from file hosts.file sequentially but using default |
| 33 |
# number of parallel jobs |
| 34 |
$ cdist config -j -f hosts.file |
| 35 |
|
| 36 |
# Configure hosts read from file hosts.file in parallel using 16 |
| 37 |
# parallel jobs |
| 38 |
$ cdist config -j 16 -p -f hosts.file |
| 39 |
|
| 40 |
|
| 41 |
Caveats |
| 42 |
------- |
| 43 |
When operating in parallel, either by operating in parallel for each host |
| 44 |
(-p/--parallel) or by parallel jobs within a host (-j/--jobs), and depending |
| 45 |
on target SSH server and its configuration you may encounter connection drops. |
| 46 |
This is controlled with sshd :strong:MaxStartups configuration options. |
| 47 |
You may also encounter session open refusal. This happens with ssh multiplexing |
| 48 |
when you reach maximum number of open sessions permitted per network |
| 49 |
connection. In this case ssh will disable multiplexing. |
| 50 |
This limit is controlled with sshd :strong:MaxSessions configuration |
| 51 |
options. For more details refer to :strong:`sshd_config`\ (5). |
| 52 |
|
| 53 |
For example, if you reach :strong:`MaxSessions` sessions you may get the |
| 54 |
following output: |
| 55 |
|
| 56 |
.. code-block:: sh |
| 57 |
|
| 58 |
$ cdist config -b -j 11 -v 78.47.116.244 |
| 59 |
INFO: cdist: version 4.2.2-55-g640b7f9 |
| 60 |
INFO: 78.47.116.244: Running global explorers |
| 61 |
INFO: 78.47.116.244: Remote transfer in 11 parallel jobs |
| 62 |
channel 22: open failed: administratively prohibited: open failed |
| 63 |
mux_client_request_session: session request failed: Session open refused by peer |
| 64 |
ControlSocket /tmp/tmpuah6fw_t/d886d4b7e4425a102a54bfaff4d2288b/ssh-control-path already exists, disabling multiplexing |
| 65 |
INFO: 78.47.116.244: Running global explorers in 11 parallel jobs |
| 66 |
channel 22: open failed: administratively prohibited: open failed |
| 67 |
mux_client_request_session: session request failed: Session open refused by peer |
| 68 |
ControlSocket /tmp/tmpuah6fw_t/d886d4b7e4425a102a54bfaff4d2288b/ssh-control-path already exists, disabling multiplexing |
| 69 |
INFO: 78.47.116.244: Running initial manifest /tmp/tmpuah6fw_t/d886d4b7e4425a102a54bfaff4d2288b/data/conf/manifest/init |
| 70 |
INFO: 78.47.116.244: Running manifest and explorers for __file/root/host.file |
| 71 |
INFO: 78.47.116.244: Generating code for __file/root/host.file |
| 72 |
INFO: 78.47.116.244: Finished successful run in 18.655028820037842 seconds |
| 73 |
INFO: cdist: Total processing time for 1 host(s): 19.159148693084717 |