Debian Codegraph @

docs/src/cdist-parallelization.rst

raw
1 Parallelization
2 ===============
3
4 Description
5 -----------
6 cdist has two modes of parallel operation.
7
8 One of them is to operate on each host in separate process. This is enabled
9 with :strong:`-p/--parallel` option.
10
11 The other way is to operate in parallel within one host where you specify
12 the number of jobs. This is enabled with :strong:`-j/--jobs` option where you
13 can specify the number of parallel jobs. By default,
14 :strong:`multiprocessing.cpu_count()` is used. For this mode global explorers,
15 object preparation and object run are supported.
16
17 You can, of course, use those two options together. This means that each host
18 will be processed by its own process. Within each process cdist will operate
19 using specified number of parallel jobs.
20
21 For more info on those options see :strong:`cdist`\ (1).
22
23
24 Examples
25 --------
26
27 .. code-block:: sh
28
29 # Configure hosts read from file hosts.file in parallel
30 $ cdist config -p -f hosts.file
31
32 # Configure hosts read from file hosts.file sequentially but using default
33 # number of parallel jobs
34 $ cdist config -j -f hosts.file
35
36 # Configure hosts read from file hosts.file in parallel using 16
37 # parallel jobs
38 $ cdist config -j 16 -p -f hosts.file
39
40
41 Caveats
42 -------
43 When operating in parallel, either by operating in parallel for each host
44 (-p/--parallel) or by parallel jobs within a host (-j/--jobs), and depending
45 on target SSH server and its configuration you may encounter connection drops.
46 This is controlled with sshd :strong:MaxStartups configuration options.
47 You may also encounter session open refusal. This happens with ssh multiplexing
48 when you reach maximum number of open sessions permitted per network
49 connection. In this case ssh will disable multiplexing.
50 This limit is controlled with sshd :strong:MaxSessions configuration
51 options. For more details refer to :strong:`sshd_config`\ (5).
52
53 For example, if you reach :strong:`MaxSessions` sessions you may get the
54 following output:
55
56 .. code-block:: sh
57
58 $ cdist config -b -j 11 -v 78.47.116.244
59 INFO: cdist: version 4.2.2-55-g640b7f9
60 INFO: 78.47.116.244: Running global explorers
61 INFO: 78.47.116.244: Remote transfer in 11 parallel jobs
62 channel 22: open failed: administratively prohibited: open failed
63 mux_client_request_session: session request failed: Session open refused by peer
64 ControlSocket /tmp/tmpuah6fw_t/d886d4b7e4425a102a54bfaff4d2288b/ssh-control-path already exists, disabling multiplexing
65 INFO: 78.47.116.244: Running global explorers in 11 parallel jobs
66 channel 22: open failed: administratively prohibited: open failed
67 mux_client_request_session: session request failed: Session open refused by peer
68 ControlSocket /tmp/tmpuah6fw_t/d886d4b7e4425a102a54bfaff4d2288b/ssh-control-path already exists, disabling multiplexing
69 INFO: 78.47.116.244: Running initial manifest /tmp/tmpuah6fw_t/d886d4b7e4425a102a54bfaff4d2288b/data/conf/manifest/init
70 INFO: 78.47.116.244: Running manifest and explorers for __file/root/host.file
71 INFO: 78.47.116.244: Generating code for __file/root/host.file
72 INFO: 78.47.116.244: Finished successful run in 18.655028820037842 seconds
73 INFO: cdist: Total processing time for 1 host(s): 19.159148693084717