Debian Codegraph @

vignettes/parallelly-12-remote-workers.md

raw
1 <!--
2 %\VignetteIndexEntry{Parallel Workers on Other Machines}
3 %\VignetteAuthor{Henrik Bengtsson}
4 %\VignetteKeyword{R}
5 %\VignetteKeyword{package}
6 %\VignetteKeyword{vignette}
7 %\VignetteEngine{parallelly::selfonly}
8 -->
9
10
11 # Introduction
12
13 Sometimes it is not sufficient to parallelize on a single computer -
14 it cannot provide all of the compute power we are looking for. When we
15 hit this limit, a natural next level is to look at other computers
16 near us, e.g. desktops in an office or other computers we have access
17 to remotely. In this vignette, we will cover how to run parallel R
18 workers on other machines. Sometimes we distinguish between local
19 machines and on remote machines, where _local_ machines are machines
20 considered to be on the same local area network (LAN) and that might
21 share a common file system. _Remote_ machines are machines that are on
22 a different network and that do not share a common file system with
23 the main R computer. In most cases the distinction between local and
24 remote machines does not matter, but in some cases we can take
25 advantage of workers being local.
26
27 Regardless of running parallel workers on local or remote machines, we
28 need a way to connect to the machines and launch R on them.
29
30
31 ## SSH and R configuration (once)
32
33 ### Verifying SSH access
34
35 The most common approach to connect to another machine is via Secure
36 Shell (SSH). Linux, macOS, and MS Windows all have a built-in SSH
37 client called `ssh`. Consider we have another Linux machine called
38 `n1.remote.org`, it can be accessed via SSH, and we have an account
39 `alice` on that machine. For the case of these instructions, it does
40 not matter whether `n1.remote.org` is on our local network (LAN) or a
41 remote machine on the internet. Also, to make it clear that we do not have to have the same username on `n1.remote.org` and on our local machine, we will use `ally` as the username on our local machine.
42
43 To access the `alice` user account on `n1.remote.org` from our local
44 computer, we open a terminal on the local computer and then SSH to the
45 other machine as:
46
47 ```sh
48 {ally@local}$ ssh [email protected]
49 [email protected]'s password: *************
50 {alice@n1}$
51 ```
52
53 The commands to call are what follows after the prompt. The prompt on
54 our local machine is `{ally@local}$`, which tells us that our username
55 is `ally` and the name of the local machine is `local`. The prompt on
56 the `n1.remote.org` machine is `{alice@n1}$`, which tells us that our
57 username on that machine is `alice` and that the machine is called
58 `n1` on that system.
59
60 To return to our local machine, exit the SSH shell by typing `exit`;
61
62 ```sh
63 {alice@n1}$ exit
64 {ally@local}$
65 ```
66
67 If we get this far, we have confirmed that we have SSH access to this
68 machine.
69
70
71 ### Configure password-less SSH access
72
73 Launching parallel R workers is typically done automatically in the
74 background, which means it is cumbersome, or even impossible, to enter
75 the SSH password for each machine we wish to connect to. The solution
76 is to configure SSH to connect with _public-private keys_, which
77 pre-establish SSH authentication between the main machine and the
78 machine to connect to. As this is common practice when working with
79 SSH, there are numerous online tutorials explaining how to configure
80 private-public SSH key pairs. Please consult one of them for the
81 details, but the gist is to use (i) `ssh-keygen` to generate the
82 public-private SSH keys on your local machine, and then (ii)
83 `ssh-copy-id` to deploy the public key on the machine you want to
84 connect to.
85
86 Step 1: Generate public-private SSH keys locally
87
88 ```sh
89 {ally@local}$ ssh-keygen
90 Generating public/private rsa key pair.
91 Enter file in which to save the key (/home/ally/.ssh/id_rsa):
92 Created directory '/home/ally/.ssh'.
93 Enter passphrase (empty for no passphrase):
94 Enter same passphrase again:
95 Your identification has been saved in /home/ally/.ssh/id_rsa
96 Your public key has been saved in /home/ally/.ssh/id_rsa.pub
97 The key fingerprint is:
98 SHA256:Sx48uXZTUL12SKKUzWB77e/Pm3TifqrDIbOnJ0pEWHY ally@local
99 The key's randomart image is:
100 +---[RSA 3072]----+
101 | o E=.. |
102 | + ooo+.o |
103 | . ..o..o.o |
104 | o ..o .+ .|
105 | S .... |
106 | + =o.. . |
107 | * o= ...o|
108 | o .o.=..++|
109 | ...=.++=*|
110 +----[SHA256]-----+
111 ```
112
113
114 Step 2: Copy the public SSH key to the other machine
115
116 ```sh
117 {ally@local}$ ssh-copy-id [email protected]
118 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/ally/.ssh/id_rsa.pub"
119 /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
120 /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
121 [email protected]:s password: *************
122
123 Number of key(s) added: 1
124
125 Now try logging into the machine, with: "ssh '[email protected]'"
126 and check to make sure that only the key(s) you wanted were added.
127 ```
128
129 At this point, we should be able to SSH to the other machine without
130 having to enter a password;
131
132 ```sh
133 {ally@local}$ ssh [email protected]
134 {alice@n1}$
135 ```
136
137 Type `exit` to return to your local machine.
138
139 Note, if you later want to connect to other machines,
140 e.g. `n2.remote.org` or `hpc.my-university.edu`, you may re-use the
141 above generated keys for those systems too. In other words, you do not
142 have to use `ssh-keygen` to generate new keys for those machines.
143
144
145 ### Verifying R exists on the other machine
146
147 In order to run parallel R workers on another machine, it (i) needs to
148 be installed on that machine, and (ii) ideally readily available by
149 calling `Rscript`. Parallel R workers are launched via `Rscript`,
150 instead of the more commonly known `R` command - both come with all R
151 installation, i.e. if you have one of them, you have the other too.
152
153 To verify that R is installed on the other machine, SSH to the machine and call `Rscript --version`;
154
155 ```sh
156 {ally@local}$ ssh [email protected]
157 {alice@n1}$ Rscript --version
158 Rscript (R) version 4.5.2 (2025-10-31)
159 ```
160
161 If you get:
162
163 ```sh
164 {alice@n1}$ Rscript --version
165 Rscript: command not found
166 ```
167
168 then R is either not installed on that machine, or it cannot be
169 found. If it is installed, but cannot be found, make sure that
170 environment variable `PATH` is configured properly on that machine.
171
172
173 ### Final checks
174
175 With password-less SSH access, and R being available, on the other
176 machine, we should be able to SSH into the other machine and query the
177 R version in a single call:
178
179 ```sh
180 {ally@local}$ ssh [email protected] Rscript --version
181 Rscript (R) version 4.5.2 (2025-10-31)
182 {ally@local}$
183 ```
184
185 This is all that is needed to launch one or more parallel R workers on
186 machine `n1.remote.org` running under user `alice`. We can test this
187 from within R with the **parallelly** package using:
188
189 ```sh
190 {ally@local}$ R --quiet
191 library(parallelly)
192 cl <- makeClusterPSOCK("n1.remote.org", user = "alice")
193 print(cl)
194 #> Socket cluster with 1 node on host 'n1.remote.org' (R version 4.5.2
195 #> (2025-10-31), platform x86_64-pc-linux-gnu)
196 parallel::stopCluster(cl)
197 ```
198
199 If you want to run parallel workers on other machines, repeat the
200 above for each machine. After this, you will be able to launch
201 parallel R workers on these machines with little effort.
202
203
204 ### Machine-specific SSH customization (recommended)
205
206 Some machines do not use the default port 22 to answer on SSH
207 connection requests. If the machine uses another port, say, port 2201,
208 then we can specify that via option `-p port`, when we connect to it,
209 e.g.
210
211 ```sh
212 {ally@local}$ ssh -p 2201 [email protected]
213 ```
214
215 In R, we can specify argument `port=port` as in:
216
217 ```sh
218 cl <- makeClusterPSOCK("n1.remote.org", port = 2201, user = "alice")
219 ```
220
221 Now, it can be tedious to have to remember custom SSH ports and
222 usernames when setting up remote workers in R. It also adds noise and
223 distraction having such details in the R script, and not to mention
224 the fact that the R script has a specific username hardcoded into the
225 code makes the script less reproducible for other users - they need to
226 change the code to match their username. One way to avoid having to
227 give specific SSH options when calling `ssh` in the terminal, or
228 `makeClusterPSOCK()` in R, is to configure these settings in SSH. This
229 can be done via a file called `~/.ssh/config` _on your local
230 machine_. This file does not exist by default, so you would have to
231 create it yourself, if missing. It is a plain text file, so you should
232 use a plain text editor to create and edit it.
233
234 To configure SSH to use port 2201 and username `alice` whenever
235 connecting to `n1.remote.org`, the `~/.ssh/config` file should
236 contain the following entry:
237
238 ```plain
239 Host n1.remote.org
240 User alice
241 Port 2201
242 ```
243
244 With this, we can connect to `n1.remote.org` by just using:
245
246 ```sh
247 {ally@local}$ ssh n1.remote.org
248 {alice@n1}$
249 ```
250
251 SSH will then connect to the machine as if we had specified also `-p
252 2201` and `-l alice`. These settings will also be picked up when we
253 connect via R, meaning the following will also work:
254
255 ```sh
256 cl <- makeClusterPSOCK("n1.remote.org")
257 ```
258
259 To achieve the same for other machines, add another entry for them,
260 e.g.
261
262 ```plain
263 Host n1.remote.org
264 User alice
265 Port 2201
266
267 Host n2.remote.org
268 User alice
269 Port 2201
270
271 Host hpc.my-university.edu
272 User alice.bobson
273 ```
274
275 When hosts on the same system share the same setting, one can use
276 globbing to configure them the same way. For instance, the above can
277 be shorted to:
278
279 ```plain
280 Host n?.remote.org
281 User alice
282 Port 2201
283
284 Host hpc.my-university.edu
285 User alice.bobson
286 ```
287
288 Being able to connect to remote machines by just specifying their
289 hostnames is convenient and simplifies also the R code. Because of
290 this, we recommend setting up also `~/.ssh/config`.
291
292
293 # Examples
294
295 ## Example: Two parallel workers on a single remote machine
296
297 Our first example sets up two parallel workers on the remote machine
298 `n1.remote.org`. For this to work, we need SSH access to the machine,
299 and it must have R installed, as explained in the above
300 section. Contrary to local parallel workers, the number of parallel
301 workers on remote machines is specified by repeating the machine name
302 an equal number of times;
303
304 ```r
305 library(parallelly)
306 workers <- c("n1.remote.org", "n1.remote.org")
307 cl <- makeClusterPSOCK(workers, user = "alice")
308 print(cl)
309 #> Socket cluster with 2 nodes are on host 'n1.remote.org' (R version 4.5.2
310 #> (2025-10-31), platform x86_64-pc-linux-gnu)
311 ```
312
313 _Comment_: In the **parallel** package, a parallel worker is referred
314 to a parallel node, or short _node_, which is why we use the same term
315 in the **parallelly** package.
316
317 Note, contrary to parallel workers running on the local machine,
318 parallel workers on remote machines are launched sequentially, that is
319 one after each other. Because of this, the setup time for a remote
320 parallel cluster will increase linearly with the number of remote
321 parallel workers.
322
323 _Technical details_: If we would add `verbose = TRUE` to
324 `makeClusterPSOCK()`, we would learn that the parallel workers are
325 launched in the background by R using something like:
326
327 ```
328 '/usr/bin/ssh' -R 11058:localhost:11058 -l alice n1.remote.org Rscript ...
329 '/usr/bin/ssh' -R 11059:localhost:11059 -l alice n1.remote.org Rscript ...
330 ```
331
332 This tells us that there is one active SSH connection per parallel
333 worker. It also reveals that that each of these connections uses a so
334 called _reverse tunnel_, which is used to establish a unique
335 communication channel between the main R process and the corresponding
336 parallel worker. It also this use of reverse tunneling that avoids
337 having to configure dynamic DNS (DDNS) and port-forwarding in our
338 local firewalls, which is cumbersome and requires administrative
339 rights. When using **parallelly**, there is no need for administrative
340 rights - any non-privileged user can launch remote parallel R workers.
341
342
343 ## Example: Two parallel workers on two remote machines
344
345 This example sets up a parallel worker on each of two remote machines
346 `n1.remote.org` and `n2.remote.org`. It works very similar to the
347 previous example, but now the two SSH connections go to two different
348 machines rather than the same.
349
350 ```r
351 library(parallelly)
352 workers <- c("n1.remote.org", "n2.remote.org")
353 cl <- makeClusterPSOCK(workers, user = "alice")
354 print(cl)
355 #> Socket cluster with 2 nodes where 1 node is on host 'n1.remote.org'
356 #> (R version 4.5.2 (2025-10-31), platform x86_64-pc-linux-gnu)
357 #> 1 node is on host 'n2.remote.org' (R version 4.5.2 (2025-10-31),
358 #> platform x86_64-pc-linux-gnu)
359 ```
360
361 _Technical details_: If we would add `verbose = TRUE` also in this
362 case, we would see:
363
364 ```
365 '/usr/bin/ssh' -R 11464:localhost:11464 -l alice n1.remote.org Rscript ...
366 '/usr/bin/ssh' -R 11465:localhost:11464 -l alice n2.remote.org Rscript ...
367 ```
368
369 Recall, if we have configured SSH to pick up the username `alice` from
370 `~/.ssh/config` on our local machine, as shown in the previous
371 section, we could have skipped the `user` argument, and just used:
372
373 ```r
374 workers <- c("n1.remote.org", "n2.remote.org")
375 cl <- makeClusterPSOCK(workers)
376 ```
377
378 Note how these instructions for setting up a parallel cluster on these
379 two machines would be the identical for another user that has
380 configured their personal `~/.ssh/config` file.
381
382
383 ## Example: Three parallel workers on two remote machines
384
385 When we now understand that we control the number of parallel workers
386 on a specific machine by replicate the machine name, we also know how
387 to launch different number of parallel workers on different machines.
388 From now on, we will also assume that the remote username no longer
389 has to be specified, because it has already been configured via the
390 `~/.ssh/config` file. With this, we can sets up two parallel workers
391 on `n1.remote.org` and one on `n2.remote.org`, by:
392
393 ```r
394 library(parallelly)
395 workers <- c("n1.remote.org", "n1.remote.org", "n2.remote.org")
396 cl <- makeClusterPSOCK(workers)
397 print(cl)
398 #> Socket cluster with 3 nodes where 2 nodes are on host 'n1.remote.org'
399 #> (R version 4.5.2 (2025-10-31), platform x86_64-pc-linux-gnu)
400 #> 1 node is on host 'n2.remote.org' (R version 4.5.2 (2025-10-31),
401 #> platform x86_64-pc-linux-gnu)
402 ```
403
404 Again, the `user` argument does not have to be specified, because it is configured in `~/.ssh/config`.
405
406 To generalize to many workers, we can use the `rep()` function. For example,
407
408 ```r
409 workers <- c(rep("n1.remote.org", 3), rep("n2.remote.org", 4))
410 ```
411
412 sets up three workers on `n1.remote.org` and four on `n2.remote.org`,
413 totaling seven parallel workers.
414
415
416 ## Example: A mix of local and remote workers
417
418 As an alternative to `makeClusterPSOCK(n)`, we can use
419 `makeClusterPSOCK(workers)` to set up parallelly workers running on
420 the local machine. By convention, the name `localhost` is an alias to
421 your local machine. This means, we can use:
422
423 ```sh
424 library(parallelly)
425 workers <- rep("localhost", 4)
426 cl_local <- makeClusterPSOCK(workers)
427 print(cl_local)
428 #> Socket cluster with 4 nodes on host 'localhost' (R version 4.5.2
429 #> (2025-10-31), platform x86_64-pc-linux-gnu)
430 ```
431
432 to launch four local parallel workers. Note how we did not have to
433 specify `user = "ally"`. This is because the default username is
434 always the local username. Next, assume we want to add another four
435 parallel workers running on `n1.remote.org`. We already know we can
436 set these up as:
437
438 ```sh
439 library(parallelly)
440 workers <- rep("n1.remote.org", 4)
441 cl_remote <- makeClusterPSOCK(workers, user = "alice")
442 print(cl_remote)
443 #> Socket cluster with 4 nodes on host 'n1.remote.org' (R version 4.5.2
444 #> (2025-10-31), platform x86_64-pc-linux-gnu)
445 ```
446
447 At this point, we have two independent clusters of parallel workers:
448 `cl_local` and `cl_remote`. We can combine them into a single
449 cluster using:
450
451 ```r
452 cl <- c(cl_local, cl_remote)
453 print(cl)
454 #> Socket cluster with 8 nodes where 4 nodes are on host 'localhost'
455 #> (R version 4.5.2 (2025-10-31), platform x86_64-pc-linux-gnu)
456 #> nodes are on host 'n1.remote.org' (R version 4.5.2 (2025-10-31),
457 #> platform x86_64-pc-linux-gnu)
458 ```
459
460 To emphasize the usefulness of customizing our SSH connections via
461 `~/.ssh/config`, if the remote username would already have been
462 already configured there, we would be able to set up the full cluster
463 in one single call, as in:
464
465 ```sh
466 library(parallelly)
467 workers <- c(rep("localhost", 4), rep("n1.remote.org", 4)
468 cl <- makeClusterPSOCK(workers)
469 ```
470
471
472 ## Example: Parallel workers on a remote machine accessed via dedicated login machine
473
474 Sometimes a remote machine, where we want to run R, is only accessible
475 via an intermediate login machine, which in SSH terms may also be
476 referred to as a "jumphost". For example, assume machine
477 `secret1.remote.org` can only be accessed by first logging into
478 `login.remote.org` as in:
479
480 ```sh
481 {ally@local}$ ssh [email protected]
482 {alice@login}$ ssh [email protected]
483 {alice@secret1}$
484 ```
485
486 To achieve the same in a single SSH call, we can specify the
487 "jumphost" `-J hostname` option for SSH, as in:
488
489 ```sh
490 {ally@local}$ ssh -J [email protected] [email protected]
491 {alice@secret1}$
492 ```
493
494 We can use the `rshopts` argument of `makeClusterPSOCK()` to achieve
495 the same when setting up parallel workers. To launch three parallel
496 workers on `secret1.remote.org`, use:
497
498 ```r
499 workers <- rep("secret1.remote.org", 3)
500 cl <- makeClusterPSOCK(
501 workers,
502 rshopts = c("-J", "login.remote.org"),
503 user = "alice"
504 )
505 ```
506
507 A more convenient solution is to configure the jumphost in
508 `~/.ssh/config`, as in:
509
510 ```plain
511 Host *.remote.org
512 User alice
513
514 Host secret?.remote.org
515 ProxyJump login.remote.org
516 ```
517
518 This will cause any SSH connection to a machine on the `remote.org`
519 network to use username `alice`. It will also cause any SSH
520 connection to machines `secret1.remote.org`, `secret2.remote.org`, and
521 so on, to use jumphost `login.remote.org`. You can verify that all
522 this work by:
523
524 ```sh
525 {ally@local}$ ssh login.remote.org
526 {alice@login}$
527 ```
528
529 and then:
530
531 ```sh
532 {ally@local}$ ssh secret1.remote.org
533 {alice@secret1}$
534 ```
535
536 If the above work, then the following will work from within R:
537
538 ```r
539 library(parallelly)
540 workers <- rep("secret1.remote.org", 3)
541 cl <- makeClusterPSOCK(workers)
542 ```
543
544
545 # Special needs and tweaks
546
547 The above sections cover most common use cases for setting up a
548 parallel cluster from a local Linux, macOS, and MS Windows
549 machine. However, there are cases where there above does not work, or
550 you prefer to use another solution. This section aims to cover such
551 alternatives.
552
553
554 ## Example: Remote workers ignoring any remote .Rprofile settings
555
556 To launch parallel workers skipping any `~/.Rprofile` settings on the
557 remote machines, we can pass option `--no-init-file` to `Rscript` via
558 argument `rscript_args`. For example,
559
560 ```r
561 workers <- rep("n1.remote.org", 2)
562 cl <- makeClusterPSOCK(workers, rscript_args = "--no-init-file")
563 ```
564
565 will launch two parallel workers on `n1.remote.org` ignoring any
566 `.Rprofile` files.
567
568
569 ## Example: Use PuTTY on MS Windows to connect to remote worker
570
571 If you run on an MS Windows machine and prefer to use PuTTY to manage
572 your SSH connections, or for other reasons cannot use the built-in
573 `ssh` client, you can tell `makeClusterPSOCK()` to use PuTTY and your
574 PuTTY settings via various arguments.
575
576 Here is an example that launches two parallel workers on
577 `n1.remote.org` running under user `alice` connecting via SSH port
578 2201 using PuTTY and public-private SSH keys in file
579 `C:/Users/ally/.ssh/putty.ppk`:
580
581 ```r
582 workers <- "n1.remote.org"
583 cl <- makeClusterPSOCK(
584 workers,
585 user = "alice",
586 rshcmd = "<putty-plink>",
587 rshopts = c("-P", 2201, "-i", "C:/Users/ally/.ssh/putty.ppk")
588 )
589 ```
590
591
592 ## Example: Two remote workers running on MS Windows
593
594 Thus far we have considered our remote machines to run a Unix-like
595 operating system, e.g. Linux or macOS. If your remote machines run MS
596 Windows, you can use similar techniques to launch parallel workers
597 there as well. For this to work, the remote MS Windows machines must
598 accept incoming SSH connections, which is something most Windows
599 machines are not configured to do by default. If you do not know how to set
600 that up, or if you do not have the system permissions to do so, please
601 reach out to your system administrator of those machines.
602
603 Assuming we have SSH access to two MS Windows machines,
604 `mswin1.remote.org` and `mswin2.remote.org`, everything works the same
605 as before, except that we need to specify also argument `rscript_sh =
606 "cmd"`;
607
608 ```r
609 workers <- c("mswin1.remote.org", "mswin2.remote.org")
610 cl <- makeClusterPSOCK(workers, rscript_sh = "cmd")
611 ```
612
613 That argument specifies that the parallel R workers should be launched
614 on the remote machines via MS Windows' `cmd.exe` shell.