| 1 |
<!-- |
| 2 |
%\VignetteIndexEntry{Parallel Workers on Other Machines} |
| 3 |
%\VignetteAuthor{Henrik Bengtsson} |
| 4 |
%\VignetteKeyword{R} |
| 5 |
%\VignetteKeyword{package} |
| 6 |
%\VignetteKeyword{vignette} |
| 7 |
%\VignetteEngine{parallelly::selfonly} |
| 8 |
--> |
| 9 |
|
| 10 |
|
| 11 |
# Introduction |
| 12 |
|
| 13 |
Sometimes it is not sufficient to parallelize on a single computer - |
| 14 |
it cannot provide all of the compute power we are looking for. When we |
| 15 |
hit this limit, a natural next level is to look at other computers |
| 16 |
near us, e.g. desktops in an office or other computers we have access |
| 17 |
to remotely. In this vignette, we will cover how to run parallel R |
| 18 |
workers on other machines. Sometimes we distinguish between local |
| 19 |
machines and on remote machines, where _local_ machines are machines |
| 20 |
considered to be on the same local area network (LAN) and that might |
| 21 |
share a common file system. _Remote_ machines are machines that are on |
| 22 |
a different network and that do not share a common file system with |
| 23 |
the main R computer. In most cases the distinction between local and |
| 24 |
remote machines does not matter, but in some cases we can take |
| 25 |
advantage of workers being local. |
| 26 |
|
| 27 |
Regardless of running parallel workers on local or remote machines, we |
| 28 |
need a way to connect to the machines and launch R on them. |
| 29 |
|
| 30 |
|
| 31 |
## SSH and R configuration (once) |
| 32 |
|
| 33 |
### Verifying SSH access |
| 34 |
|
| 35 |
The most common approach to connect to another machine is via Secure |
| 36 |
Shell (SSH). Linux, macOS, and MS Windows all have a built-in SSH |
| 37 |
client called `ssh`. Consider we have another Linux machine called |
| 38 |
`n1.remote.org`, it can be accessed via SSH, and we have an account |
| 39 |
`alice` on that machine. For the case of these instructions, it does |
| 40 |
not matter whether `n1.remote.org` is on our local network (LAN) or a |
| 41 |
remote machine on the internet. Also, to make it clear that we do not have to have the same username on `n1.remote.org` and on our local machine, we will use `ally` as the username on our local machine. |
| 42 |
|
| 43 |
To access the `alice` user account on `n1.remote.org` from our local |
| 44 |
computer, we open a terminal on the local computer and then SSH to the |
| 45 |
other machine as: |
| 46 |
|
| 47 |
```sh |
| 48 |
{ally@local}$ ssh [email protected] |
| 49 |
[email protected]'s password: ************* |
| 50 |
{alice@n1}$ |
| 51 |
``` |
| 52 |
|
| 53 |
The commands to call are what follows after the prompt. The prompt on |
| 54 |
our local machine is `{ally@local}$`, which tells us that our username |
| 55 |
is `ally` and the name of the local machine is `local`. The prompt on |
| 56 |
the `n1.remote.org` machine is `{alice@n1}$`, which tells us that our |
| 57 |
username on that machine is `alice` and that the machine is called |
| 58 |
`n1` on that system. |
| 59 |
|
| 60 |
To return to our local machine, exit the SSH shell by typing `exit`; |
| 61 |
|
| 62 |
```sh |
| 63 |
{alice@n1}$ exit |
| 64 |
{ally@local}$ |
| 65 |
``` |
| 66 |
|
| 67 |
If we get this far, we have confirmed that we have SSH access to this |
| 68 |
machine. |
| 69 |
|
| 70 |
|
| 71 |
### Configure password-less SSH access |
| 72 |
|
| 73 |
Launching parallel R workers is typically done automatically in the |
| 74 |
background, which means it is cumbersome, or even impossible, to enter |
| 75 |
the SSH password for each machine we wish to connect to. The solution |
| 76 |
is to configure SSH to connect with _public-private keys_, which |
| 77 |
pre-establish SSH authentication between the main machine and the |
| 78 |
machine to connect to. As this is common practice when working with |
| 79 |
SSH, there are numerous online tutorials explaining how to configure |
| 80 |
private-public SSH key pairs. Please consult one of them for the |
| 81 |
details, but the gist is to use (i) `ssh-keygen` to generate the |
| 82 |
public-private SSH keys on your local machine, and then (ii) |
| 83 |
`ssh-copy-id` to deploy the public key on the machine you want to |
| 84 |
connect to. |
| 85 |
|
| 86 |
Step 1: Generate public-private SSH keys locally |
| 87 |
|
| 88 |
```sh |
| 89 |
{ally@local}$ ssh-keygen |
| 90 |
Generating public/private rsa key pair. |
| 91 |
Enter file in which to save the key (/home/ally/.ssh/id_rsa): |
| 92 |
Created directory '/home/ally/.ssh'. |
| 93 |
Enter passphrase (empty for no passphrase): |
| 94 |
Enter same passphrase again: |
| 95 |
Your identification has been saved in /home/ally/.ssh/id_rsa |
| 96 |
Your public key has been saved in /home/ally/.ssh/id_rsa.pub |
| 97 |
The key fingerprint is: |
| 98 |
SHA256:Sx48uXZTUL12SKKUzWB77e/Pm3TifqrDIbOnJ0pEWHY ally@local |
| 99 |
The key's randomart image is: |
| 100 |
+---[RSA 3072]----+ |
| 101 |
| o E=.. | |
| 102 |
| + ooo+.o | |
| 103 |
| . ..o..o.o | |
| 104 |
| o ..o .+ .| |
| 105 |
| S .... | |
| 106 |
| + =o.. . | |
| 107 |
| * o= ...o| |
| 108 |
| o .o.=..++| |
| 109 |
| ...=.++=*| |
| 110 |
+----[SHA256]-----+ |
| 111 |
``` |
| 112 |
|
| 113 |
|
| 114 |
Step 2: Copy the public SSH key to the other machine |
| 115 |
|
| 116 |
```sh |
| 117 |
{ally@local}$ ssh-copy-id [email protected] |
| 118 |
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/ally/.ssh/id_rsa.pub" |
| 119 |
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed |
| 120 |
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys |
| 121 |
[email protected]:s password: ************* |
| 122 |
|
| 123 |
Number of key(s) added: 1 |
| 124 |
|
| 125 |
Now try logging into the machine, with: "ssh '[email protected]'" |
| 126 |
and check to make sure that only the key(s) you wanted were added. |
| 127 |
``` |
| 128 |
|
| 129 |
At this point, we should be able to SSH to the other machine without |
| 130 |
having to enter a password; |
| 131 |
|
| 132 |
```sh |
| 133 |
{ally@local}$ ssh [email protected] |
| 134 |
{alice@n1}$ |
| 135 |
``` |
| 136 |
|
| 137 |
Type `exit` to return to your local machine. |
| 138 |
|
| 139 |
Note, if you later want to connect to other machines, |
| 140 |
e.g. `n2.remote.org` or `hpc.my-university.edu`, you may re-use the |
| 141 |
above generated keys for those systems too. In other words, you do not |
| 142 |
have to use `ssh-keygen` to generate new keys for those machines. |
| 143 |
|
| 144 |
|
| 145 |
### Verifying R exists on the other machine |
| 146 |
|
| 147 |
In order to run parallel R workers on another machine, it (i) needs to |
| 148 |
be installed on that machine, and (ii) ideally readily available by |
| 149 |
calling `Rscript`. Parallel R workers are launched via `Rscript`, |
| 150 |
instead of the more commonly known `R` command - both come with all R |
| 151 |
installation, i.e. if you have one of them, you have the other too. |
| 152 |
|
| 153 |
To verify that R is installed on the other machine, SSH to the machine and call `Rscript --version`; |
| 154 |
|
| 155 |
```sh |
| 156 |
{ally@local}$ ssh [email protected] |
| 157 |
{alice@n1}$ Rscript --version |
| 158 |
Rscript (R) version 4.5.2 (2025-10-31) |
| 159 |
``` |
| 160 |
|
| 161 |
If you get: |
| 162 |
|
| 163 |
```sh |
| 164 |
{alice@n1}$ Rscript --version |
| 165 |
Rscript: command not found |
| 166 |
``` |
| 167 |
|
| 168 |
then R is either not installed on that machine, or it cannot be |
| 169 |
found. If it is installed, but cannot be found, make sure that |
| 170 |
environment variable `PATH` is configured properly on that machine. |
| 171 |
|
| 172 |
|
| 173 |
### Final checks |
| 174 |
|
| 175 |
With password-less SSH access, and R being available, on the other |
| 176 |
machine, we should be able to SSH into the other machine and query the |
| 177 |
R version in a single call: |
| 178 |
|
| 179 |
```sh |
| 180 |
{ally@local}$ ssh [email protected] Rscript --version |
| 181 |
Rscript (R) version 4.5.2 (2025-10-31) |
| 182 |
{ally@local}$ |
| 183 |
``` |
| 184 |
|
| 185 |
This is all that is needed to launch one or more parallel R workers on |
| 186 |
machine `n1.remote.org` running under user `alice`. We can test this |
| 187 |
from within R with the **parallelly** package using: |
| 188 |
|
| 189 |
```sh |
| 190 |
{ally@local}$ R --quiet |
| 191 |
library(parallelly) |
| 192 |
cl <- makeClusterPSOCK("n1.remote.org", user = "alice") |
| 193 |
print(cl) |
| 194 |
#> Socket cluster with 1 node on host 'n1.remote.org' (R version 4.5.2 |
| 195 |
#> (2025-10-31), platform x86_64-pc-linux-gnu) |
| 196 |
parallel::stopCluster(cl) |
| 197 |
``` |
| 198 |
|
| 199 |
If you want to run parallel workers on other machines, repeat the |
| 200 |
above for each machine. After this, you will be able to launch |
| 201 |
parallel R workers on these machines with little effort. |
| 202 |
|
| 203 |
|
| 204 |
### Machine-specific SSH customization (recommended) |
| 205 |
|
| 206 |
Some machines do not use the default port 22 to answer on SSH |
| 207 |
connection requests. If the machine uses another port, say, port 2201, |
| 208 |
then we can specify that via option `-p port`, when we connect to it, |
| 209 |
e.g. |
| 210 |
|
| 211 |
```sh |
| 212 |
{ally@local}$ ssh -p 2201 [email protected] |
| 213 |
``` |
| 214 |
|
| 215 |
In R, we can specify argument `port=port` as in: |
| 216 |
|
| 217 |
```sh |
| 218 |
cl <- makeClusterPSOCK("n1.remote.org", port = 2201, user = "alice") |
| 219 |
``` |
| 220 |
|
| 221 |
Now, it can be tedious to have to remember custom SSH ports and |
| 222 |
usernames when setting up remote workers in R. It also adds noise and |
| 223 |
distraction having such details in the R script, and not to mention |
| 224 |
the fact that the R script has a specific username hardcoded into the |
| 225 |
code makes the script less reproducible for other users - they need to |
| 226 |
change the code to match their username. One way to avoid having to |
| 227 |
give specific SSH options when calling `ssh` in the terminal, or |
| 228 |
`makeClusterPSOCK()` in R, is to configure these settings in SSH. This |
| 229 |
can be done via a file called `~/.ssh/config` _on your local |
| 230 |
machine_. This file does not exist by default, so you would have to |
| 231 |
create it yourself, if missing. It is a plain text file, so you should |
| 232 |
use a plain text editor to create and edit it. |
| 233 |
|
| 234 |
To configure SSH to use port 2201 and username `alice` whenever |
| 235 |
connecting to `n1.remote.org`, the `~/.ssh/config` file should |
| 236 |
contain the following entry: |
| 237 |
|
| 238 |
```plain |
| 239 |
Host n1.remote.org |
| 240 |
User alice |
| 241 |
Port 2201 |
| 242 |
``` |
| 243 |
|
| 244 |
With this, we can connect to `n1.remote.org` by just using: |
| 245 |
|
| 246 |
```sh |
| 247 |
{ally@local}$ ssh n1.remote.org |
| 248 |
{alice@n1}$ |
| 249 |
``` |
| 250 |
|
| 251 |
SSH will then connect to the machine as if we had specified also `-p |
| 252 |
2201` and `-l alice`. These settings will also be picked up when we |
| 253 |
connect via R, meaning the following will also work: |
| 254 |
|
| 255 |
```sh |
| 256 |
cl <- makeClusterPSOCK("n1.remote.org") |
| 257 |
``` |
| 258 |
|
| 259 |
To achieve the same for other machines, add another entry for them, |
| 260 |
e.g. |
| 261 |
|
| 262 |
```plain |
| 263 |
Host n1.remote.org |
| 264 |
User alice |
| 265 |
Port 2201 |
| 266 |
|
| 267 |
Host n2.remote.org |
| 268 |
User alice |
| 269 |
Port 2201 |
| 270 |
|
| 271 |
Host hpc.my-university.edu |
| 272 |
User alice.bobson |
| 273 |
``` |
| 274 |
|
| 275 |
When hosts on the same system share the same setting, one can use |
| 276 |
globbing to configure them the same way. For instance, the above can |
| 277 |
be shorted to: |
| 278 |
|
| 279 |
```plain |
| 280 |
Host n?.remote.org |
| 281 |
User alice |
| 282 |
Port 2201 |
| 283 |
|
| 284 |
Host hpc.my-university.edu |
| 285 |
User alice.bobson |
| 286 |
``` |
| 287 |
|
| 288 |
Being able to connect to remote machines by just specifying their |
| 289 |
hostnames is convenient and simplifies also the R code. Because of |
| 290 |
this, we recommend setting up also `~/.ssh/config`. |
| 291 |
|
| 292 |
|
| 293 |
# Examples |
| 294 |
|
| 295 |
## Example: Two parallel workers on a single remote machine |
| 296 |
|
| 297 |
Our first example sets up two parallel workers on the remote machine |
| 298 |
`n1.remote.org`. For this to work, we need SSH access to the machine, |
| 299 |
and it must have R installed, as explained in the above |
| 300 |
section. Contrary to local parallel workers, the number of parallel |
| 301 |
workers on remote machines is specified by repeating the machine name |
| 302 |
an equal number of times; |
| 303 |
|
| 304 |
```r |
| 305 |
library(parallelly) |
| 306 |
workers <- c("n1.remote.org", "n1.remote.org") |
| 307 |
cl <- makeClusterPSOCK(workers, user = "alice") |
| 308 |
print(cl) |
| 309 |
#> Socket cluster with 2 nodes are on host 'n1.remote.org' (R version 4.5.2 |
| 310 |
#> (2025-10-31), platform x86_64-pc-linux-gnu) |
| 311 |
``` |
| 312 |
|
| 313 |
_Comment_: In the **parallel** package, a parallel worker is referred |
| 314 |
to a parallel node, or short _node_, which is why we use the same term |
| 315 |
in the **parallelly** package. |
| 316 |
|
| 317 |
Note, contrary to parallel workers running on the local machine, |
| 318 |
parallel workers on remote machines are launched sequentially, that is |
| 319 |
one after each other. Because of this, the setup time for a remote |
| 320 |
parallel cluster will increase linearly with the number of remote |
| 321 |
parallel workers. |
| 322 |
|
| 323 |
_Technical details_: If we would add `verbose = TRUE` to |
| 324 |
`makeClusterPSOCK()`, we would learn that the parallel workers are |
| 325 |
launched in the background by R using something like: |
| 326 |
|
| 327 |
``` |
| 328 |
'/usr/bin/ssh' -R 11058:localhost:11058 -l alice n1.remote.org Rscript ... |
| 329 |
'/usr/bin/ssh' -R 11059:localhost:11059 -l alice n1.remote.org Rscript ... |
| 330 |
``` |
| 331 |
|
| 332 |
This tells us that there is one active SSH connection per parallel |
| 333 |
worker. It also reveals that that each of these connections uses a so |
| 334 |
called _reverse tunnel_, which is used to establish a unique |
| 335 |
communication channel between the main R process and the corresponding |
| 336 |
parallel worker. It also this use of reverse tunneling that avoids |
| 337 |
having to configure dynamic DNS (DDNS) and port-forwarding in our |
| 338 |
local firewalls, which is cumbersome and requires administrative |
| 339 |
rights. When using **parallelly**, there is no need for administrative |
| 340 |
rights - any non-privileged user can launch remote parallel R workers. |
| 341 |
|
| 342 |
|
| 343 |
## Example: Two parallel workers on two remote machines |
| 344 |
|
| 345 |
This example sets up a parallel worker on each of two remote machines |
| 346 |
`n1.remote.org` and `n2.remote.org`. It works very similar to the |
| 347 |
previous example, but now the two SSH connections go to two different |
| 348 |
machines rather than the same. |
| 349 |
|
| 350 |
```r |
| 351 |
library(parallelly) |
| 352 |
workers <- c("n1.remote.org", "n2.remote.org") |
| 353 |
cl <- makeClusterPSOCK(workers, user = "alice") |
| 354 |
print(cl) |
| 355 |
#> Socket cluster with 2 nodes where 1 node is on host 'n1.remote.org' |
| 356 |
#> (R version 4.5.2 (2025-10-31), platform x86_64-pc-linux-gnu) |
| 357 |
#> 1 node is on host 'n2.remote.org' (R version 4.5.2 (2025-10-31), |
| 358 |
#> platform x86_64-pc-linux-gnu) |
| 359 |
``` |
| 360 |
|
| 361 |
_Technical details_: If we would add `verbose = TRUE` also in this |
| 362 |
case, we would see: |
| 363 |
|
| 364 |
``` |
| 365 |
'/usr/bin/ssh' -R 11464:localhost:11464 -l alice n1.remote.org Rscript ... |
| 366 |
'/usr/bin/ssh' -R 11465:localhost:11464 -l alice n2.remote.org Rscript ... |
| 367 |
``` |
| 368 |
|
| 369 |
Recall, if we have configured SSH to pick up the username `alice` from |
| 370 |
`~/.ssh/config` on our local machine, as shown in the previous |
| 371 |
section, we could have skipped the `user` argument, and just used: |
| 372 |
|
| 373 |
```r |
| 374 |
workers <- c("n1.remote.org", "n2.remote.org") |
| 375 |
cl <- makeClusterPSOCK(workers) |
| 376 |
``` |
| 377 |
|
| 378 |
Note how these instructions for setting up a parallel cluster on these |
| 379 |
two machines would be the identical for another user that has |
| 380 |
configured their personal `~/.ssh/config` file. |
| 381 |
|
| 382 |
|
| 383 |
## Example: Three parallel workers on two remote machines |
| 384 |
|
| 385 |
When we now understand that we control the number of parallel workers |
| 386 |
on a specific machine by replicate the machine name, we also know how |
| 387 |
to launch different number of parallel workers on different machines. |
| 388 |
From now on, we will also assume that the remote username no longer |
| 389 |
has to be specified, because it has already been configured via the |
| 390 |
`~/.ssh/config` file. With this, we can sets up two parallel workers |
| 391 |
on `n1.remote.org` and one on `n2.remote.org`, by: |
| 392 |
|
| 393 |
```r |
| 394 |
library(parallelly) |
| 395 |
workers <- c("n1.remote.org", "n1.remote.org", "n2.remote.org") |
| 396 |
cl <- makeClusterPSOCK(workers) |
| 397 |
print(cl) |
| 398 |
#> Socket cluster with 3 nodes where 2 nodes are on host 'n1.remote.org' |
| 399 |
#> (R version 4.5.2 (2025-10-31), platform x86_64-pc-linux-gnu) |
| 400 |
#> 1 node is on host 'n2.remote.org' (R version 4.5.2 (2025-10-31), |
| 401 |
#> platform x86_64-pc-linux-gnu) |
| 402 |
``` |
| 403 |
|
| 404 |
Again, the `user` argument does not have to be specified, because it is configured in `~/.ssh/config`. |
| 405 |
|
| 406 |
To generalize to many workers, we can use the `rep()` function. For example, |
| 407 |
|
| 408 |
```r |
| 409 |
workers <- c(rep("n1.remote.org", 3), rep("n2.remote.org", 4)) |
| 410 |
``` |
| 411 |
|
| 412 |
sets up three workers on `n1.remote.org` and four on `n2.remote.org`, |
| 413 |
totaling seven parallel workers. |
| 414 |
|
| 415 |
|
| 416 |
## Example: A mix of local and remote workers |
| 417 |
|
| 418 |
As an alternative to `makeClusterPSOCK(n)`, we can use |
| 419 |
`makeClusterPSOCK(workers)` to set up parallelly workers running on |
| 420 |
the local machine. By convention, the name `localhost` is an alias to |
| 421 |
your local machine. This means, we can use: |
| 422 |
|
| 423 |
```sh |
| 424 |
library(parallelly) |
| 425 |
workers <- rep("localhost", 4) |
| 426 |
cl_local <- makeClusterPSOCK(workers) |
| 427 |
print(cl_local) |
| 428 |
#> Socket cluster with 4 nodes on host 'localhost' (R version 4.5.2 |
| 429 |
#> (2025-10-31), platform x86_64-pc-linux-gnu) |
| 430 |
``` |
| 431 |
|
| 432 |
to launch four local parallel workers. Note how we did not have to |
| 433 |
specify `user = "ally"`. This is because the default username is |
| 434 |
always the local username. Next, assume we want to add another four |
| 435 |
parallel workers running on `n1.remote.org`. We already know we can |
| 436 |
set these up as: |
| 437 |
|
| 438 |
```sh |
| 439 |
library(parallelly) |
| 440 |
workers <- rep("n1.remote.org", 4) |
| 441 |
cl_remote <- makeClusterPSOCK(workers, user = "alice") |
| 442 |
print(cl_remote) |
| 443 |
#> Socket cluster with 4 nodes on host 'n1.remote.org' (R version 4.5.2 |
| 444 |
#> (2025-10-31), platform x86_64-pc-linux-gnu) |
| 445 |
``` |
| 446 |
|
| 447 |
At this point, we have two independent clusters of parallel workers: |
| 448 |
`cl_local` and `cl_remote`. We can combine them into a single |
| 449 |
cluster using: |
| 450 |
|
| 451 |
```r |
| 452 |
cl <- c(cl_local, cl_remote) |
| 453 |
print(cl) |
| 454 |
#> Socket cluster with 8 nodes where 4 nodes are on host 'localhost' |
| 455 |
#> (R version 4.5.2 (2025-10-31), platform x86_64-pc-linux-gnu) |
| 456 |
#> nodes are on host 'n1.remote.org' (R version 4.5.2 (2025-10-31), |
| 457 |
#> platform x86_64-pc-linux-gnu) |
| 458 |
``` |
| 459 |
|
| 460 |
To emphasize the usefulness of customizing our SSH connections via |
| 461 |
`~/.ssh/config`, if the remote username would already have been |
| 462 |
already configured there, we would be able to set up the full cluster |
| 463 |
in one single call, as in: |
| 464 |
|
| 465 |
```sh |
| 466 |
library(parallelly) |
| 467 |
workers <- c(rep("localhost", 4), rep("n1.remote.org", 4) |
| 468 |
cl <- makeClusterPSOCK(workers) |
| 469 |
``` |
| 470 |
|
| 471 |
|
| 472 |
## Example: Parallel workers on a remote machine accessed via dedicated login machine |
| 473 |
|
| 474 |
Sometimes a remote machine, where we want to run R, is only accessible |
| 475 |
via an intermediate login machine, which in SSH terms may also be |
| 476 |
referred to as a "jumphost". For example, assume machine |
| 477 |
`secret1.remote.org` can only be accessed by first logging into |
| 478 |
`login.remote.org` as in: |
| 479 |
|
| 480 |
```sh |
| 481 |
{ally@local}$ ssh [email protected] |
| 482 |
{alice@login}$ ssh [email protected] |
| 483 |
{alice@secret1}$ |
| 484 |
``` |
| 485 |
|
| 486 |
To achieve the same in a single SSH call, we can specify the |
| 487 |
"jumphost" `-J hostname` option for SSH, as in: |
| 488 |
|
| 489 |
```sh |
| 490 |
{ally@local}$ ssh -J [email protected] [email protected] |
| 491 |
{alice@secret1}$ |
| 492 |
``` |
| 493 |
|
| 494 |
We can use the `rshopts` argument of `makeClusterPSOCK()` to achieve |
| 495 |
the same when setting up parallel workers. To launch three parallel |
| 496 |
workers on `secret1.remote.org`, use: |
| 497 |
|
| 498 |
```r |
| 499 |
workers <- rep("secret1.remote.org", 3) |
| 500 |
cl <- makeClusterPSOCK( |
| 501 |
workers, |
| 502 |
rshopts = c("-J", "login.remote.org"), |
| 503 |
user = "alice" |
| 504 |
) |
| 505 |
``` |
| 506 |
|
| 507 |
A more convenient solution is to configure the jumphost in |
| 508 |
`~/.ssh/config`, as in: |
| 509 |
|
| 510 |
```plain |
| 511 |
Host *.remote.org |
| 512 |
User alice |
| 513 |
|
| 514 |
Host secret?.remote.org |
| 515 |
ProxyJump login.remote.org |
| 516 |
``` |
| 517 |
|
| 518 |
This will cause any SSH connection to a machine on the `remote.org` |
| 519 |
network to use username `alice`. It will also cause any SSH |
| 520 |
connection to machines `secret1.remote.org`, `secret2.remote.org`, and |
| 521 |
so on, to use jumphost `login.remote.org`. You can verify that all |
| 522 |
this work by: |
| 523 |
|
| 524 |
```sh |
| 525 |
{ally@local}$ ssh login.remote.org |
| 526 |
{alice@login}$ |
| 527 |
``` |
| 528 |
|
| 529 |
and then: |
| 530 |
|
| 531 |
```sh |
| 532 |
{ally@local}$ ssh secret1.remote.org |
| 533 |
{alice@secret1}$ |
| 534 |
``` |
| 535 |
|
| 536 |
If the above work, then the following will work from within R: |
| 537 |
|
| 538 |
```r |
| 539 |
library(parallelly) |
| 540 |
workers <- rep("secret1.remote.org", 3) |
| 541 |
cl <- makeClusterPSOCK(workers) |
| 542 |
``` |
| 543 |
|
| 544 |
|
| 545 |
# Special needs and tweaks |
| 546 |
|
| 547 |
The above sections cover most common use cases for setting up a |
| 548 |
parallel cluster from a local Linux, macOS, and MS Windows |
| 549 |
machine. However, there are cases where there above does not work, or |
| 550 |
you prefer to use another solution. This section aims to cover such |
| 551 |
alternatives. |
| 552 |
|
| 553 |
|
| 554 |
## Example: Remote workers ignoring any remote .Rprofile settings |
| 555 |
|
| 556 |
To launch parallel workers skipping any `~/.Rprofile` settings on the |
| 557 |
remote machines, we can pass option `--no-init-file` to `Rscript` via |
| 558 |
argument `rscript_args`. For example, |
| 559 |
|
| 560 |
```r |
| 561 |
workers <- rep("n1.remote.org", 2) |
| 562 |
cl <- makeClusterPSOCK(workers, rscript_args = "--no-init-file") |
| 563 |
``` |
| 564 |
|
| 565 |
will launch two parallel workers on `n1.remote.org` ignoring any |
| 566 |
`.Rprofile` files. |
| 567 |
|
| 568 |
|
| 569 |
## Example: Use PuTTY on MS Windows to connect to remote worker |
| 570 |
|
| 571 |
If you run on an MS Windows machine and prefer to use PuTTY to manage |
| 572 |
your SSH connections, or for other reasons cannot use the built-in |
| 573 |
`ssh` client, you can tell `makeClusterPSOCK()` to use PuTTY and your |
| 574 |
PuTTY settings via various arguments. |
| 575 |
|
| 576 |
Here is an example that launches two parallel workers on |
| 577 |
`n1.remote.org` running under user `alice` connecting via SSH port |
| 578 |
2201 using PuTTY and public-private SSH keys in file |
| 579 |
`C:/Users/ally/.ssh/putty.ppk`: |
| 580 |
|
| 581 |
```r |
| 582 |
workers <- "n1.remote.org" |
| 583 |
cl <- makeClusterPSOCK( |
| 584 |
workers, |
| 585 |
user = "alice", |
| 586 |
rshcmd = "<putty-plink>", |
| 587 |
rshopts = c("-P", 2201, "-i", "C:/Users/ally/.ssh/putty.ppk") |
| 588 |
) |
| 589 |
``` |
| 590 |
|
| 591 |
|
| 592 |
## Example: Two remote workers running on MS Windows |
| 593 |
|
| 594 |
Thus far we have considered our remote machines to run a Unix-like |
| 595 |
operating system, e.g. Linux or macOS. If your remote machines run MS |
| 596 |
Windows, you can use similar techniques to launch parallel workers |
| 597 |
there as well. For this to work, the remote MS Windows machines must |
| 598 |
accept incoming SSH connections, which is something most Windows |
| 599 |
machines are not configured to do by default. If you do not know how to set |
| 600 |
that up, or if you do not have the system permissions to do so, please |
| 601 |
reach out to your system administrator of those machines. |
| 602 |
|
| 603 |
Assuming we have SSH access to two MS Windows machines, |
| 604 |
`mswin1.remote.org` and `mswin2.remote.org`, everything works the same |
| 605 |
as before, except that we need to specify also argument `rscript_sh = |
| 606 |
"cmd"`; |
| 607 |
|
| 608 |
```r |
| 609 |
workers <- c("mswin1.remote.org", "mswin2.remote.org") |
| 610 |
cl <- makeClusterPSOCK(workers, rscript_sh = "cmd") |
| 611 |
``` |
| 612 |
|
| 613 |
That argument specifies that the parallel R workers should be launched |
| 614 |
on the remote machines via MS Windows' `cmd.exe` shell. |