| 1 |
<!DOCTYPE html> |
| 2 |
<html lang="en"> |
| 3 |
<head> |
| 4 |
<meta charset="UTF-8"> |
| 5 |
<title>Parallel Workers on Other Machines</title> |
| 6 |
<style> |
| 7 |
body { |
| 8 |
font-family: sans-serif; |
| 9 |
line-height: 1.6; |
| 10 |
padding-left: 3ex; |
| 11 |
padding-right: 3ex; |
| 12 |
background-color: white; |
| 13 |
color: black; |
| 14 |
} |
| 15 |
|
| 16 |
a { |
| 17 |
color: #4183C4; |
| 18 |
text-decoration: none; |
| 19 |
} |
| 20 |
|
| 21 |
h1, h2, h3 { |
| 22 |
margin: 2ex 0 1ex; |
| 23 |
padding: 0; |
| 24 |
font-weight: bold; |
| 25 |
-webkit-font-smoothing: antialiased; |
| 26 |
cursor: text; |
| 27 |
position: relative; |
| 28 |
} |
| 29 |
|
| 30 |
h2 { |
| 31 |
border-bottom: 1px solid #cccccc; |
| 32 |
} |
| 33 |
|
| 34 |
code { |
| 35 |
margin: 0 2px; |
| 36 |
padding: 0 5px; |
| 37 |
white-space: nowrap; |
| 38 |
border: 1px solid #eaeaea; |
| 39 |
background-color: #f8f8f8; |
| 40 |
border-radius: 3px; |
| 41 |
} |
| 42 |
|
| 43 |
pre code { |
| 44 |
margin: 0; |
| 45 |
padding: 0; |
| 46 |
white-space: pre; |
| 47 |
border: none; |
| 48 |
background: transparent; |
| 49 |
} |
| 50 |
|
| 51 |
pre { |
| 52 |
background-color: #f8f8f8; |
| 53 |
border: 1px solid #cccccc; |
| 54 |
line-height: 2.5x; |
| 55 |
overflow: auto; |
| 56 |
padding: 0.6ex 1ex; |
| 57 |
border-radius: 3px; |
| 58 |
} |
| 59 |
|
| 60 |
pre code { |
| 61 |
background-color: transparent; |
| 62 |
border: none; |
| 63 |
} |
| 64 |
</style> |
| 65 |
</head> |
| 66 |
<body> |
| 67 |
<h1>Parallel Workers on Other Machines</h1> |
| 68 |
<!-- |
| 69 |
%\VignetteIndexEntry{Parallel Workers on Other Machines} |
| 70 |
%\VignetteAuthor{Henrik Bengtsson} |
| 71 |
%\VignetteKeyword{R} |
| 72 |
%\VignetteKeyword{package} |
| 73 |
%\VignetteKeyword{vignette} |
| 74 |
%\VignetteEngine{parallelly::selfonly} |
| 75 |
--> |
| 76 |
<h1>Introduction</h1> |
| 77 |
<p>Sometimes it is not sufficient to parallelize on a single computer - |
| 78 |
it cannot provide all of the compute power we are looking for. When we |
| 79 |
hit this limit, a natural next level is to look at other computers |
| 80 |
near us, e.g. desktops in an office or other computers we have access |
| 81 |
to remotely. In this vignette, we will cover how to run parallel R |
| 82 |
workers on other machines. Sometimes we distinguish between local |
| 83 |
machines and on remote machines, where <em>local</em> machines are machines |
| 84 |
considered to be on the same local area network (LAN) and that might |
| 85 |
share a common file system. <em>Remote</em> machines are machines that are on |
| 86 |
a different network and that do not share a common file system with |
| 87 |
the main R computer. In most cases the distinction between local and |
| 88 |
remote machines does not matter, but in some cases we can take |
| 89 |
advantage of workers being local.</p> |
| 90 |
<p>Regardless of running parallel workers on local or remote machines, we |
| 91 |
need a way to connect to the machines and launch R on them.</p> |
| 92 |
<h2>SSH and R configuration (once)</h2> |
| 93 |
<h3>Verifying SSH access</h3> |
| 94 |
<p>The most common approach to connect to another machine is via Secure |
| 95 |
Shell (SSH). Linux, macOS, and MS Windows all have a built-in SSH |
| 96 |
client called <code>ssh</code>. Consider we have another Linux machine called |
| 97 |
<code>n1.remote.org</code>, it can be accessed via SSH, and we have an account |
| 98 |
<code>alice</code> on that machine. For the case of these instructions, it does |
| 99 |
not matter whether <code>n1.remote.org</code> is on our local network (LAN) or a |
| 100 |
remote machine on the internet. Also, to make it clear that we do not have to have the same username on <code>n1.remote.org</code> and on our local machine, we will use <code>ally</code> as the username on our local machine.</p> |
| 101 |
<p>To access the <code>alice</code> user account on <code>n1.remote.org</code> from our local |
| 102 |
computer, we open a terminal on the local computer and then SSH to the |
| 103 |
other machine as:</p> |
| 104 |
<pre><code class="language-sh">{ally@local}$ ssh [email protected] |
| 105 |
[email protected]'s password: ************* |
| 106 |
{alice@n1}$ |
| 107 |
</code></pre> |
| 108 |
<p>The commands to call are what follows after the prompt. The prompt on |
| 109 |
our local machine is <code>{ally@local}$</code>, which tells us that our username |
| 110 |
is <code>ally</code> and the name of the local machine is <code>local</code>. The prompt on |
| 111 |
the <code>n1.remote.org</code> machine is <code>{alice@n1}$</code>, which tells us that our |
| 112 |
username on that machine is <code>alice</code> and that the machine is called |
| 113 |
<code>n1</code> on that system.</p> |
| 114 |
<p>To return to our local machine, exit the SSH shell by typing <code>exit</code>;</p> |
| 115 |
<pre><code class="language-sh">{alice@n1}$ exit |
| 116 |
{ally@local}$ |
| 117 |
</code></pre> |
| 118 |
<p>If we get this far, we have confirmed that we have SSH access to this |
| 119 |
machine.</p> |
| 120 |
<h3>Configure password-less SSH access</h3> |
| 121 |
<p>Launching parallel R workers is typically done automatically in the |
| 122 |
background, which means it is cumbersome, or even impossible, to enter |
| 123 |
the SSH password for each machine we wish to connect to. The solution |
| 124 |
is to configure SSH to connect with <em>public-private keys</em>, which |
| 125 |
pre-establish SSH authentication between the main machine and the |
| 126 |
machine to connect to. As this is common practice when working with |
| 127 |
SSH, there are numerous online tutorials explaining how to configure |
| 128 |
private-public SSH key pairs. Please consult one of them for the |
| 129 |
details, but the gist is to use (i) <code>ssh-keygen</code> to generate the |
| 130 |
public-private SSH keys on your local machine, and then (ii) |
| 131 |
<code>ssh-copy-id</code> to deploy the public key on the machine you want to |
| 132 |
connect to.</p> |
| 133 |
<p>Step 1: Generate public-private SSH keys locally</p> |
| 134 |
<pre><code class="language-sh">{ally@local}$ ssh-keygen |
| 135 |
Generating public/private rsa key pair. |
| 136 |
Enter file in which to save the key (/home/ally/.ssh/id_rsa): |
| 137 |
Created directory '/home/ally/.ssh'. |
| 138 |
Enter passphrase (empty for no passphrase): |
| 139 |
Enter same passphrase again: |
| 140 |
Your identification has been saved in /home/ally/.ssh/id_rsa |
| 141 |
Your public key has been saved in /home/ally/.ssh/id_rsa.pub |
| 142 |
The key fingerprint is: |
| 143 |
SHA256:Sx48uXZTUL12SKKUzWB77e/Pm3TifqrDIbOnJ0pEWHY ally@local |
| 144 |
The key's randomart image is: |
| 145 |
+---[RSA 3072]----+ |
| 146 |
| o E=.. | |
| 147 |
| + ooo+.o | |
| 148 |
| . ..o..o.o | |
| 149 |
| o ..o .+ .| |
| 150 |
| S .... | |
| 151 |
| + =o.. . | |
| 152 |
| * o= ...o| |
| 153 |
| o .o.=..++| |
| 154 |
| ...=.++=*| |
| 155 |
+----[SHA256]-----+ |
| 156 |
</code></pre> |
| 157 |
<p>Step 2: Copy the public SSH key to the other machine</p> |
| 158 |
<pre><code class="language-sh">{ally@local}$ ssh-copy-id [email protected] |
| 159 |
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/ally/.ssh/id_rsa.pub" |
| 160 |
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed |
| 161 |
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys |
| 162 |
[email protected]:s password: ************* |
| 163 |
|
| 164 |
Number of key(s) added: 1 |
| 165 |
|
| 166 |
Now try logging into the machine, with: "ssh '[email protected]'" |
| 167 |
and check to make sure that only the key(s) you wanted were added. |
| 168 |
</code></pre> |
| 169 |
<p>At this point, we should be able to SSH to the other machine without |
| 170 |
having to enter a password;</p> |
| 171 |
<pre><code class="language-sh">{ally@local}$ ssh [email protected] |
| 172 |
{alice@n1}$ |
| 173 |
</code></pre> |
| 174 |
<p>Type <code>exit</code> to return to your local machine.</p> |
| 175 |
<p>Note, if you later want to connect to other machines, |
| 176 |
e.g. <code>n2.remote.org</code> or <code>hpc.my-university.edu</code>, you may re-use the |
| 177 |
above generated keys for those systems too. In other words, you do not |
| 178 |
have to use <code>ssh-keygen</code> to generate new keys for those machines.</p> |
| 179 |
<h3>Verifying R exists on the other machine</h3> |
| 180 |
<p>In order to run parallel R workers on another machine, it (i) needs to |
| 181 |
be installed on that machine, and (ii) ideally readily available by |
| 182 |
calling <code>Rscript</code>. Parallel R workers are launched via <code>Rscript</code>, |
| 183 |
instead of the more commonly known <code>R</code> command - both come with all R |
| 184 |
installation, i.e. if you have one of them, you have the other too.</p> |
| 185 |
<p>To verify that R is installed on the other machine, SSH to the machine and call <code>Rscript --version</code>;</p> |
| 186 |
<pre><code class="language-sh">{ally@local}$ ssh [email protected] |
| 187 |
{alice@n1}$ Rscript --version |
| 188 |
Rscript (R) version 4.5.2 (2025-10-31) |
| 189 |
</code></pre> |
| 190 |
<p>If you get:</p> |
| 191 |
<pre><code class="language-sh">{alice@n1}$ Rscript --version |
| 192 |
Rscript: command not found |
| 193 |
</code></pre> |
| 194 |
<p>then R is either not installed on that machine, or it cannot be |
| 195 |
found. If it is installed, but cannot be found, make sure that |
| 196 |
environment variable <code>PATH</code> is configured properly on that machine.</p> |
| 197 |
<h3>Final checks</h3> |
| 198 |
<p>With password-less SSH access, and R being available, on the other |
| 199 |
machine, we should be able to SSH into the other machine and query the |
| 200 |
R version in a single call:</p> |
| 201 |
<pre><code class="language-sh">{ally@local}$ ssh [email protected] Rscript --version |
| 202 |
Rscript (R) version 4.5.2 (2025-10-31) |
| 203 |
{ally@local}$ |
| 204 |
</code></pre> |
| 205 |
<p>This is all that is needed to launch one or more parallel R workers on |
| 206 |
machine <code>n1.remote.org</code> running under user <code>alice</code>. We can test this |
| 207 |
from within R with the <strong>parallelly</strong> package using:</p> |
| 208 |
<pre><code class="language-sh">{ally@local}$ R --quiet |
| 209 |
library(parallelly) |
| 210 |
cl <- makeClusterPSOCK("n1.remote.org", user = "alice") |
| 211 |
print(cl) |
| 212 |
#> Socket cluster with 1 node on host 'n1.remote.org' (R version 4.5.2 |
| 213 |
#> (2025-10-31), platform x86_64-pc-linux-gnu) |
| 214 |
parallel::stopCluster(cl) |
| 215 |
</code></pre> |
| 216 |
<p>If you want to run parallel workers on other machines, repeat the |
| 217 |
above for each machine. After this, you will be able to launch |
| 218 |
parallel R workers on these machines with little effort.</p> |
| 219 |
<h3>Machine-specific SSH customization (recommended)</h3> |
| 220 |
<p>Some machines do not use the default port 22 to answer on SSH |
| 221 |
connection requests. If the machine uses another port, say, port 2201, |
| 222 |
then we can specify that via option <code>-p port</code>, when we connect to it, |
| 223 |
e.g.</p> |
| 224 |
<pre><code class="language-sh">{ally@local}$ ssh -p 2201 [email protected] |
| 225 |
</code></pre> |
| 226 |
<p>In R, we can specify argument <code>port=port</code> as in:</p> |
| 227 |
<pre><code class="language-sh">cl <- makeClusterPSOCK("n1.remote.org", port = 2201, user = "alice") |
| 228 |
</code></pre> |
| 229 |
<p>Now, it can be tedious to have to remember custom SSH ports and |
| 230 |
usernames when setting up remote workers in R. It also adds noise and |
| 231 |
distraction having such details in the R script, and not to mention |
| 232 |
the fact that the R script has a specific username hardcoded into the |
| 233 |
code makes the script less reproducible for other users - they need to |
| 234 |
change the code to match their username. One way to avoid having to |
| 235 |
give specific SSH options when calling <code>ssh</code> in the terminal, or |
| 236 |
<code>makeClusterPSOCK()</code> in R, is to configure these settings in SSH. This |
| 237 |
can be done via a file called <code>~/.ssh/config</code> <em>on your local |
| 238 |
machine</em>. This file does not exist by default, so you would have to |
| 239 |
create it yourself, if missing. It is a plain text file, so you should |
| 240 |
use a plain text editor to create and edit it.</p> |
| 241 |
<p>To configure SSH to use port 2201 and username <code>alice</code> whenever |
| 242 |
connecting to <code>n1.remote.org</code>, the <code>~/.ssh/config</code> file should |
| 243 |
contain the following entry:</p> |
| 244 |
<pre><code class="language-plain">Host n1.remote.org |
| 245 |
User alice |
| 246 |
Port 2201 |
| 247 |
</code></pre> |
| 248 |
<p>With this, we can connect to <code>n1.remote.org</code> by just using:</p> |
| 249 |
<pre><code class="language-sh">{ally@local}$ ssh n1.remote.org |
| 250 |
{alice@n1}$ |
| 251 |
</code></pre> |
| 252 |
<p>SSH will then connect to the machine as if we had specified also <code>-p 2201</code> and <code>-l alice</code>. These settings will also be picked up when we |
| 253 |
connect via R, meaning the following will also work:</p> |
| 254 |
<pre><code class="language-sh">cl <- makeClusterPSOCK("n1.remote.org") |
| 255 |
</code></pre> |
| 256 |
<p>To achieve the same for other machines, add another entry for them, |
| 257 |
e.g.</p> |
| 258 |
<pre><code class="language-plain">Host n1.remote.org |
| 259 |
User alice |
| 260 |
Port 2201 |
| 261 |
|
| 262 |
Host n2.remote.org |
| 263 |
User alice |
| 264 |
Port 2201 |
| 265 |
|
| 266 |
Host hpc.my-university.edu |
| 267 |
User alice.bobson |
| 268 |
</code></pre> |
| 269 |
<p>When hosts on the same system share the same setting, one can use |
| 270 |
globbing to configure them the same way. For instance, the above can |
| 271 |
be shorted to:</p> |
| 272 |
<pre><code class="language-plain">Host n?.remote.org |
| 273 |
User alice |
| 274 |
Port 2201 |
| 275 |
|
| 276 |
Host hpc.my-university.edu |
| 277 |
User alice.bobson |
| 278 |
</code></pre> |
| 279 |
<p>Being able to connect to remote machines by just specifying their |
| 280 |
hostnames is convenient and simplifies also the R code. Because of |
| 281 |
this, we recommend setting up also <code>~/.ssh/config</code>.</p> |
| 282 |
<h1>Examples</h1> |
| 283 |
<h2>Example: Two parallel workers on a single remote machine</h2> |
| 284 |
<p>Our first example sets up two parallel workers on the remote machine |
| 285 |
<code>n1.remote.org</code>. For this to work, we need SSH access to the machine, |
| 286 |
and it must have R installed, as explained in the above |
| 287 |
section. Contrary to local parallel workers, the number of parallel |
| 288 |
workers on remote machines is specified by repeating the machine name |
| 289 |
an equal number of times;</p> |
| 290 |
<pre><code class="language-r">library(parallelly) |
| 291 |
workers <- c("n1.remote.org", "n1.remote.org") |
| 292 |
cl <- makeClusterPSOCK(workers, user = "alice") |
| 293 |
print(cl) |
| 294 |
#> Socket cluster with 2 nodes are on host 'n1.remote.org' (R version 4.5.2 |
| 295 |
#> (2025-10-31), platform x86_64-pc-linux-gnu) |
| 296 |
</code></pre> |
| 297 |
<p><em>Comment</em>: In the <strong>parallel</strong> package, a parallel worker is referred |
| 298 |
to a parallel node, or short <em>node</em>, which is why we use the same term |
| 299 |
in the <strong>parallelly</strong> package.</p> |
| 300 |
<p>Note, contrary to parallel workers running on the local machine, |
| 301 |
parallel workers on remote machines are launched sequentially, that is |
| 302 |
one after each other. Because of this, the setup time for a remote |
| 303 |
parallel cluster will increase linearly with the number of remote |
| 304 |
parallel workers.</p> |
| 305 |
<p><em>Technical details</em>: If we would add <code>verbose = TRUE</code> to |
| 306 |
<code>makeClusterPSOCK()</code>, we would learn that the parallel workers are |
| 307 |
launched in the background by R using something like:</p> |
| 308 |
<pre><code>'/usr/bin/ssh' -R 11058:localhost:11058 -l alice n1.remote.org Rscript ... |
| 309 |
'/usr/bin/ssh' -R 11059:localhost:11059 -l alice n1.remote.org Rscript ... |
| 310 |
</code></pre> |
| 311 |
<p>This tells us that there is one active SSH connection per parallel |
| 312 |
worker. It also reveals that that each of these connections uses a so |
| 313 |
called <em>reverse tunnel</em>, which is used to establish a unique |
| 314 |
communication channel between the main R process and the corresponding |
| 315 |
parallel worker. It also this use of reverse tunneling that avoids |
| 316 |
having to configure dynamic DNS (DDNS) and port-forwarding in our |
| 317 |
local firewalls, which is cumbersome and requires administrative |
| 318 |
rights. When using <strong>parallelly</strong>, there is no need for administrative |
| 319 |
rights - any non-privileged user can launch remote parallel R workers.</p> |
| 320 |
<h2>Example: Two parallel workers on two remote machines</h2> |
| 321 |
<p>This example sets up a parallel worker on each of two remote machines |
| 322 |
<code>n1.remote.org</code> and <code>n2.remote.org</code>. It works very similar to the |
| 323 |
previous example, but now the two SSH connections go to two different |
| 324 |
machines rather than the same.</p> |
| 325 |
<pre><code class="language-r">library(parallelly) |
| 326 |
workers <- c("n1.remote.org", "n2.remote.org") |
| 327 |
cl <- makeClusterPSOCK(workers, user = "alice") |
| 328 |
print(cl) |
| 329 |
#> Socket cluster with 2 nodes where 1 node is on host 'n1.remote.org' |
| 330 |
#> (R version 4.5.2 (2025-10-31), platform x86_64-pc-linux-gnu) |
| 331 |
#> 1 node is on host 'n2.remote.org' (R version 4.5.2 (2025-10-31), |
| 332 |
#> platform x86_64-pc-linux-gnu) |
| 333 |
</code></pre> |
| 334 |
<p><em>Technical details</em>: If we would add <code>verbose = TRUE</code> also in this |
| 335 |
case, we would see:</p> |
| 336 |
<pre><code>'/usr/bin/ssh' -R 11464:localhost:11464 -l alice n1.remote.org Rscript ... |
| 337 |
'/usr/bin/ssh' -R 11465:localhost:11464 -l alice n2.remote.org Rscript ... |
| 338 |
</code></pre> |
| 339 |
<p>Recall, if we have configured SSH to pick up the username <code>alice</code> from |
| 340 |
<code>~/.ssh/config</code> on our local machine, as shown in the previous |
| 341 |
section, we could have skipped the <code>user</code> argument, and just used:</p> |
| 342 |
<pre><code class="language-r">workers <- c("n1.remote.org", "n2.remote.org") |
| 343 |
cl <- makeClusterPSOCK(workers) |
| 344 |
</code></pre> |
| 345 |
<p>Note how these instructions for setting up a parallel cluster on these |
| 346 |
two machines would be the identical for another user that has |
| 347 |
configured their personal <code>~/.ssh/config</code> file.</p> |
| 348 |
<h2>Example: Three parallel workers on two remote machines</h2> |
| 349 |
<p>When we now understand that we control the number of parallel workers |
| 350 |
on a specific machine by replicate the machine name, we also know how |
| 351 |
to launch different number of parallel workers on different machines. |
| 352 |
From now on, we will also assume that the remote username no longer |
| 353 |
has to be specified, because it has already been configured via the |
| 354 |
<code>~/.ssh/config</code> file. With this, we can sets up two parallel workers |
| 355 |
on <code>n1.remote.org</code> and one on <code>n2.remote.org</code>, by:</p> |
| 356 |
<pre><code class="language-r">library(parallelly) |
| 357 |
workers <- c("n1.remote.org", "n1.remote.org", "n2.remote.org") |
| 358 |
cl <- makeClusterPSOCK(workers) |
| 359 |
print(cl) |
| 360 |
#> Socket cluster with 3 nodes where 2 nodes are on host 'n1.remote.org' |
| 361 |
#> (R version 4.5.2 (2025-10-31), platform x86_64-pc-linux-gnu) |
| 362 |
#> 1 node is on host 'n2.remote.org' (R version 4.5.2 (2025-10-31), |
| 363 |
#> platform x86_64-pc-linux-gnu) |
| 364 |
</code></pre> |
| 365 |
<p>Again, the <code>user</code> argument does not have to be specified, because it is configured in <code>~/.ssh/config</code>.</p> |
| 366 |
<p>To generalize to many workers, we can use the <code>rep()</code> function. For example,</p> |
| 367 |
<pre><code class="language-r">workers <- c(rep("n1.remote.org", 3), rep("n2.remote.org", 4)) |
| 368 |
</code></pre> |
| 369 |
<p>sets up three workers on <code>n1.remote.org</code> and four on <code>n2.remote.org</code>, |
| 370 |
totaling seven parallel workers.</p> |
| 371 |
<h2>Example: A mix of local and remote workers</h2> |
| 372 |
<p>As an alternative to <code>makeClusterPSOCK(n)</code>, we can use |
| 373 |
<code>makeClusterPSOCK(workers)</code> to set up parallelly workers running on |
| 374 |
the local machine. By convention, the name <code>localhost</code> is an alias to |
| 375 |
your local machine. This means, we can use:</p> |
| 376 |
<pre><code class="language-sh">library(parallelly) |
| 377 |
workers <- rep("localhost", 4) |
| 378 |
cl_local <- makeClusterPSOCK(workers) |
| 379 |
print(cl_local) |
| 380 |
#> Socket cluster with 4 nodes on host 'localhost' (R version 4.5.2 |
| 381 |
#> (2025-10-31), platform x86_64-pc-linux-gnu) |
| 382 |
</code></pre> |
| 383 |
<p>to launch four local parallel workers. Note how we did not have to |
| 384 |
specify <code>user = "ally"</code>. This is because the default username is |
| 385 |
always the local username. Next, assume we want to add another four |
| 386 |
parallel workers running on <code>n1.remote.org</code>. We already know we can |
| 387 |
set these up as:</p> |
| 388 |
<pre><code class="language-sh">library(parallelly) |
| 389 |
workers <- rep("n1.remote.org", 4) |
| 390 |
cl_remote <- makeClusterPSOCK(workers, user = "alice") |
| 391 |
print(cl_remote) |
| 392 |
#> Socket cluster with 4 nodes on host 'n1.remote.org' (R version 4.5.2 |
| 393 |
#> (2025-10-31), platform x86_64-pc-linux-gnu) |
| 394 |
</code></pre> |
| 395 |
<p>At this point, we have two independent clusters of parallel workers: |
| 396 |
<code>cl_local</code> and <code>cl_remote</code>. We can combine them into a single |
| 397 |
cluster using:</p> |
| 398 |
<pre><code class="language-r">cl <- c(cl_local, cl_remote) |
| 399 |
print(cl) |
| 400 |
#> Socket cluster with 8 nodes where 4 nodes are on host 'localhost' |
| 401 |
#> (R version 4.5.2 (2025-10-31), platform x86_64-pc-linux-gnu) |
| 402 |
#> nodes are on host 'n1.remote.org' (R version 4.5.2 (2025-10-31), |
| 403 |
#> platform x86_64-pc-linux-gnu) |
| 404 |
</code></pre> |
| 405 |
<p>To emphasize the usefulness of customizing our SSH connections via |
| 406 |
<code>~/.ssh/config</code>, if the remote username would already have been |
| 407 |
already configured there, we would be able to set up the full cluster |
| 408 |
in one single call, as in:</p> |
| 409 |
<pre><code class="language-sh">library(parallelly) |
| 410 |
workers <- c(rep("localhost", 4), rep("n1.remote.org", 4) |
| 411 |
cl <- makeClusterPSOCK(workers) |
| 412 |
</code></pre> |
| 413 |
<h2>Example: Parallel workers on a remote machine accessed via dedicated login machine</h2> |
| 414 |
<p>Sometimes a remote machine, where we want to run R, is only accessible |
| 415 |
via an intermediate login machine, which in SSH terms may also be |
| 416 |
referred to as a “jumphost”. For example, assume machine |
| 417 |
<code>secret1.remote.org</code> can only be accessed by first logging into |
| 418 |
<code>login.remote.org</code> as in:</p> |
| 419 |
<pre><code class="language-sh">{ally@local}$ ssh [email protected] |
| 420 |
{alice@login}$ ssh [email protected] |
| 421 |
{alice@secret1}$ |
| 422 |
</code></pre> |
| 423 |
<p>To achieve the same in a single SSH call, we can specify the |
| 424 |
“jumphost” <code>-J hostname</code> option for SSH, as in:</p> |
| 425 |
<pre><code class="language-sh">{ally@local}$ ssh -J [email protected] [email protected] |
| 426 |
{alice@secret1}$ |
| 427 |
</code></pre> |
| 428 |
<p>We can use the <code>rshopts</code> argument of <code>makeClusterPSOCK()</code> to achieve |
| 429 |
the same when setting up parallel workers. To launch three parallel |
| 430 |
workers on <code>secret1.remote.org</code>, use:</p> |
| 431 |
<pre><code class="language-r">workers <- rep("secret1.remote.org", 3) |
| 432 |
cl <- makeClusterPSOCK( |
| 433 |
workers, |
| 434 |
rshopts = c("-J", "login.remote.org"), |
| 435 |
user = "alice" |
| 436 |
) |
| 437 |
</code></pre> |
| 438 |
<p>A more convenient solution is to configure the jumphost in |
| 439 |
<code>~/.ssh/config</code>, as in:</p> |
| 440 |
<pre><code class="language-plain">Host *.remote.org |
| 441 |
User alice |
| 442 |
|
| 443 |
Host secret?.remote.org |
| 444 |
ProxyJump login.remote.org |
| 445 |
</code></pre> |
| 446 |
<p>This will cause any SSH connection to a machine on the <code>remote.org</code> |
| 447 |
network to use username <code>alice</code>. It will also cause any SSH |
| 448 |
connection to machines <code>secret1.remote.org</code>, <code>secret2.remote.org</code>, and |
| 449 |
so on, to use jumphost <code>login.remote.org</code>. You can verify that all |
| 450 |
this work by:</p> |
| 451 |
<pre><code class="language-sh">{ally@local}$ ssh login.remote.org |
| 452 |
{alice@login}$ |
| 453 |
</code></pre> |
| 454 |
<p>and then:</p> |
| 455 |
<pre><code class="language-sh">{ally@local}$ ssh secret1.remote.org |
| 456 |
{alice@secret1}$ |
| 457 |
</code></pre> |
| 458 |
<p>If the above work, then the following will work from within R:</p> |
| 459 |
<pre><code class="language-r">library(parallelly) |
| 460 |
workers <- rep("secret1.remote.org", 3) |
| 461 |
cl <- makeClusterPSOCK(workers) |
| 462 |
</code></pre> |
| 463 |
<h1>Special needs and tweaks</h1> |
| 464 |
<p>The above sections cover most common use cases for setting up a |
| 465 |
parallel cluster from a local Linux, macOS, and MS Windows |
| 466 |
machine. However, there are cases where there above does not work, or |
| 467 |
you prefer to use another solution. This section aims to cover such |
| 468 |
alternatives.</p> |
| 469 |
<h2>Example: Remote workers ignoring any remote .Rprofile settings</h2> |
| 470 |
<p>To launch parallel workers skipping any <code>~/.Rprofile</code> settings on the |
| 471 |
remote machines, we can pass option <code>--no-init-file</code> to <code>Rscript</code> via |
| 472 |
argument <code>rscript_args</code>. For example,</p> |
| 473 |
<pre><code class="language-r">workers <- rep("n1.remote.org", 2) |
| 474 |
cl <- makeClusterPSOCK(workers, rscript_args = "--no-init-file") |
| 475 |
</code></pre> |
| 476 |
<p>will launch two parallel workers on <code>n1.remote.org</code> ignoring any |
| 477 |
<code>.Rprofile</code> files.</p> |
| 478 |
<h2>Example: Use PuTTY on MS Windows to connect to remote worker</h2> |
| 479 |
<p>If you run on an MS Windows machine and prefer to use PuTTY to manage |
| 480 |
your SSH connections, or for other reasons cannot use the built-in |
| 481 |
<code>ssh</code> client, you can tell <code>makeClusterPSOCK()</code> to use PuTTY and your |
| 482 |
PuTTY settings via various arguments.</p> |
| 483 |
<p>Here is an example that launches two parallel workers on |
| 484 |
<code>n1.remote.org</code> running under user <code>alice</code> connecting via SSH port |
| 485 |
2201 using PuTTY and public-private SSH keys in file |
| 486 |
<code>C:/Users/ally/.ssh/putty.ppk</code>:</p> |
| 487 |
<pre><code class="language-r">workers <- "n1.remote.org" |
| 488 |
cl <- makeClusterPSOCK( |
| 489 |
workers, |
| 490 |
user = "alice", |
| 491 |
rshcmd = "<putty-plink>", |
| 492 |
rshopts = c("-P", 2201, "-i", "C:/Users/ally/.ssh/putty.ppk") |
| 493 |
) |
| 494 |
</code></pre> |
| 495 |
<h2>Example: Two remote workers running on MS Windows</h2> |
| 496 |
<p>Thus far we have considered our remote machines to run a Unix-like |
| 497 |
operating system, e.g. Linux or macOS. If your remote machines run MS |
| 498 |
Windows, you can use similar techniques to launch parallel workers |
| 499 |
there as well. For this to work, the remote MS Windows machines must |
| 500 |
accept incoming SSH connections, which is something most Windows |
| 501 |
machines are not configured to do by default. If you do not know how to set |
| 502 |
that up, or if you do not have the system permissions to do so, please |
| 503 |
reach out to your system administrator of those machines.</p> |
| 504 |
<p>Assuming we have SSH access to two MS Windows machines, |
| 505 |
<code>mswin1.remote.org</code> and <code>mswin2.remote.org</code>, everything works the same |
| 506 |
as before, except that we need to specify also argument <code>rscript_sh = "cmd"</code>;</p> |
| 507 |
<pre><code class="language-r">workers <- c("mswin1.remote.org", "mswin2.remote.org") |
| 508 |
cl <- makeClusterPSOCK(workers, rscript_sh = "cmd") |
| 509 |
</code></pre> |
| 510 |
<p>That argument specifies that the parallel R workers should be launched |
| 511 |
on the remote machines via MS Windows’ <code>cmd.exe</code> shell.</p> |
| 512 |
</body> |
| 513 |
</html> |