Debian Codegraph @

inst/doc/parallelly-17-hpc-workers.html

raw
1 <!DOCTYPE html>
2 <html lang="en">
3 <head>
4 <meta charset="UTF-8">
5 <title>Parallel Workers on High-Performance Compute Environments</title>
6 <style>
7 body {
8 font-family: sans-serif;
9 line-height: 1.6;
10 padding-left: 3ex;
11 padding-right: 3ex;
12 background-color: white;
13 color: black;
14 }
15
16 a {
17 color: #4183C4;
18 text-decoration: none;
19 }
20
21 h1, h2, h3 {
22 margin: 2ex 0 1ex;
23 padding: 0;
24 font-weight: bold;
25 -webkit-font-smoothing: antialiased;
26 cursor: text;
27 position: relative;
28 }
29
30 h2 {
31 border-bottom: 1px solid #cccccc;
32 }
33
34 code {
35 margin: 0 2px;
36 padding: 0 5px;
37 white-space: nowrap;
38 border: 1px solid #eaeaea;
39 background-color: #f8f8f8;
40 border-radius: 3px;
41 }
42
43 pre code {
44 margin: 0;
45 padding: 0;
46 white-space: pre;
47 border: none;
48 background: transparent;
49 }
50
51 pre {
52 background-color: #f8f8f8;
53 border: 1px solid #cccccc;
54 line-height: 2.5x;
55 overflow: auto;
56 padding: 0.6ex 1ex;
57 border-radius: 3px;
58 }
59
60 pre code {
61 background-color: transparent;
62 border: none;
63 }
64 </style>
65 </head>
66 <body>
67 <h1>Parallel Workers on High-Performance Compute Environments</h1>
68 <!--
69 %\VignetteIndexEntry{Parallel Workers on High-Performance Compute Environments}
70 %\VignetteAuthor{Henrik Bengtsson}
71 %\VignetteKeyword{R}
72 %\VignetteKeyword{package}
73 %\VignetteKeyword{vignette}
74 %\VignetteKeyword{HPC}
75 %\VignetteEngine{parallelly::selfonly}
76 -->
77 <h1>Introduction</h1>
78 <p>This vignette illustrates how to launch parallel workers via job
79 schedulers running in high-performance compute (HPC) environments.</p>
80 <h1>Examples</h1>
81 <h2>Example: Launch parallel workers via the Grid Engine job scheduler</h2>
82 <p>‘Grid Engine’ is a high-performance compute (HPC) job scheduler where
83 one can request compute resources on multiple nodes, each running
84 multiple cores. Examples of Grid Engine schedulers are Oracle Grid
85 Engine (formerly Sun Grid Engine), Univa Grid Engine, and Son of Grid
86 Engine - all commonly referred to as SGE schedulers. Each SGE cluster
87 may have its own configuration with its own way of requesting
88 parallel slots.</p>
89 <p>Consider the following two files: <code>script.sh</code> and <code>script.R</code>.</p>
90 <p>script.sh:</p>
91 <pre><code class="language-sh">#! /usr/bin/env bash
92 #$ -cwd ## Run in current working directory
93 #$ -j y ## Merge stdout and stderr
94 #$ -l mem_free=100M ## 100 MiB RAM per slot
95 #$ -l h_rt=00:10:00 ## 10 minutes runtime
96 #$ -pe mpi 8 ## 8 compute slots
97
98 echo &quot;Information on R:&quot;
99 Rscript --version
100
101 echo &quot;Running R script:&quot;
102 Rscript script.R
103 </code></pre>
104 <p>script.R:</p>
105 <pre><code class="language-r">library(parallelly)
106 library(parallel)
107
108 cl &lt;- makeClusterPSOCK(
109 availableWorkers(),
110 rshcmd = &quot;qrsh&quot;, rshopts = c(&quot;-inherit&quot;, &quot;-nostdin&quot;, &quot;-V&quot;)
111 )
112 print(cl)
113
114 # Perform calculations in parallel
115 X &lt;- 1:100
116 y &lt;- parLapply(cl = cl, X, fun = sqrt)
117 y &lt;- unlist(y)
118 z &lt;- sum(y)
119 print(z)
120
121 stopCluster(cl)
122 </code></pre>
123 <p>The <code>script.sh</code> file is a job script that we submit to the scheduler
124 that runs the R script <code>script.R</code> when launched. If we submit
125 <code>script.sh</code> as:</p>
126 <pre><code class="language-sh">$ qsub script.sh
127 </code></pre>
128 <p>it will by default request eight slots - on one or more machines,
129 which then R and <strong>parallelly</strong> will set up a parallel cluster
130 on. Exactly on which machines depends on where the job scheduler finds
131 these requested slots.</p>
132 <p>Here is the output from one such run, where the scheduler happened to
133 allot the slots across three machines:</p>
134 <pre><code class="language-sh">Information on R:
135 Rscript (R) version 4.5.2 (2025-10-31)
136 Running R script:
137 Socket cluster with 8 nodes where 4 nodes are on host ‘localhost’
138 (R version 4.5.2 (2025-10-31), platform x86_64-pc-linux-gnu), 3
139 nodes are on host ‘qb3-id130’ (R version 4.5.2 (2025-10-31),
140 platform x86_64-pc-linux-gnu), 1 node is on host ‘qb3-as16’ (R
141 version 4.5.2 (2025-10-31), platform x86_64-pc-linux-gnu)
142 [1] 671.4629
143 </code></pre>
144 <h2>Example: Launch parallel workers via the Fujitsu Technical Computing Suite job scheduler</h2>
145 <p>The ‘Fujitsu Technical Computing Suite’ is a high-performance compute
146 (HPC) job scheduler where one can request compute resources on
147 multiple nodes, each running multiple cores.</p>
148 <p>Consider the following two files: <code>script.sh</code> and <code>script.R</code>.</p>
149 <p>script.sh:</p>
150 <pre><code class="language-sh">#! /usr/bin/env bash
151
152 echo &quot;Information on R:&quot;
153 Rscript --version
154
155 echo &quot;Running R script:&quot;
156 Rscript script.R
157 </code></pre>
158 <p>script.R:</p>
159 <pre><code class="language-r">library(parallelly)
160 library(parallel)
161
162 cl &lt;- makeClusterPSOCK(
163 availableWorkers(),
164 rshcmd = &quot;pjrsh&quot;
165 )
166 print(cl)
167
168 # Perform calculations in parallel
169 X &lt;- 1:100
170 y &lt;- parLapply(cl = cl, X, fun = sqrt)
171 y &lt;- unlist(y)
172 z &lt;- sum(y)
173 print(z)
174
175 stopCluster(cl)
176 </code></pre>
177 <p>The <code>script.sh</code> file is a job script that we submit to the scheduler
178 that runs the R script <code>script.R</code> when launched. We can submit
179 <code>script.sh</code> as:</p>
180 <pre><code class="language-sh">$ pjsub -L vnode=3 -L vnode-core=18 script.sh
181 </code></pre>
182 <p>to request 18 CPU cores on three compute nodes, which in total
183 requests 3*18=54 compute slots.</p>
184 </body>
185 </html>