| 1 |
<!DOCTYPE html> |
| 2 |
<html lang="en"> |
| 3 |
<head> |
| 4 |
<meta charset="UTF-8"> |
| 5 |
<title>Parallel Workers with CPU and Memory Limited</title> |
| 6 |
<style> |
| 7 |
body { |
| 8 |
font-family: sans-serif; |
| 9 |
line-height: 1.6; |
| 10 |
padding-left: 3ex; |
| 11 |
padding-right: 3ex; |
| 12 |
background-color: white; |
| 13 |
color: black; |
| 14 |
} |
| 15 |
|
| 16 |
a { |
| 17 |
color: #4183C4; |
| 18 |
text-decoration: none; |
| 19 |
} |
| 20 |
|
| 21 |
h1, h2, h3 { |
| 22 |
margin: 2ex 0 1ex; |
| 23 |
padding: 0; |
| 24 |
font-weight: bold; |
| 25 |
-webkit-font-smoothing: antialiased; |
| 26 |
cursor: text; |
| 27 |
position: relative; |
| 28 |
} |
| 29 |
|
| 30 |
h2 { |
| 31 |
border-bottom: 1px solid #cccccc; |
| 32 |
} |
| 33 |
|
| 34 |
code { |
| 35 |
margin: 0 2px; |
| 36 |
padding: 0 5px; |
| 37 |
white-space: nowrap; |
| 38 |
border: 1px solid #eaeaea; |
| 39 |
background-color: #f8f8f8; |
| 40 |
border-radius: 3px; |
| 41 |
} |
| 42 |
|
| 43 |
pre code { |
| 44 |
margin: 0; |
| 45 |
padding: 0; |
| 46 |
white-space: pre; |
| 47 |
border: none; |
| 48 |
background: transparent; |
| 49 |
} |
| 50 |
|
| 51 |
pre { |
| 52 |
background-color: #f8f8f8; |
| 53 |
border: 1px solid #cccccc; |
| 54 |
line-height: 2.5x; |
| 55 |
overflow: auto; |
| 56 |
padding: 0.6ex 1ex; |
| 57 |
border-radius: 3px; |
| 58 |
} |
| 59 |
|
| 60 |
pre code { |
| 61 |
background-color: transparent; |
| 62 |
border: none; |
| 63 |
} |
| 64 |
</style> |
| 65 |
</head> |
| 66 |
<body> |
| 67 |
<h1>Parallel Workers with CPU and Memory Limited</h1> |
| 68 |
<!-- |
| 69 |
%\VignetteIndexEntry{Parallel Workers with CPU and Memory Limited} |
| 70 |
%\VignetteAuthor{Henrik Bengtsson} |
| 71 |
%\VignetteKeyword{R} |
| 72 |
%\VignetteKeyword{package} |
| 73 |
%\VignetteKeyword{vignette} |
| 74 |
%\VignetteEngine{parallelly::selfonly} |
| 75 |
--> |
| 76 |
<h1>Introduction</h1> |
| 77 |
<p>This vignette gives examples of how to restrict CPU and memory usage of |
| 78 |
parallel workers. This can be useful for optimizing the performance of |
| 79 |
the parallel workers, but also lower the risk that they overuse the |
| 80 |
CPU and memory on the machines they are running on.</p> |
| 81 |
<h1>Examples</h1> |
| 82 |
<h2>Example: Linux parallel workers with a lower process priority (“nice”)</h2> |
| 83 |
<p>On Unix, we can run any process with a lower CPU priority using the |
| 84 |
<code>nice</code> command. This can be used when we want to lower the risk of |
| 85 |
negatively affecting other users and processes that run on the same |
| 86 |
machine due to our R workers overusing the CPUs by mistake. To achieve |
| 87 |
this, we can prepend <code>nice</code> to the <code>Rscript</code> call via the <code>rscript</code> |
| 88 |
argument. This works both on local and remote Linux machines, |
| 89 |
e.g.</p> |
| 90 |
<pre><code class="language-r">library(parallelly) |
| 91 |
cl <- makeClusterPSOCK(2, rscript = c("nice", "*")) |
| 92 |
</code></pre> |
| 93 |
<pre><code class="language-r">library(parallelly) |
| 94 |
workers <- rep("n1.remote.org", 2) |
| 95 |
cl <- makeClusterPSOCK(2, rscript = c("nice", "*")) |
| 96 |
</code></pre> |
| 97 |
<p>The special <code>*</code> value expands to the proper <code>Rscript</code> on the machine |
| 98 |
where the parallel workers are launched.</p> |
| 99 |
<h2>Example: Linux parallel workers CPU and memory limited by CGroups</h2> |
| 100 |
<p>This example launches two parallel workers each limited to 100% CPU |
| 101 |
quota and 50 MiB of memory using Linux CGroups management. The 100% |
| 102 |
CPU quota limit constrains each worker to use at most one CPU worth of |
| 103 |
processing preventing them from overusing the machine, e.g. through |
| 104 |
unintended nested parallelization. For more details, see <code>man systemd.resource-control</code>.</p> |
| 105 |
<pre><code class="language-r">library(parallelly) |
| 106 |
cl <- makeClusterPSOCK( |
| 107 |
2L, |
| 108 |
rscript = c( |
| 109 |
"systemd-run", "--user", "--scope", |
| 110 |
"-p", "CPUQuota=100%", |
| 111 |
"-p", "MemoryMax=50M", "-p", "MemorySwapMax=50M", |
| 112 |
"*" |
| 113 |
) |
| 114 |
) |
| 115 |
</code></pre> |
| 116 |
<p>Note, depending on your CGroups configuration, a non-privileged user |
| 117 |
may or may not be able to set the CPU quota. If not, the <code>-p CPUQuota=100%</code> will be silently ignored.</p> |
| 118 |
<p>The 50 MiB memory limit is strict - if a worker uses more than this, |
| 119 |
the operating system will terminate the worker instantly. To |
| 120 |
illustrate what happens, we first start by generating 1 million |
| 121 |
numeric values each consuming 8 bytes, which in total consumes ~8 MB, |
| 122 |
and then calculate the mean, the memory consumption is within 50-MiB |
| 123 |
memory limit that each parallel worker has available;</p> |
| 124 |
<pre><code class="language-r">library(parallel) |
| 125 |
mu <- clusterEvalQ(cl, { x <- rnorm(n = 1e6); mean(x) }) |
| 126 |
mu <- unlist(mu) |
| 127 |
print(mu) |
| 128 |
#> [1] 0.0008072657 -0.0019693992 |
| 129 |
</code></pre> |
| 130 |
<p>However, if we generate 10 times more values, the memory consumption |
| 131 |
will grow to at least 80 MB, which is over the 50-MiB memory limit, |
| 132 |
and we will get an error:</p> |
| 133 |
<pre><code class="language-r">mu <- clusterEvalQ(cl, { x <- rnorm(n = 10e6); mean(x) }) |
| 134 |
#> Error in unserialize(node$con) : error reading from connection |
| 135 |
</code></pre> |
| 136 |
<p>This is because the operating system terminated the two background R |
| 137 |
processes, because they overused the memory. This is why the main R |
| 138 |
process no longer can communicate with the parallel workers. We can |
| 139 |
see that both workers are down, by calling:</p> |
| 140 |
<pre><code class="language-r">isNodeAlive(cl) |
| 141 |
#> [1] FALSE FALSE |
| 142 |
</code></pre> |
| 143 |
<p>We can use <code>cloneNode()</code> to relaunch workers that are no longer |
| 144 |
alive, e.g.</p> |
| 145 |
<pre><code class="language-r">is_down <- !isNodeAlive(cl) |
| 146 |
cl[is_down] <- cloneNode(cl[is_down]) |
| 147 |
isNodeAlive(cl) |
| 148 |
#> [1] TRUE TRUE |
| 149 |
</code></pre> |
| 150 |
<h2>Example: MS Windows parallel workers with specific CPU affinities</h2> |
| 151 |
<p>This example works only on MS Windows machines. It launches four |
| 152 |
local workers, where two are running on CPU Group #0 and two on CPU |
| 153 |
Group #1.</p> |
| 154 |
<pre><code class="language-r">library(parallelly) |
| 155 |
rscript <- I(c( |
| 156 |
Sys.getenv("COMSPEC"), "/c", |
| 157 |
"start", "/B", |
| 158 |
"/NODE", cpu_group=NA_integer_, |
| 159 |
"/AFFINITY", "0xFFFFFFFFFFFFFFFE", |
| 160 |
"*") |
| 161 |
) |
| 162 |
|
| 163 |
rscript["cpu_group"] <- 0 |
| 164 |
cl_0 <- makeClusterPSOCK(2, rscript = rscript) |
| 165 |
|
| 166 |
rscript["cpu_group"] <- 1 |
| 167 |
cl_1 <- makeClusterPSOCK(2, rscript = rscript) |
| 168 |
|
| 169 |
cl <- c(cl_0, cl_1) |
| 170 |
</code></pre> |
| 171 |
<p>The special <code>*</code> value expands to the proper <code>Rscript</code> on the machine |
| 172 |
where the parallel workers are launched.</p> |
| 173 |
<!-- See also: https://lovickconsulting.com/2021/11/18/running-r-clusters-on-an-amd-threadripper-3990x-in-windows-10-2/ --> |
| 174 |
</body> |
| 175 |
</html> |