| 1 |
<!-- |
| 2 |
%\VignetteIndexEntry{Parallel Workers with CPU and Memory Limited} |
| 3 |
%\VignetteAuthor{Henrik Bengtsson} |
| 4 |
%\VignetteKeyword{R} |
| 5 |
%\VignetteKeyword{package} |
| 6 |
%\VignetteKeyword{vignette} |
| 7 |
%\VignetteEngine{parallelly::selfonly} |
| 8 |
--> |
| 9 |
|
| 10 |
|
| 11 |
# Introduction |
| 12 |
|
| 13 |
This vignette gives examples of how to restrict CPU and memory usage of |
| 14 |
parallel workers. This can be useful for optimizing the performance of |
| 15 |
the parallel workers, but also lower the risk that they overuse the |
| 16 |
CPU and memory on the machines they are running on. |
| 17 |
|
| 18 |
|
| 19 |
# Examples |
| 20 |
|
| 21 |
## Example: Linux parallel workers with a lower process priority ("nice") |
| 22 |
|
| 23 |
On Unix, we can run any process with a lower CPU priority using the |
| 24 |
`nice` command. This can be used when we want to lower the risk of |
| 25 |
negatively affecting other users and processes that run on the same |
| 26 |
machine due to our R workers overusing the CPUs by mistake. To achieve |
| 27 |
this, we can prepend `nice` to the `Rscript` call via the `rscript` |
| 28 |
argument. This works both on local and remote Linux machines, |
| 29 |
e.g. |
| 30 |
|
| 31 |
```r |
| 32 |
library(parallelly) |
| 33 |
cl <- makeClusterPSOCK(2, rscript = c("nice", "*")) |
| 34 |
``` |
| 35 |
|
| 36 |
```r |
| 37 |
library(parallelly) |
| 38 |
workers <- rep("n1.remote.org", 2) |
| 39 |
cl <- makeClusterPSOCK(2, rscript = c("nice", "*")) |
| 40 |
``` |
| 41 |
|
| 42 |
The special `*` value expands to the proper `Rscript` on the machine |
| 43 |
where the parallel workers are launched. |
| 44 |
|
| 45 |
|
| 46 |
|
| 47 |
## Example: Linux parallel workers CPU and memory limited by CGroups |
| 48 |
|
| 49 |
This example launches two parallel workers each limited to 100% CPU |
| 50 |
quota and 50 MiB of memory using Linux CGroups management. The 100% |
| 51 |
CPU quota limit constrains each worker to use at most one CPU worth of |
| 52 |
processing preventing them from overusing the machine, e.g. through |
| 53 |
unintended nested parallelization. For more details, see `man |
| 54 |
systemd.resource-control`. |
| 55 |
|
| 56 |
```r |
| 57 |
library(parallelly) |
| 58 |
cl <- makeClusterPSOCK( |
| 59 |
2L, |
| 60 |
rscript = c( |
| 61 |
"systemd-run", "--user", "--scope", |
| 62 |
"-p", "CPUQuota=100%", |
| 63 |
"-p", "MemoryMax=50M", "-p", "MemorySwapMax=50M", |
| 64 |
"*" |
| 65 |
) |
| 66 |
) |
| 67 |
``` |
| 68 |
|
| 69 |
Note, depending on your CGroups configuration, a non-privileged user |
| 70 |
may or may not be able to set the CPU quota. If not, the `-p |
| 71 |
CPUQuota=100%` will be silently ignored. |
| 72 |
|
| 73 |
The 50 MiB memory limit is strict - if a worker uses more than this, |
| 74 |
the operating system will terminate the worker instantly. To |
| 75 |
illustrate what happens, we first start by generating 1 million |
| 76 |
numeric values each consuming 8 bytes, which in total consumes ~8 MB, |
| 77 |
and then calculate the mean, the memory consumption is within 50-MiB |
| 78 |
memory limit that each parallel worker has available; |
| 79 |
|
| 80 |
```r |
| 81 |
library(parallel) |
| 82 |
mu <- clusterEvalQ(cl, { x <- rnorm(n = 1e6); mean(x) }) |
| 83 |
mu <- unlist(mu) |
| 84 |
print(mu) |
| 85 |
#> [1] 0.0008072657 -0.0019693992 |
| 86 |
``` |
| 87 |
|
| 88 |
However, if we generate 10 times more values, the memory consumption |
| 89 |
will grow to at least 80 MB, which is over the 50-MiB memory limit, |
| 90 |
and we will get an error: |
| 91 |
|
| 92 |
```r |
| 93 |
mu <- clusterEvalQ(cl, { x <- rnorm(n = 10e6); mean(x) }) |
| 94 |
#> Error in unserialize(node$con) : error reading from connection |
| 95 |
``` |
| 96 |
|
| 97 |
This is because the operating system terminated the two background R |
| 98 |
processes, because they overused the memory. This is why the main R |
| 99 |
process no longer can communicate with the parallel workers. We can |
| 100 |
see that both workers are down, by calling: |
| 101 |
|
| 102 |
```r |
| 103 |
isNodeAlive(cl) |
| 104 |
#> [1] FALSE FALSE |
| 105 |
``` |
| 106 |
|
| 107 |
We can use `cloneNode()` to relaunch workers that are no longer |
| 108 |
alive, e.g. |
| 109 |
|
| 110 |
```r |
| 111 |
is_down <- !isNodeAlive(cl) |
| 112 |
cl[is_down] <- cloneNode(cl[is_down]) |
| 113 |
isNodeAlive(cl) |
| 114 |
#> [1] TRUE TRUE |
| 115 |
``` |
| 116 |
|
| 117 |
|
| 118 |
## Example: MS Windows parallel workers with specific CPU affinities |
| 119 |
|
| 120 |
This example works only on MS Windows machines. It launches four |
| 121 |
local workers, where two are running on CPU Group #0 and two on CPU |
| 122 |
Group #1. |
| 123 |
|
| 124 |
```r |
| 125 |
library(parallelly) |
| 126 |
rscript <- I(c( |
| 127 |
Sys.getenv("COMSPEC"), "/c", |
| 128 |
"start", "/B", |
| 129 |
"/NODE", cpu_group=NA_integer_, |
| 130 |
"/AFFINITY", "0xFFFFFFFFFFFFFFFE", |
| 131 |
"*") |
| 132 |
) |
| 133 |
|
| 134 |
rscript["cpu_group"] <- 0 |
| 135 |
cl_0 <- makeClusterPSOCK(2, rscript = rscript) |
| 136 |
|
| 137 |
rscript["cpu_group"] <- 1 |
| 138 |
cl_1 <- makeClusterPSOCK(2, rscript = rscript) |
| 139 |
|
| 140 |
cl <- c(cl_0, cl_1) |
| 141 |
``` |
| 142 |
|
| 143 |
The special `*` value expands to the proper `Rscript` on the machine |
| 144 |
where the parallel workers are launched. |
| 145 |
|
| 146 |
<!-- See also: https://lovickconsulting.com/2021/11/18/running-r-clusters-on-an-amd-threadripper-3990x-in-windows-10-2/ --> |