| 1 |
.. _reduce-io-load: |
| 2 |
|
| 3 |
Reduce I/O load |
| 4 |
=============== |
| 5 |
|
| 6 |
With large :command:`module` setup containing hundreds or even thousands of |
| 7 |
modulefiles, the processing of commands :subcmd:`module avail<avail>` or |
| 8 |
:subcmd:`module load<load>`. This is especially the case when all these |
| 9 |
modulefiles are hosted in a shared filesystem mounted on all nodes of a |
| 10 |
supercomputer concurrently used by many users. |
| 11 |
|
| 12 |
Such slowness comes from the analysis of the file hierarchy under each enabled |
| 13 |
modulepath that determines what files are modulefiles, what are the module |
| 14 |
aliases or symbolic versions set on them, etc. This analysis generates a |
| 15 |
significant number of I/O operations that linearly grow with the number of |
| 16 |
modulefiles. Slowness can be more or less observed depending on the underlying |
| 17 |
storage system and the number of available modulefiles. |
| 18 |
|
| 19 |
This recipe provides examples of the features of Modules that could be |
| 20 |
leveraged to reduce the I/O load implied by modulepath analysis. It starts |
| 21 |
with the generation of an example modulefile setup that will be progressively |
| 22 |
tweaked to observe I/O load reduction. |
| 23 |
|
| 24 |
A general assumption is made here that the more I/O operations there are, the |
| 25 |
slower the :command:`module` commands could be. So reducing the number of |
| 26 |
these I/O operations leads to reducing the I/O load. |
| 27 |
|
| 28 |
The `strace`_ utility, the Linux syscall tracer, will be used to watch the |
| 29 |
I/O operations produced by each :command:`module` command run. Examples will |
| 30 |
focus on the :subcmd:`module avail<avail>` command, which is one of the |
| 31 |
most I/O intensive and widely utilized by users. |
| 32 |
|
| 33 |
.. _strace: https://strace.io/ |
| 34 |
|
| 35 |
Implementation |
| 36 |
-------------- |
| 37 |
|
| 38 |
Several features of Modules could be used to reduce the number of I/O |
| 39 |
operations to analyze the content of modulepath: |
| 40 |
|
| 41 |
* `Modules Tcl extension library`_ |
| 42 |
* `Modulepath rc file`_ |
| 43 |
* `mcookie_check configuration option`_ |
| 44 |
* `Virtual modules`_ |
| 45 |
* `Module cache`_ |
| 46 |
|
| 47 |
Each of the above features contributes to an I/O reduction as described in the |
| 48 |
following section. Combined use of all these features will give the biggest |
| 49 |
I/O operation save. |
| 50 |
|
| 51 |
Example setup |
| 52 |
------------- |
| 53 |
|
| 54 |
For this recipe, a full modulefile setup has to be generated and it will be |
| 55 |
progressively modified for the different Modules features that will be used. |
| 56 |
|
| 57 |
2 modulepaths are created with 15 module names in each, each provided in 4 |
| 58 |
different versions. In one modulepath a :file:`.version` rc file is set for |
| 59 |
each module name to determine a default version. In the other modulepath a |
| 60 |
:file:`.modulerc` file is set for each module name to define a symbolic |
| 61 |
version. |
| 62 |
|
| 63 |
.. code-block:: sh |
| 64 |
|
| 65 |
mkdir -p example/reduce-io-load/applications |
| 66 |
mkdir -p example/reduce-io-load/libraries |
| 67 |
# create dummy application modulefiles |
| 68 |
cd example/reduce-io-load/applications |
| 69 |
for n in a b c d e f g h i j k l m n o; do |
| 70 |
mkdir app$n; |
| 71 |
for v in 1.0 2.0 3.0 4.0; do |
| 72 |
echo '#%Module' >app$n/$v; |
| 73 |
done; |
| 74 |
# define default version with .version file |
| 75 |
echo '#%Module' >app$n/.version |
| 76 |
echo 'set ModulesVersion 2.0' >>app$n/.version |
| 77 |
done |
| 78 |
cd - |
| 79 |
# create dummy library modulefiles |
| 80 |
cd example/reduce-io-load/libraries |
| 81 |
for n in a b c d e f g h i j k l m n o; do |
| 82 |
mkdir lib$n; |
| 83 |
for v in 1.0 2.0 3.0 4.0; do |
| 84 |
echo '#%Module' >lib$n/$v; |
| 85 |
done; |
| 86 |
# define symbolic version with .modulerc file |
| 87 |
echo -e '#%Module' >lib$n/.modulerc |
| 88 |
echo -e "module-version lib$n/3.0 sym" >>lib$n/.modulerc |
| 89 |
done |
| 90 |
cd - |
| 91 |
|
| 92 |
Some non-modulefile files are added within these modulepaths to simulate |
| 93 |
documentation files left by mistake in these directories: |
| 94 |
|
| 95 |
.. code-block:: sh |
| 96 |
|
| 97 |
touch example/reduce-io-load/applications/appg/README |
| 98 |
touch example/reduce-io-load/applications/appl/README |
| 99 |
touch example/reduce-io-load/libraries/libg/README |
| 100 |
touch example/reduce-io-load/libraries/libl/README |
| 101 |
|
| 102 |
The file permission mode of some modulefiles is set to protect them from being |
| 103 |
read and simulate a hidden modulefile: |
| 104 |
|
| 105 |
.. code-block:: sh |
| 106 |
|
| 107 |
chmod a-r example/reduce-io-load/applications/appg/1.0 |
| 108 |
chmod a-r example/reduce-io-load/applications/appl/1.0 |
| 109 |
chmod a-r example/reduce-io-load/libraries/libg/1.0 |
| 110 |
chmod a-r example/reduce-io-load/libraries/libl/1.0 |
| 111 |
|
| 112 |
Additional modulefiles are created with a magic cookie header telling that |
| 113 |
they are not compatible with the current version of Modules: |
| 114 |
|
| 115 |
.. code-block:: sh |
| 116 |
|
| 117 |
echo '#%Module99' >example/reduce-io-load/applications/appg/5.0 |
| 118 |
echo '#%Module99' >example/reduce-io-load/applications/appl/5.0 |
| 119 |
echo '#%Module99' >example/reduce-io-load/libraries/libg/5.0 |
| 120 |
echo '#%Module99' >example/reduce-io-load/libraries/libl/5.0 |
| 121 |
|
| 122 |
Once all of the above steps done, we end up with a small scale regular |
| 123 |
modulefile setup containing 116 modulefiles available to us. |
| 124 |
|
| 125 |
.. parsed-literal:: |
| 126 |
|
| 127 |
:ps:`$` module purge |
| 128 |
:ps:`$` export MODULEPATH= |
| 129 |
:ps:`$` module use example/reduce-io-load/applications |
| 130 |
:ps:`$` module use example/reduce-io-load/libraries |
| 131 |
:ps:`$` module -o "" avail -t | wc -l |
| 132 |
116 |
| 133 |
|
| 134 |
|
| 135 |
Modules Tcl extension library |
| 136 |
----------------------------- |
| 137 |
|
| 138 |
Modules is shipped by default with a Tcl extension library that extends the |
| 139 |
Tcl language in order to provide more optimized I/O commands to read a file or |
| 140 |
a directory content than native Tcl commands do. |
| 141 |
|
| 142 |
**Compatible with Modules v4.3+** |
| 143 |
|
| 144 |
If we rebuild Modules without this library enabled, we will see the benefits |
| 145 |
of having it enabled: |
| 146 |
|
| 147 |
.. code-block:: sh |
| 148 |
|
| 149 |
make distclean |
| 150 |
./configure --disable-libtclenvmodules |
| 151 |
make modulecmd-test.tcl |
| 152 |
chmod +x modulecmd-test.tcl |
| 153 |
eval $(tclsh ./modulecmd-test.tcl bash autoinit) |
| 154 |
|
| 155 |
The :command:`strace` tool gives the number and the kind of I/O operations |
| 156 |
performed during the ``module avail`` command: |
| 157 |
|
| 158 |
.. parsed-literal:: |
| 159 |
|
| 160 |
:ps:`$` strace -f -c -S name -e trace=%file,%desc -U calls,errors,name \\ |
| 161 |
--silence=attach $MODULES_CMD bash avail 2>no_extlib.out |
| 162 |
|
| 163 |
Once this first stat output is obtained, rebuild Modules with Tcl extension |
| 164 |
library enabled and fetch :command:`icdiff` tool to compare results. |
| 165 |
|
| 166 |
.. code-block:: sh |
| 167 |
|
| 168 |
make distclean |
| 169 |
./configure --enable-libtclenvmodules |
| 170 |
make modulecmd-test.tcl lib/libtclenvmodules.so |
| 171 |
chmod +x modulecmd-test.tcl |
| 172 |
make icdiff |
| 173 |
|
| 174 |
Then collect stats and compare results obtained: |
| 175 |
|
| 176 |
.. parsed-literal:: |
| 177 |
|
| 178 |
:ps:`$` strace -f -c -S name -e trace=%file,%desc -U calls,errors,name \\ |
| 179 |
--silence=attach $MODULES_CMD bash avail 2>with_extlib.out |
| 180 |
:ps:`$` ./icdiff --cols=76 no_extlib.out with_extlib.out |
| 181 |
:sgrhi:`no_extlib.out` :sgrhi:`with_extlib.out` |
| 182 |
calls errors syscall calls errors syscall |
| 183 |
--------- --------- ---------------- --------- --------- ---------------- |
| 184 |
:sgrb:`36` 2 access :sgra:`37` 2 access |
| 185 |
:sgrb:`248` close :sgra:`217` close |
| 186 |
2 dup2 2 dup2 |
| 187 |
8 6 execve 8 6 execve |
| 188 |
:sgrb:`166` :sgrhi:`fcntl` :sgra:`12` :sgrhi:`fcntl` |
| 189 |
:sgrb:`1` getcwd :sgra:`2` getcwd |
| 190 |
:sgrb:`128` :sgrhi:`getdents64` :sgra:`64` :sgrhi:`getdents64` |
| 191 |
:sgrb:`166 161` :sgrhi:`ioctl` :sgra:`12 7` :sgrhi:`ioctl` |
| 192 |
9 4 lseek 9 4 lseek |
| 193 |
:sgrb:`50` mmap :sgra:`54` mmap |
| 194 |
:sgrb:`86` newfstatat :sgra:`55` newfstatat |
| 195 |
:sgrb:`250` 10 openat :sgra:`219` 10 openat |
| 196 |
2 pipe 2 pipe |
| 197 |
10 pread64 10 pread64 |
| 198 |
:sgrb:`354` read :sgra:`355` read |
| 199 |
:sgrb:`1452 1452` :sgrhi:`readlink` :sgra:`25 25` :sgrhi:`readlink` |
| 200 |
:sgrb:`354` 2 :sgrhi:`stat` :sgra:`196` 2 :sgrhi:`stat` |
| 201 |
1 unlink 1 unlink |
| 202 |
20 write 20 write |
| 203 |
--------- --------- ---------------- --------- --------- ---------------- |
| 204 |
:sgrb:`3335 1637` :sgrhi:`total` :sgra:`1292 56` :sgrhi:`total` |
| 205 |
|
| 206 |
Modules Tcl extension library greatly reduces the number of filesystem I/O |
| 207 |
operations by removing unneeded ``ioctl``, ``fcntl`` and ``readlink`` system |
| 208 |
calls done (by Tcl ``open`` command) to read each file. Directory content read |
| 209 |
is also improved by fetching hidden and regular files in one pass, which |
| 210 |
divides by 2 the number of ``getdents`` call. ``stat`` calls are also reduced |
| 211 |
as files found in directories are not checked prior attempting to opening |
| 212 |
them. |
| 213 |
|
| 214 |
Modulepath rc file |
| 215 |
------------------ |
| 216 |
|
| 217 |
A ``.modulerc`` file found at the root of an enabled modulepath directory is |
| 218 |
now evaluated when modulepath is walked through to locate modulefiles. This |
| 219 |
file could hold the rc definition of the whole modules located in the |
| 220 |
modulepath, instead of having specific ``.modulerc`` or ``.version`` file for |
| 221 |
each module directory within the modulepath. |
| 222 |
|
| 223 |
**Compatible with Modules v4.3+** |
| 224 |
|
| 225 |
Let's migrate the ``.modulerc`` definition under each module directory in the |
| 226 |
``.modulerc`` file at the root of the modulepath directory. And also translate |
| 227 |
the content of ``.version`` files in :mfcmd:`module-version` commands that |
| 228 |
could be stored in this top-level rc file. Then all the ``.modulerc`` and |
| 229 |
``.version`` files under module directories are deleted to only keep one |
| 230 |
``.modulerc`` per modulepath. |
| 231 |
|
| 232 |
.. code-block:: sh |
| 233 |
|
| 234 |
cd example/reduce-io-load/applications |
| 235 |
echo '#%Module' >.modulerc |
| 236 |
for n in *; do |
| 237 |
v=$(grep set $n/.version | cut -d ' ' -f 3); |
| 238 |
echo "module-version $n/$v default" >>.modulerc; |
| 239 |
rm -f $n/.version |
| 240 |
done |
| 241 |
cd - |
| 242 |
cd example/reduce-io-load/libraries |
| 243 |
echo '#%Module' >.modulerc |
| 244 |
for n in *; do |
| 245 |
grep module-version $n/.modulerc >>.modulerc; |
| 246 |
rm -f $n/.modulerc |
| 247 |
done |
| 248 |
cd - |
| 249 |
|
| 250 |
Once this change on the module trees has been done, collect new statistics and |
| 251 |
compare them to those generated previously. |
| 252 |
|
| 253 |
.. parsed-literal:: |
| 254 |
|
| 255 |
:ps:`$` strace -f -c -S name -e trace=%file,%desc -U calls,errors,name \\ |
| 256 |
--silence=attach $MODULES_CMD bash avail 2>with_modulepath_rc.out |
| 257 |
:ps:`$` ./icdiff --cols=76 with_extlib.out with_modulepath_rc.out |
| 258 |
:sgrhi:`with_extlib.out` :sgrhi:`with_modulepath_rc.out` |
| 259 |
calls errors syscall calls errors syscall |
| 260 |
--------- --------- ---------------- --------- --------- ---------------- |
| 261 |
:sgrb:`37` 2 :sgrhi:`access` :sgra:`9` 2 :sgrhi:`access` |
| 262 |
:sgrb:`217` :sgrhi:`close` :sgra:`189` :sgrhi:`close` |
| 263 |
2 dup2 2 dup2 |
| 264 |
8 6 execve 8 6 execve |
| 265 |
12 fcntl 12 fcntl |
| 266 |
2 getcwd 2 getcwd |
| 267 |
64 getdents64 64 getdents64 |
| 268 |
12 7 ioctl 12 7 ioctl |
| 269 |
9 4 lseek 9 4 lseek |
| 270 |
54 mmap 54 mmap |
| 271 |
55 newfstatat 55 newfstatat |
| 272 |
:sgrb:`219` 10 :sgrhi:`openat` :sgra:`191` 10 :sgrhi:`openat` |
| 273 |
2 pipe 2 pipe |
| 274 |
10 pread64 10 pread64 |
| 275 |
:sgrb:`355` :sgrhi:`read` :sgra:`299` :sgrhi:`read` |
| 276 |
25 25 readlink 25 25 readlink |
| 277 |
:sgrb:`196` 2 :sgrhi:`stat` :sgra:`168` 2 :sgrhi:`stat` |
| 278 |
1 unlink 1 unlink |
| 279 |
12 write 12 write |
| 280 |
--------- --------- ---------------- --------- --------- ---------------- |
| 281 |
:sgrb:`1292` 56 :sgrhi:`total` :sgra:`1124` 56 :sgrhi:`total` |
| 282 |
|
| 283 |
With this change we have saved the ``access``, ``stat``, ``open``, ``read`` |
| 284 |
and ``close`` calls needed to analyze the 15 ``.modulerc`` and 15 ``.version`` |
| 285 |
files that have been removed and replaced by 2 top-level ``.modulerc`` files. |
| 286 |
|
| 287 |
mcookie_check configuration option |
| 288 |
---------------------------------- |
| 289 |
|
| 290 |
A new configuration option named :mconfig:`mcookie_check` has been introduced |
| 291 |
to control the verification made to files to determine if they are |
| 292 |
modulefiles. By default this configuration option is set to ``always`` and |
| 293 |
when searching for modulefiles within enabled modulepaths each file below |
| 294 |
these directories is opened to check if it starts with the Modules magic |
| 295 |
cookie (i.e., ``#%Module`` file signature). |
| 296 |
|
| 297 |
These historical checks lead to a large number of I/O operations but if the |
| 298 |
option is set to the ``eval`` value, files are not checked anymore when |
| 299 |
searching for modulefiles, only when evaluating them. |
| 300 |
|
| 301 |
**Compatible with Modules v5.1+** |
| 302 |
|
| 303 |
Here we are setting the :mconfig:`mcookie_check` configuration to the ``eval`` |
| 304 |
value to skip all test on files hosted in modulepath directories and see what |
| 305 |
happens when listing available modulefiles: |
| 306 |
|
| 307 |
.. parsed-literal:: |
| 308 |
|
| 309 |
:ps:`$` module config mcookie_check eval |
| 310 |
:ps:`$` module -o "" avail -t | wc -l |
| 311 |
128 |
| 312 |
:ps:`$` module avail -o "" appg appl libg libl |
| 313 |
appg/1.0 appg/2.0 appg/3.0 appg/4.0 appg/5.0 appg/README |
| 314 |
|
| 315 |
appl/1.0 appl/2.0 appl/3.0 appl/4.0 appl/5.0 appl/README |
| 316 |
|
| 317 |
libg/1.0 libg/2.0 libg/3.0(sym) libg/4.0 libg/5.0 libg/README |
| 318 |
|
| 319 |
libl/1.0 libl/2.0 libl/3.0(sym) libl/4.0 libl/5.0 libl/README |
| 320 |
|
| 321 |
12 more modulefiles are returned. Indeed the file checks done by default are |
| 322 |
useful but we could disable them if a special care is given to the content of |
| 323 |
modulepath directories. First, we have to eliminate the non-modulefiles stored |
| 324 |
in these directories: |
| 325 |
|
| 326 |
.. code-block:: sh |
| 327 |
|
| 328 |
rm -f example/reduce-io-load/applications/appg/README |
| 329 |
rm -f example/reduce-io-load/applications/appl/README |
| 330 |
rm -f example/reduce-io-load/libraries/libg/README |
| 331 |
rm -f example/reduce-io-load/libraries/libl/README |
| 332 |
|
| 333 |
Then a :mfcmd:`module-hide` command in modulepath's ``.modulerc`` file should |
| 334 |
be added for each the modulefile that is read-protected through the use of |
| 335 |
file permission modes. This way :command:`module` will know that modulefile is |
| 336 |
hidden. |
| 337 |
|
| 338 |
.. code-block:: sh |
| 339 |
|
| 340 |
cd example/reduce-io-load/applications |
| 341 |
echo 'module-hide --hard appg/1.0' >>.modulerc |
| 342 |
echo 'module-hide --hard appl/1.0' >>.modulerc |
| 343 |
cd - |
| 344 |
cd example/reduce-io-load/libraries |
| 345 |
echo 'module-hide --hard libg/1.0' >>.modulerc |
| 346 |
echo 'module-hide --hard libl/1.0' >>.modulerc |
| 347 |
cd - |
| 348 |
|
| 349 |
A :mfcmd:`module-hide` command in modulepath's ``.modulerc`` file should also |
| 350 |
be defined for each modulefile requiring a specific Modules version in its |
| 351 |
magic cookie. :command:`module` will skip such modulefiles in case its version |
| 352 |
is not compatible with them. |
| 353 |
|
| 354 |
.. code-block:: sh |
| 355 |
|
| 356 |
cd example/reduce-io-load/applications |
| 357 |
echo 'if {$ModuleToolVersion < 99} {' >>.modulerc |
| 358 |
echo ' module-hide --hard appg/5.0' >>.modulerc |
| 359 |
echo ' module-hide --hard appl/5.0' >>.modulerc |
| 360 |
echo '}' >>.modulerc |
| 361 |
cd - |
| 362 |
cd example/reduce-io-load/libraries |
| 363 |
echo 'if {$ModuleToolVersion < 99} {' >>.modulerc |
| 364 |
echo ' module-hide --hard libg/5.0' >>.modulerc |
| 365 |
echo ' module-hide --hard libl/5.0' >>.modulerc |
| 366 |
echo '}' >>.modulerc |
| 367 |
cd - |
| 368 |
|
| 369 |
Once this specific setup has been achieved to get in full control of the |
| 370 |
content of each modulepaths, we get the accurate listing result with the |
| 371 |
*eval* mode of :mconfig:`mcookie_check`. |
| 372 |
|
| 373 |
.. parsed-literal:: |
| 374 |
|
| 375 |
:ps:`$` module -o "" avail -t | wc -l |
| 376 |
116 |
| 377 |
:ps:`$` strace -f -c -S name -e trace=%file,%desc -U calls,errors,name \\ |
| 378 |
--silence=attach $MODULES_CMD bash avail 2>mcookie_check_eval.out |
| 379 |
:ps:`$` ./icdiff --cols=76 with_modulepath_rc.out mcookie_check_eval.out |
| 380 |
:sgrhi:`with_modulepath_rc.out` :sgrhi:`mcookie_check_eval.out` |
| 381 |
calls errors syscall calls errors syscall |
| 382 |
--------- --------- ---------------- --------- --------- ---------------- |
| 383 |
9 2 access 9 2 access |
| 384 |
:sgrb:`189` :sgrhi:`close` :sgra:`65` :sgrhi:`close` |
| 385 |
2 dup2 2 dup2 |
| 386 |
8 6 execve 8 6 execve |
| 387 |
12 fcntl 12 fcntl |
| 388 |
2 getcwd 2 getcwd |
| 389 |
64 getdents64 64 getdents64 |
| 390 |
12 7 ioctl 12 7 ioctl |
| 391 |
9 4 lseek 9 4 lseek |
| 392 |
54 mmap 54 mmap |
| 393 |
55 newfstatat 55 newfstatat |
| 394 |
:sgrb:`191 10` :sgrhi:`openat` :sgra:`63` 6 :sgrhi:`openat` |
| 395 |
2 pipe 2 pipe |
| 396 |
10 pread64 10 pread64 |
| 397 |
:sgrb:`299` :sgrhi:`read` :sgra:`175` :sgrhi:`read` |
| 398 |
25 25 readlink 25 25 readlink |
| 399 |
:sgrb:`168` 2 stat :sgra:`164` 2 stat |
| 400 |
1 unlink 1 unlink |
| 401 |
12 write 12 write |
| 402 |
--------- --------- ---------------- --------- --------- ---------------- |
| 403 |
:sgrb:`1124 56` :sgrhi:`total` :sgra:`744 52` :sgrhi:`total` |
| 404 |
|
| 405 |
A significant drop in I/O calls can be observed from this finely tuned setup. |
| 406 |
``open``, ``read`` and ``close`` calls are saved for the 4 non-modulefiles |
| 407 |
deleted, the 8 modulefiles made hidden and of course for the 116 regular |
| 408 |
modulefiles available. As demonstrated, such gain requires a complete control |
| 409 |
over the modulepath content and specific care for unsupported or |
| 410 |
read-protected files. |
| 411 |
|
| 412 |
Virtual modules |
| 413 |
--------------- |
| 414 |
|
| 415 |
A virtual module stands for a module name associated to a modulefile. Instead |
| 416 |
of looking for files under modulepaths to get modulefiles, a virtual module |
| 417 |
is defined in ``.modulerc`` file with the :mfcmd:`module-virtual` modulefile |
| 418 |
command which saves walk down I/O operations to analyze modulepath directory |
| 419 |
content. |
| 420 |
|
| 421 |
**Compatible with Modules v4.1+** |
| 422 |
|
| 423 |
Let's create 2 new modulepaths that will only contain a ``.modulerc`` file in |
| 424 |
which a virtual module is defined for each existing modulefile in initial |
| 425 |
modulepath. Content of the ``.modulerc`` in the initial modulepaths is also |
| 426 |
copied in the ``.modulerc`` of the *virtual* modulepaths. |
| 427 |
|
| 428 |
.. code-block:: sh |
| 429 |
|
| 430 |
mkdir example/reduce-io-load/applications-virt |
| 431 |
mkdir example/reduce-io-load/libraries-virt |
| 432 |
cd example/reduce-io-load/applications |
| 433 |
echo '#%Module' >../applications-virt/.modulerc |
| 434 |
for mod in */*; do |
| 435 |
echo "module-virtual $mod ../applications/$mod" \ |
| 436 |
>>../applications-virt/.modulerc; |
| 437 |
done |
| 438 |
grep -v '#%Module' .modulerc >>../applications-virt/.modulerc |
| 439 |
cd - |
| 440 |
cd example/reduce-io-load/libraries |
| 441 |
echo '#%Module' >../libraries-virt/.modulerc |
| 442 |
for mod in */*; do |
| 443 |
echo "module-virtual $mod ../libraries/$mod" \ |
| 444 |
>>../libraries-virt/.modulerc; |
| 445 |
done |
| 446 |
grep -v '#%Module' .modulerc >>../libraries-virt/.modulerc |
| 447 |
cd - |
| 448 |
|
| 449 |
Once the setup of the virtual modulepaths is finished, the environment of the |
| 450 |
:command:`module` command has to be changed to use these new modulepaths |
| 451 |
instead of the original ones. |
| 452 |
|
| 453 |
.. parsed-literal:: |
| 454 |
|
| 455 |
:ps:`$` module unuse example/reduce-io-load/applications |
| 456 |
:ps:`$` module unuse example/reduce-io-load/libraries |
| 457 |
:ps:`$` module use example/reduce-io-load/applications-virt |
| 458 |
:ps:`$` module use example/reduce-io-load/libraries-virt |
| 459 |
|
| 460 |
Then we can check we obtain the same output as with the original setup, 116 |
| 461 |
modulefiles available. After that collect I/O operation statistics and compare |
| 462 |
them to those previously fetched. |
| 463 |
|
| 464 |
.. parsed-literal:: |
| 465 |
|
| 466 |
:ps:`$` module -o "" avail -t | wc -l |
| 467 |
116 |
| 468 |
:ps:`$` strace -f -c -S name -e trace=%file,%desc -U calls,errors,name \\ |
| 469 |
--silence=attach $MODULES_CMD bash avail 2>with_virtual_modules.out |
| 470 |
:ps:`$` ./icdiff --cols=76 mcookie_check_eval.out with_virtual_modules.out |
| 471 |
:sgrhi:`mcookie_check_eval.out` :sgrhi:`with_virtual_modules.out` |
| 472 |
... |
| 473 |
calls errors syscall calls errors syscall |
| 474 |
--------- --------- ---------------- --------- --------- ---------------- |
| 475 |
9 2 access 9 2 access |
| 476 |
:sgrb:`65` :sgrhi:`close` :sgra:`35` :sgrhi:`close` |
| 477 |
2 dup2 2 dup2 |
| 478 |
8 6 execve 8 6 execve |
| 479 |
12 fcntl 12 fcntl |
| 480 |
2 getcwd 2 getcwd |
| 481 |
:sgrb:`64` :sgrhi:`getdents64` :sgra:`4` :sgrhi:`getdents64` |
| 482 |
12 7 ioctl 12 7 ioctl |
| 483 |
9 4 lseek 9 4 lseek |
| 484 |
54 mmap 54 mmap |
| 485 |
:sgrb:`55` :sgrhi:`newfstatat` :sgra:`25` :sgrhi:`newfstatat` |
| 486 |
:sgrb:`63` 6 :sgrhi:`openat` :sgra:`33` 6 :sgrhi:`openat` |
| 487 |
2 pipe 2 pipe |
| 488 |
10 pread64 10 pread64 |
| 489 |
175 read 175 read |
| 490 |
25 25 readlink 25 25 readlink |
| 491 |
:sgrb:`164` 2 :sgrhi:`stat` :sgra:`10` 2 :sgrhi:`stat` |
| 492 |
1 unlink 1 unlink |
| 493 |
12 write 12 write |
| 494 |
--------- --------- ---------------- --------- --------- ---------------- |
| 495 |
:sgrb:`744` 52 :sgrhi:`total` :sgra:`440` 52 :sgrhi:`total` |
| 496 |
|
| 497 |
A large I/O operation drop is observed with the *virtual* modulepath setup. |
| 498 |
The analysis of the 15 module directories under each of the 2 original |
| 499 |
modulepaths is not anymore needed as the ``.modulerc`` in the 2 *virtual* |
| 500 |
modulepaths already point to the modulefile location. ``stat``, ``open``, |
| 501 |
``getdents`` and ``close`` I/O calls are saved due to that. |
| 502 |
|
| 503 |
Module cache |
| 504 |
------------ |
| 505 |
|
| 506 |
Module cache can be built for every modulepaths with :subcmd:`cachebuild` |
| 507 |
sub-command. When a cache file is found for an enabled modulepath, this file |
| 508 |
is evaluated instead of walking down the content of the modulepath directory. |
| 509 |
|
| 510 |
**Compatible with Modules v5.3+** |
| 511 |
|
| 512 |
Here we start over the module setup at the end of the `Modulepath rc file`_ |
| 513 |
section, restoring :mconfig:`mcookie_check` configuration to default, building |
| 514 |
cache and setting cache read buffer to maximum value. |
| 515 |
|
| 516 |
.. parsed-literal:: |
| 517 |
|
| 518 |
:ps:`$` module unuse example/reduce-io-load/applications-virt |
| 519 |
:ps:`$` module unuse example/reduce-io-load/libraries-virt |
| 520 |
:ps:`$` module use example/reduce-io-load/applications |
| 521 |
:ps:`$` module use example/reduce-io-load/libraries |
| 522 |
:ps:`$` module config mcookie_check always |
| 523 |
:ps:`$` module cachebuild |
| 524 |
Creating :sgrhi:`example/reduce-io-load/libraries/.modulecache` |
| 525 |
Creating :sgrhi:`example/reduce-io-load/applications/.modulecache` |
| 526 |
:ps:`$` module config cache_buffer_bytes 1000000 |
| 527 |
|
| 528 |
Once things are setup, new statistics are collected and compared between when |
| 529 |
cache is used or when it is ignored. |
| 530 |
|
| 531 |
.. parsed-literal:: |
| 532 |
|
| 533 |
:ps:`$` module -o "" avail -t | wc -l |
| 534 |
116 |
| 535 |
:ps:`$` strace -f -c -S name -e trace=%file,%desc -U calls,errors,name \\ |
| 536 |
--silence=attach $MODULES_CMD bash avail 2>with_cache.out |
| 537 |
:ps:`$` strace -f -c -S name -e trace=%file,%desc -U calls,errors,name \\ |
| 538 |
--silence=attach $MODULES_CMD bash avail --ignore-cache 2>no_cache.out |
| 539 |
:ps:`$` ./icdiff --cols=76 no_cache.out with_cache.out |
| 540 |
:sgrhi:`no_cache.out` :sgrhi:`with_cache.out` |
| 541 |
calls errors syscall calls errors syscall |
| 542 |
--------- --------- ---------------- --------- --------- ---------------- |
| 543 |
:sgrb:`9` 2 access :sgra:`10` 2 access |
| 544 |
:sgrb:`183` :sgrhi:`close` :sgra:`29` :sgrhi:`close` |
| 545 |
2 dup2 2 dup2 |
| 546 |
9 7 execve 9 7 execve |
| 547 |
:sgrb:`12` fcntl :sgra:`15` fcntl |
| 548 |
1 getcwd 1 getcwd |
| 549 |
:sgrb:`64` :sgrhi:`getdents64` |
| 550 |
:sgrb:`12 7` ioctl :sgra:`15 10` ioctl |
| 551 |
:sgrb:`9` 4 lseek :sgra:`10` 4 lseek |
| 552 |
:sgrb:`37` mmap :sgra:`38` mmap |
| 553 |
:sgrb:`221` 2 :sgrhi:`newfstatat` :sgra:`30` 2 :sgrhi:`newfstatat` |
| 554 |
:sgrb:`187` 10 :sgrhi:`openat` :sgra:`33` 10 :sgrhi:`openat` |
| 555 |
2 pipe2 2 pipe2 |
| 556 |
10 pread64 10 pread64 |
| 557 |
:sgrb:`324` :sgrhi:`read` :sgra:`209` :sgrhi:`read` |
| 558 |
:sgrb:`35 35` readlink :sgra:`54 54` readlink |
| 559 |
1 unlink 1 unlink |
| 560 |
12 write 12 write |
| 561 |
--------- --------- ---------------- --------- --------- ---------------- |
| 562 |
:sgrb:`1130 67` :sgrhi:`total` :sgra:`480 89` :sgrhi:`total` |
| 563 |
|
| 564 |
When a cache is found, one file is read instead of checking all directories |
| 565 |
and files in modulepath directory. Only the modulefiles and directories that |
| 566 |
are not accessible for everyone are live checked after reading cache file to |
| 567 |
see if these elements are available to current user. |
| 568 |
|
| 569 |
It explains the significant I/O call drop that can be observed here. Some I/O |
| 570 |
calls are slightly higher due to the evaluation of module cache files. |
| 571 |
|
| 572 |
|
| 573 |
Wrap-up |
| 574 |
------- |
| 575 |
|
| 576 |
Combining all the 4 first features or last one detailed above leads to a |
| 577 |
significant drop in I/O operations. Almost all remaining I/O calls are made |
| 578 |
for the initialization of the :command:`module` command run. |
| 579 |
|
| 580 |
It is advised to run this recipe code on your setup to observe the I/O load |
| 581 |
gain you could obtain. As said earlier the less I/O operations there are, the |
| 582 |
faster the :command:`module` command could be. But this highly depends on your |
| 583 |
storage system, on the number of modulefiles and on the number of active |
| 584 |
users. You may not notice a big difference if your modulefiles are installed |
| 585 |
on a local SSD storage whereas it can be a game changer if instead the |
| 586 |
modulefiles are hosted on a shared HDD filesystem that is accessed by |
| 587 |
hundreds of users. |
| 588 |
|
| 589 |
.. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: |