| 1 |
.. _module-cache: |
| 2 |
|
| 3 |
Module cache |
| 4 |
============ |
| 5 |
|
| 6 |
Objective is to consolidate available module information in a few number of |
| 7 |
files in order to process these files when doing global module queries like |
| 8 |
done by :subcmd:`avail`, :subcmd:`spider` or :subcmd:`whatis` sub-commands |
| 9 |
instead of walking through the whole modulepath directories to compute this |
| 10 |
information. |
| 11 |
|
| 12 |
With cache files, it is expected to have lesser IO load and more efficiently |
| 13 |
handle environments with thousands modulefiles stored in a parallel |
| 14 |
filesystem. |
| 15 |
|
| 16 |
How to store cache |
| 17 |
------------------ |
| 18 |
|
| 19 |
Module cache will be a file aggregating available module information. Module |
| 20 |
are organized into modulepaths that are enabled in user environment. Goal is |
| 21 |
to have one module cache file per modulepath. This file will be stored under |
| 22 |
the modulepath root with the ``.modulecache`` file name. |
| 23 |
|
| 24 |
Depending on modulepath ownership, modulepath may or may not have a cache file |
| 25 |
built. It seems more efficient to save cache on a per-modulepath basis as a |
| 26 |
user may enable diverse modulepaths, some of them not maintained by the admins |
| 27 |
of the machine they are running on. |
| 28 |
|
| 29 |
Having a central system cache may miss some user-managed modulepaths. |
| 30 |
|
| 31 |
Having a per-user cache file does not seem a good idea. If built for the user, |
| 32 |
a lot more information may be pre-processed: dynamic information from the |
| 33 |
modulerc files for instance may be processed at cache file creation, not when |
| 34 |
using the cache file. But it cannot be expected from the user to build its |
| 35 |
cache and keep it up to date. Having the cache on the user side can lead |
| 36 |
to additional issue with users complaining to the support that they cannot |
| 37 |
find a given module, but due to an outdated cache file. |
| 38 |
|
| 39 |
In definitive, there will be one cache file per modulepath directory. |
| 40 |
|
| 41 |
Cache content |
| 42 |
------------- |
| 43 |
|
| 44 |
As cache files should be used instead of walking through modulepath directory |
| 45 |
to search for valid modulercs and modulefiles. Cache file should contain |
| 46 |
everything to be autonomous to proceed ``avail``, ``spider`` or ``whatis`` |
| 47 |
queries. |
| 48 |
|
| 49 |
Yet module search can be very dynamic as things set for instance in modulerc |
| 50 |
files may dynamically produce new modulefiles or define properties specific to |
| 51 |
some users or groups. |
| 52 |
|
| 53 |
So the produced cache file cannot be the static view generated for the user |
| 54 |
who has built the cache file. Thus dynamic shaping is preserved by storing |
| 55 |
scripts and commands in the cache file. |
| 56 |
|
| 57 |
Modulerc files may be highly dynamic so their full content is purely saved |
| 58 |
into the cache file. An entry is set for each valid modulerc file. Same goes |
| 59 |
for modulefiles as some conditional structure in their code can dynamically |
| 60 |
change dependency, whatis or other definitions. |
| 61 |
|
| 62 |
It is expected that modulefile will be evaluated on ``avail``, ``spider`` or |
| 63 |
``whatis`` commands for complex queries. So for such use case it is important |
| 64 |
to have modulefile content cached. |
| 65 |
|
| 66 |
Thus by reading the cache file, in one IO the content of all the modulerc and |
| 67 |
modulefiles of the modulepath will be fetched. |
| 68 |
|
| 69 |
Goal is to record in cache file every information obtained as a result of |
| 70 |
``findModules`` procedure. Thus cache file contains modulefile modification |
| 71 |
times, which is reported on ``--long`` output. Modulefile validity check |
| 72 |
result is also recorded if modulefile is not valid. Modulerc validity check |
| 73 |
is not recorded as it is not an information produced by ``findModules``. |
| 74 |
|
| 75 |
Note: modulercs and modulefiles are not tested when generating cache file to |
| 76 |
see if they are valid, if their code is correctly executed. Their bare content |
| 77 |
is recorded. Same result is obtained this way whether a cache file is used or |
| 78 |
not. |
| 79 |
|
| 80 |
Cache format |
| 81 |
------------ |
| 82 |
|
| 83 |
Goal is to make the cache file a Tcl script, like modulefiles. It will not |
| 84 |
be exactly a modulefile as specific commands will be setup to evaluate cache |
| 85 |
file and cache file will not make use of modulefile commands. See `Cache |
| 86 |
evaluation`_ section for details on how a cache file is evaluated. |
| 87 |
|
| 88 |
Cache file starts with a ``#%Module`` string like modulefiles. A version |
| 89 |
number is appended next to these first characters (for instance |
| 90 |
``%Module5.3``). See `Cache validity`_ section for details on the cache file |
| 91 |
*prefix* string. |
| 92 |
|
| 93 |
Modulefile content in modulepath is defined in cache with the |
| 94 |
``modulefile-content`` command. Which takes in this order: |
| 95 |
|
| 96 |
* relative file path |
| 97 |
* file last modification date (as a Unix epoch time) |
| 98 |
* file module header |
| 99 |
* file body content. |
| 100 |
|
| 101 |
For instance: |
| 102 |
|
| 103 |
* ``modulefile-content foo/1.0 1234567890 #%Module {body}`` |
| 104 |
* or ``modulefile-content {foo/w s} 1234567890 #%Module5.2 {body}`` |
| 105 |
|
| 106 |
Similar command for modulerc: ``modulerc-content`` which accepts following |
| 107 |
syntax: |
| 108 |
|
| 109 |
* relative file path |
| 110 |
* file module header |
| 111 |
* file body content. |
| 112 |
|
| 113 |
For instance: ``modulerc-content foo/.modulerc #%Module {body}``. Modification |
| 114 |
time is not needed for modulerc as this information is not reported on an |
| 115 |
``avail``, ``spider`` sub-commands in long mode. |
| 116 |
|
| 117 |
Every modulercs and modulefiles file contained in modulepath are recorded |
| 118 |
in cache with ``modulerc-content`` or ``modulefile-content`` command. |
| 119 |
Modulefiles whose name start with a dot character are also recorded in cache. |
| 120 |
|
| 121 |
Invalid modulefiles are recorded in cache file through ``modulefile-invalid`` |
| 122 |
command rather ``modulefile-content``. It accepts following arguments: |
| 123 |
|
| 124 |
* relative file path |
| 125 |
* invalidity kind |
| 126 |
* error message |
| 127 |
|
| 128 |
For instance: ``modulefile-invalid foo/2.0 invalid {Magic cookie '#%Module' |
| 129 |
missing}`` |
| 130 |
|
| 131 |
Files or directories that have limited access are recorded with specific |
| 132 |
commands: |
| 133 |
|
| 134 |
* ``limited-access-file foo/1.0`` |
| 135 |
* ``limited-access-directory foo`` |
| 136 |
|
| 137 |
Limited access means for a file that it cannot be read by user that builds |
| 138 |
cache or other users. For a directory it means that it cannot be either read |
| 139 |
or searched by user that builds cache or other users. |
| 140 |
|
| 141 |
A modulefile or a modulerc is not recorded with ``modulefile-content`` or |
| 142 |
``modulerc-content`` if file or one of its parent directory has limited |
| 143 |
access. This way only content that can be read by everyone is recorded into |
| 144 |
cache file. Sensitive information are excluded from cache file. |
| 145 |
|
| 146 |
Note: a cache file generated by a privileged user (which has access to every |
| 147 |
thing) and a cache file generated by a less privileged user will be the same |
| 148 |
as limited-access information will not be included in cache file. |
| 149 |
|
| 150 |
Files or directories recorded as limited access will need to be tested (and |
| 151 |
walked down for directories) when cache file will be evaluated. It is |
| 152 |
important to distinguish files from directories to save some file stat test |
| 153 |
to determine if an element is a directory when limited access elements will be |
| 154 |
tested. No need for files to distinguish modulefile from modulerc as this |
| 155 |
difference is visible with file name. |
| 156 |
|
| 157 |
Note: Limited access tests are skipped on Windows platform as Unix-style file |
| 158 |
permission cannot be tested there. |
| 159 |
|
| 160 |
Recording full modulefile content or subset of elements |
| 161 |
------------------------------------------------------- |
| 162 |
|
| 163 |
It was initially drafted that only a subset of element of modulefiles would be |
| 164 |
recorded in cache file, to reduce size of this file and reduce its evaluation |
| 165 |
time. |
| 166 |
|
| 167 |
Recording full modulefile content is in the end preferred as: |
| 168 |
|
| 169 |
* this solution is simpler to implement |
| 170 |
* cache file size is not too big in the end (~ hundreds of KB for a thousand |
| 171 |
of modulefiles) |
| 172 |
* evaluation time of large cache file is acceptable (time taken to evaluate |
| 173 |
cannot be noticed by user) |
| 174 |
* simpler to understand and manage for sites |
| 175 |
|
| 176 |
Producing a cache entry for a modulefile with only a subset of commands |
| 177 |
recorded (like variant or requirement) is only feasible for modulefiles not |
| 178 |
using conditionals or specific evaluation scheme. Recording full content will |
| 179 |
work in any scenario whereas recording a subset limits cache usage. Moreover |
| 180 |
it is hard to determine, depending of the modulefile set, where cache can be |
| 181 |
used or not if only a subset of elements is recorded. |
| 182 |
|
| 183 |
Cache validity |
| 184 |
-------------- |
| 185 |
|
| 186 |
From Modules 5.3 to 5.5 |
| 187 |
^^^^^^^^^^^^^^^^^^^^^^^ |
| 188 |
|
| 189 |
Cache file header indicates a Modules version number. It corresponds to the |
| 190 |
Modules version: |
| 191 |
|
| 192 |
* the cache file has been built with |
| 193 |
* the cache file is compatible with |
| 194 |
|
| 195 |
It seems reasonable to ask for a cache file update every time Modules is |
| 196 |
upgraded to a newer minor version (for instance from 5.3 to 5.4): |
| 197 |
|
| 198 |
* it is simpler to understand for staff and user when the cache is taken into |
| 199 |
account, when it is ignored |
| 200 |
* better to ensure cache file is accurate for the Modules version as modulerc |
| 201 |
and modulefile commands may evolve from one version to another |
| 202 |
|
| 203 |
Starting Modules 5.6 |
| 204 |
^^^^^^^^^^^^^^^^^^^^ |
| 205 |
|
| 206 |
Approach to cache validity is changed on Modules 5.6 to cover use cases where |
| 207 |
different Modules installation uses the same set of modulepaths. This is for |
| 208 |
instance the case when a service provides modulepaths with cache files in them |
| 209 |
and just requires a minimal version of Modules. Or on a computing center, when |
| 210 |
the same modulepaths are used by different supercomputers which may use |
| 211 |
different version of Modules at some point in time. |
| 212 |
|
| 213 |
With this change the cache file header still indicates a Modules version |
| 214 |
number. But this version number now corresponds to the minimal Modules version |
| 215 |
able to use this cache file. |
| 216 |
|
| 217 |
With such change, backward compatibility is expected on module cache Tcl |
| 218 |
commands. Addition of new commands will lead to an increase of the minimal |
| 219 |
version required. Argument change to existing command is not possible, only |
| 220 |
the introduction of optional argument will be ok (with increase of minimal |
| 221 |
version required). |
| 222 |
|
| 223 |
When generating cache file, minimal version required is set back to ``5.3`` as |
| 224 |
no change occurred on the cache file format since the introduction of the |
| 225 |
cache feature. |
| 226 |
|
| 227 |
As minimal Modules version check code is the same for cache file than for |
| 228 |
modulefile since the introduction of the cache feature, cache file has always |
| 229 |
been taken into account as soon as the Modules version matches the minimal |
| 230 |
requirement. |
| 231 |
|
| 232 |
Cache usage |
| 233 |
----------- |
| 234 |
|
| 235 |
Any time a modulepath directory is opened to get its content, the module cache |
| 236 |
file will be used instead if available. |
| 237 |
|
| 238 |
Modulepath content analysis is performed by ``findModules`` procedures. So |
| 239 |
any sub-command calling it (directly or through ``getModules`` or |
| 240 |
``getPathToModule``) will use the cache file. It corresponds to the following |
| 241 |
sub-commands: |
| 242 |
|
| 243 |
* lint |
| 244 |
* paths |
| 245 |
* search |
| 246 |
* whatis |
| 247 |
* aliases |
| 248 |
* avail |
| 249 |
* spider |
| 250 |
* switch |
| 251 |
* restore |
| 252 |
* save |
| 253 |
* display |
| 254 |
* path |
| 255 |
* source |
| 256 |
* load |
| 257 |
* test |
| 258 |
* edit |
| 259 |
* help |
| 260 |
|
| 261 |
It may also occur during other sub-commands that evaluates modulefiles using |
| 262 |
the ``is-avail`` command: like unload or refresh. |
| 263 |
|
| 264 |
Cache files are ignored if :mconfig:`ignore_cache` configuration option is |
| 265 |
enabled. This option can also be enabled just for one execution with the |
| 266 |
:option:`--ignore-cache` command line option. |
| 267 |
|
| 268 |
Cache files are also ignored if the currently running top-level sub-command |
| 269 |
is part of the :mconfig:`ignore_cache_subcmds` configuration option. On |
| 270 |
setup with a very large number of modulefiles, resulting in cache files of |
| 271 |
several megabytes, evaluating the whole cache file may take longer for a |
| 272 |
sub-command targeting specific modules (like :subcmd:`load`) than walking |
| 273 |
through the modulepath directory to find these modules. This option helps |
| 274 |
to skip cache use for such sub-commands while preserving cache benefit on |
| 275 |
the others (like :subcmd:`avail`). |
| 276 |
|
| 277 |
Sub-command name aliases resolve to their target sub-command prior the |
| 278 |
comparison against the option value. The :command:`ml` command used without |
| 279 |
a sub-command name translates to the :subcmd:`load` or :subcmd:`unload` |
| 280 |
sub-command it is currently processing, so ``ml`` is not a value to set in |
| 281 |
this option. |
| 282 |
|
| 283 |
Cache files are ignored if cache expiry mechanism is enabled through the |
| 284 |
:mconfig:`cache_expiry_secs` configuration option. When this option is set to |
| 285 |
0, it means a cache file never expires. This is the default behavior. If set |
| 286 |
to something else, cache file is expired if its last modification time is |
| 287 |
older than the number of seconds defined in ``cache_expiry_secs``. Option |
| 288 |
value is an integer between 1 and 31536000, which is the number of seconds |
| 289 |
during 1 year. |
| 290 |
|
| 291 |
Is there an impact at evaluating the full cache file rather making a directory |
| 292 |
walk-through to find a module? Cache file is fully read, but not all the files |
| 293 |
described in it are evaluated. Just those corresponding to the search, like it |
| 294 |
is done when walking modulepath directory and evaluating only the modulerc |
| 295 |
files corresponding to the query. So results between using cache file or not |
| 296 |
should be the same: compared to a search without cache, no extra modulefile or |
| 297 |
modulerc evaluation will be performed when a cache file is used. |
| 298 |
|
| 299 |
As cache is recorded with both ``mcookie_check`` and ``mcookie_version_check`` |
| 300 |
options enabled, these two options are not honored (if disabled) when a cache |
| 301 |
file is used. They are primarily useful to skip I/O tests when walking through |
| 302 |
the content of a modulepath directory. As these I/O tests are done during the |
| 303 |
cache build process, the options are useless when using cache files. |
| 304 |
|
| 305 |
When cache file magic cookie defines a Modules version greater than the |
| 306 |
current one, the cache file is silently ignored. Raisin error is not useful as |
| 307 |
different version of Modules may be deployed in the same site environment. |
| 308 |
|
| 309 |
When cache file is not in sync |
| 310 |
------------------------------ |
| 311 |
|
| 312 |
Files or directories are freely available through cache when used even if |
| 313 |
after cache being built: |
| 314 |
|
| 315 |
* their access is limited |
| 316 |
* they are deleted |
| 317 |
* their content changes and is not anymore valid |
| 318 |
|
| 319 |
When files or directories have their access limited prior building cache, but |
| 320 |
afterward these access limitations are lifted. These elements will require |
| 321 |
an access test to check if they are available. This test will always be |
| 322 |
successful as element accesses are not anymore limited. |
| 323 |
|
| 324 |
If files or directories do not exist when cache is built, they will not be |
| 325 |
found when cache is used. |
| 326 |
|
| 327 |
If modulefile is recorded in cache as invalid, it will stay invalid if cache |
| 328 |
is used even the modulefile is fixed. Cache need to be regenerated. |
| 329 |
|
| 330 |
Read/write performances |
| 331 |
----------------------- |
| 332 |
|
| 333 |
:mconfig:`cache_buffer_bytes` configuration option defines size of the buffer |
| 334 |
when reading or writing cache files. |
| 335 |
|
| 336 |
With a bigger buffer, fewer read or write system calls are needed to read or |
| 337 |
write cache file. On busy storage systems it can improve I/O performances. |
| 338 |
|
| 339 |
Cache evaluation |
| 340 |
---------------- |
| 341 |
|
| 342 |
A Tcl sub-interpreter is created to analyze cache files. This sub-interp is |
| 343 |
setup to evaluate cache file-specific commands, like ``modulefile-content``. |
| 344 |
|
| 345 |
When evaluated, ``modulefile-content``, ``modulerc-content`` and |
| 346 |
``modulefile-invalid`` commands populate the read cache structure of modulerc |
| 347 |
and modulefiles. This way when the modulefile for instance need to be read, |
| 348 |
its content is already found in memory cache structure. It corresponds to the |
| 349 |
following global variables: |
| 350 |
|
| 351 |
* ``::g_modfileContent`` |
| 352 |
* ``::g_fileMtime`` (only for valid modulefile) |
| 353 |
* ``::g_modfileValid`` (only for modulefile, valid or not) |
| 354 |
|
| 355 |
In addition a ``::g_cacheModpath`` array is filled with an entry dedicated for |
| 356 |
each modulepath. The content of this entry mimics the result list returned by |
| 357 |
``findModules`` procedures with information for the whole content of the |
| 358 |
modulepath. |
| 359 |
|
| 360 |
Limited access files and directories described in cache by |
| 361 |
``limited-access-file`` and ``limited-access-directory`` commands populate |
| 362 |
specific structures to indicate some entries in modulepath have to be tested |
| 363 |
(and walked down for directories) to determine if they are available to |
| 364 |
current user: |
| 365 |
|
| 366 |
* ``g_cacheFLimitedModpath`` |
| 367 |
* ``g_cacheDLimitedModpath`` |
| 368 |
|
| 369 |
These two structures are arrays with one entry per cached modulepath. Limited |
| 370 |
access entries are tested if they match search query. Test is done through |
| 371 |
``findModulesFromDirsAndFiles`` procedure which corresponds to the walk down |
| 372 |
code extracted from ``findModules``. |
| 373 |
|
| 374 |
This specific interpreter is reused between different cache file evaluations. |
| 375 |
As for modulefile interpreter, a consistency check is performed before each |
| 376 |
reuse to test that the cache file-specific commands have not be rewritten |
| 377 |
during previous cache file evaluation. |
| 378 |
|
| 379 |
Cache file evaluation is tracked to avoid evaluating twice the same cache |
| 380 |
file. |
| 381 |
|
| 382 |
Cache evaluation stops if an erroneous command or syntax is encountered. Like |
| 383 |
for erroneous modulerc, error is not reported during ``avail``, ``spider`` or |
| 384 |
``whatis`` commands unless if ran in debug mode. Error is reporting during a |
| 385 |
``load`` evaluation. Cache evaluation is considered failed if there is an |
| 386 |
error in the cache file, thus a non-cache module search will occur instead of |
| 387 |
relying on cache module listing. However descriptions of modulefile and |
| 388 |
modulerc evaluated in cache prior the error occurs are retained. |
| 389 |
|
| 390 |
cachebuild sub-command |
| 391 |
---------------------- |
| 392 |
|
| 393 |
:subcmd:`cachebuild` sub-command creates a module cache file in modulepaths. |
| 394 |
Without arguments, it attempts to create cache in every enabled modulepaths |
| 395 |
where running user has the right to write. If arguments are provided, cache |
| 396 |
is build in the directories pointed by these arguments. |
| 397 |
|
| 398 |
General properties: |
| 399 |
|
| 400 |
* Shortcut name: none |
| 401 |
* Accepted option: none |
| 402 |
* Expected number of argument: 0 to N |
| 403 |
* Accept boolean variant specification: no |
| 404 |
* Parse module version specification: no |
| 405 |
* Fully read modulefile when checking validity: yes |
| 406 |
* Sub-command only called from top level: yes |
| 407 |
* Lead to modulefile evaluation: yes (``cachebuild``) |
| 408 |
|
| 409 |
An error is returned for each specified directories where current user has no |
| 410 |
write access. |
| 411 |
|
| 412 |
An error is returned if a modulefile or a modulerc cannot be read. This error |
| 413 |
ends cache content generation for current modulepath. Build continues with |
| 414 |
next modulepath after this error. |
| 415 |
|
| 416 |
Modulepaths where current user has no write rights are skipped and reported |
| 417 |
with a warning notice. |
| 418 |
|
| 419 |
Reports a ``Creating <modulepath>`` block header message for each cache file |
| 420 |
created or updated. This report is made when verbosity is set to ``normal`` |
| 421 |
or higher mode. |
| 422 |
|
| 423 |
:mconfig:`mcookie_check` and :mconfig:`mcookie_version_check` options are both |
| 424 |
enabled when recording cache. This is produced with exact same content whether |
| 425 |
these options are enabled or not. |
| 426 |
|
| 427 |
cacheclear sub-command |
| 428 |
---------------------- |
| 429 |
|
| 430 |
:subcmd:`cacheclear` sub-command deletes all module cache file in enabled |
| 431 |
modulepaths. |
| 432 |
|
| 433 |
General properties: |
| 434 |
|
| 435 |
* Shortcut name: none |
| 436 |
* Accepted option: none |
| 437 |
* Expected number of argument: 0 |
| 438 |
* Accept boolean variant specification: no |
| 439 |
* Parse module version specification: no |
| 440 |
* Fully read modulefile when checking validity: no |
| 441 |
* Sub-command only called from top level: yes |
| 442 |
* Lead to modulefile evaluation: no |
| 443 |
|
| 444 |
Modulepaths where current user has no write rights on the modulepath directory |
| 445 |
are skipped and reported with a warning notice. |
| 446 |
|
| 447 |
Reports a ``Deleting <modulepath>`` block header message for each cache file |
| 448 |
created or updated. This report is made when verbosity is set to ``normal`` |
| 449 |
or higher mode. |
| 450 |
|
| 451 |
.. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: |