Debian Codegraph @

doc/source/design/module-cache.rst

raw
1 .. _module-cache:
2
3 Module cache
4 ============
5
6 Objective is to consolidate available module information in a few number of
7 files in order to process these files when doing global module queries like
8 done by :subcmd:`avail`, :subcmd:`spider` or :subcmd:`whatis` sub-commands
9 instead of walking through the whole modulepath directories to compute this
10 information.
11
12 With cache files, it is expected to have lesser IO load and more efficiently
13 handle environments with thousands modulefiles stored in a parallel
14 filesystem.
15
16 How to store cache
17 ------------------
18
19 Module cache will be a file aggregating available module information. Module
20 are organized into modulepaths that are enabled in user environment. Goal is
21 to have one module cache file per modulepath. This file will be stored under
22 the modulepath root with the ``.modulecache`` file name.
23
24 Depending on modulepath ownership, modulepath may or may not have a cache file
25 built. It seems more efficient to save cache on a per-modulepath basis as a
26 user may enable diverse modulepaths, some of them not maintained by the admins
27 of the machine they are running on.
28
29 Having a central system cache may miss some user-managed modulepaths.
30
31 Having a per-user cache file does not seem a good idea. If built for the user,
32 a lot more information may be pre-processed: dynamic information from the
33 modulerc files for instance may be processed at cache file creation, not when
34 using the cache file. But it cannot be expected from the user to build its
35 cache and keep it up to date. Having the cache on the user side can lead
36 to additional issue with users complaining to the support that they cannot
37 find a given module, but due to an outdated cache file.
38
39 In definitive, there will be one cache file per modulepath directory.
40
41 Cache content
42 -------------
43
44 As cache files should be used instead of walking through modulepath directory
45 to search for valid modulercs and modulefiles. Cache file should contain
46 everything to be autonomous to proceed ``avail``, ``spider`` or ``whatis``
47 queries.
48
49 Yet module search can be very dynamic as things set for instance in modulerc
50 files may dynamically produce new modulefiles or define properties specific to
51 some users or groups.
52
53 So the produced cache file cannot be the static view generated for the user
54 who has built the cache file. Thus dynamic shaping is preserved by storing
55 scripts and commands in the cache file.
56
57 Modulerc files may be highly dynamic so their full content is purely saved
58 into the cache file. An entry is set for each valid modulerc file. Same goes
59 for modulefiles as some conditional structure in their code can dynamically
60 change dependency, whatis or other definitions.
61
62 It is expected that modulefile will be evaluated on ``avail``, ``spider`` or
63 ``whatis`` commands for complex queries. So for such use case it is important
64 to have modulefile content cached.
65
66 Thus by reading the cache file, in one IO the content of all the modulerc and
67 modulefiles of the modulepath will be fetched.
68
69 Goal is to record in cache file every information obtained as a result of
70 ``findModules`` procedure. Thus cache file contains modulefile modification
71 times, which is reported on ``--long`` output. Modulefile validity check
72 result is also recorded if modulefile is not valid. Modulerc validity check
73 is not recorded as it is not an information produced by ``findModules``.
74
75 Note: modulercs and modulefiles are not tested when generating cache file to
76 see if they are valid, if their code is correctly executed. Their bare content
77 is recorded. Same result is obtained this way whether a cache file is used or
78 not.
79
80 Cache format
81 ------------
82
83 Goal is to make the cache file a Tcl script, like modulefiles. It will not
84 be exactly a modulefile as specific commands will be setup to evaluate cache
85 file and cache file will not make use of modulefile commands. See `Cache
86 evaluation`_ section for details on how a cache file is evaluated.
87
88 Cache file starts with a ``#%Module`` string like modulefiles. A version
89 number is appended next to these first characters (for instance
90 ``%Module5.3``). See `Cache validity`_ section for details on the cache file
91 *prefix* string.
92
93 Modulefile content in modulepath is defined in cache with the
94 ``modulefile-content`` command. Which takes in this order:
95
96 * relative file path
97 * file last modification date (as a Unix epoch time)
98 * file module header
99 * file body content.
100
101 For instance:
102
103 * ``modulefile-content foo/1.0 1234567890 #%Module {body}``
104 * or ``modulefile-content {foo/w s} 1234567890 #%Module5.2 {body}``
105
106 Similar command for modulerc: ``modulerc-content`` which accepts following
107 syntax:
108
109 * relative file path
110 * file module header
111 * file body content.
112
113 For instance: ``modulerc-content foo/.modulerc #%Module {body}``. Modification
114 time is not needed for modulerc as this information is not reported on an
115 ``avail``, ``spider`` sub-commands in long mode.
116
117 Every modulercs and modulefiles file contained in modulepath are recorded
118 in cache with ``modulerc-content`` or ``modulefile-content`` command.
119 Modulefiles whose name start with a dot character are also recorded in cache.
120
121 Invalid modulefiles are recorded in cache file through ``modulefile-invalid``
122 command rather ``modulefile-content``. It accepts following arguments:
123
124 * relative file path
125 * invalidity kind
126 * error message
127
128 For instance: ``modulefile-invalid foo/2.0 invalid {Magic cookie '#%Module'
129 missing}``
130
131 Files or directories that have limited access are recorded with specific
132 commands:
133
134 * ``limited-access-file foo/1.0``
135 * ``limited-access-directory foo``
136
137 Limited access means for a file that it cannot be read by user that builds
138 cache or other users. For a directory it means that it cannot be either read
139 or searched by user that builds cache or other users.
140
141 A modulefile or a modulerc is not recorded with ``modulefile-content`` or
142 ``modulerc-content`` if file or one of its parent directory has limited
143 access. This way only content that can be read by everyone is recorded into
144 cache file. Sensitive information are excluded from cache file.
145
146 Note: a cache file generated by a privileged user (which has access to every
147 thing) and a cache file generated by a less privileged user will be the same
148 as limited-access information will not be included in cache file.
149
150 Files or directories recorded as limited access will need to be tested (and
151 walked down for directories) when cache file will be evaluated. It is
152 important to distinguish files from directories to save some file stat test
153 to determine if an element is a directory when limited access elements will be
154 tested. No need for files to distinguish modulefile from modulerc as this
155 difference is visible with file name.
156
157 Note: Limited access tests are skipped on Windows platform as Unix-style file
158 permission cannot be tested there.
159
160 Recording full modulefile content or subset of elements
161 -------------------------------------------------------
162
163 It was initially drafted that only a subset of element of modulefiles would be
164 recorded in cache file, to reduce size of this file and reduce its evaluation
165 time.
166
167 Recording full modulefile content is in the end preferred as:
168
169 * this solution is simpler to implement
170 * cache file size is not too big in the end (~ hundreds of KB for a thousand
171 of modulefiles)
172 * evaluation time of large cache file is acceptable (time taken to evaluate
173 cannot be noticed by user)
174 * simpler to understand and manage for sites
175
176 Producing a cache entry for a modulefile with only a subset of commands
177 recorded (like variant or requirement) is only feasible for modulefiles not
178 using conditionals or specific evaluation scheme. Recording full content will
179 work in any scenario whereas recording a subset limits cache usage. Moreover
180 it is hard to determine, depending of the modulefile set, where cache can be
181 used or not if only a subset of elements is recorded.
182
183 Cache validity
184 --------------
185
186 From Modules 5.3 to 5.5
187 ^^^^^^^^^^^^^^^^^^^^^^^
188
189 Cache file header indicates a Modules version number. It corresponds to the
190 Modules version:
191
192 * the cache file has been built with
193 * the cache file is compatible with
194
195 It seems reasonable to ask for a cache file update every time Modules is
196 upgraded to a newer minor version (for instance from 5.3 to 5.4):
197
198 * it is simpler to understand for staff and user when the cache is taken into
199 account, when it is ignored
200 * better to ensure cache file is accurate for the Modules version as modulerc
201 and modulefile commands may evolve from one version to another
202
203 Starting Modules 5.6
204 ^^^^^^^^^^^^^^^^^^^^
205
206 Approach to cache validity is changed on Modules 5.6 to cover use cases where
207 different Modules installation uses the same set of modulepaths. This is for
208 instance the case when a service provides modulepaths with cache files in them
209 and just requires a minimal version of Modules. Or on a computing center, when
210 the same modulepaths are used by different supercomputers which may use
211 different version of Modules at some point in time.
212
213 With this change the cache file header still indicates a Modules version
214 number. But this version number now corresponds to the minimal Modules version
215 able to use this cache file.
216
217 With such change, backward compatibility is expected on module cache Tcl
218 commands. Addition of new commands will lead to an increase of the minimal
219 version required. Argument change to existing command is not possible, only
220 the introduction of optional argument will be ok (with increase of minimal
221 version required).
222
223 When generating cache file, minimal version required is set back to ``5.3`` as
224 no change occurred on the cache file format since the introduction of the
225 cache feature.
226
227 As minimal Modules version check code is the same for cache file than for
228 modulefile since the introduction of the cache feature, cache file has always
229 been taken into account as soon as the Modules version matches the minimal
230 requirement.
231
232 Cache usage
233 -----------
234
235 Any time a modulepath directory is opened to get its content, the module cache
236 file will be used instead if available.
237
238 Modulepath content analysis is performed by ``findModules`` procedures. So
239 any sub-command calling it (directly or through ``getModules`` or
240 ``getPathToModule``) will use the cache file. It corresponds to the following
241 sub-commands:
242
243 * lint
244 * paths
245 * search
246 * whatis
247 * aliases
248 * avail
249 * spider
250 * switch
251 * restore
252 * save
253 * display
254 * path
255 * source
256 * load
257 * test
258 * edit
259 * help
260
261 It may also occur during other sub-commands that evaluates modulefiles using
262 the ``is-avail`` command: like unload or refresh.
263
264 Cache files are ignored if :mconfig:`ignore_cache` configuration option is
265 enabled. This option can also be enabled just for one execution with the
266 :option:`--ignore-cache` command line option.
267
268 Cache files are also ignored if the currently running top-level sub-command
269 is part of the :mconfig:`ignore_cache_subcmds` configuration option. On
270 setup with a very large number of modulefiles, resulting in cache files of
271 several megabytes, evaluating the whole cache file may take longer for a
272 sub-command targeting specific modules (like :subcmd:`load`) than walking
273 through the modulepath directory to find these modules. This option helps
274 to skip cache use for such sub-commands while preserving cache benefit on
275 the others (like :subcmd:`avail`).
276
277 Sub-command name aliases resolve to their target sub-command prior the
278 comparison against the option value. The :command:`ml` command used without
279 a sub-command name translates to the :subcmd:`load` or :subcmd:`unload`
280 sub-command it is currently processing, so ``ml`` is not a value to set in
281 this option.
282
283 Cache files are ignored if cache expiry mechanism is enabled through the
284 :mconfig:`cache_expiry_secs` configuration option. When this option is set to
285 0, it means a cache file never expires. This is the default behavior. If set
286 to something else, cache file is expired if its last modification time is
287 older than the number of seconds defined in ``cache_expiry_secs``. Option
288 value is an integer between 1 and 31536000, which is the number of seconds
289 during 1 year.
290
291 Is there an impact at evaluating the full cache file rather making a directory
292 walk-through to find a module? Cache file is fully read, but not all the files
293 described in it are evaluated. Just those corresponding to the search, like it
294 is done when walking modulepath directory and evaluating only the modulerc
295 files corresponding to the query. So results between using cache file or not
296 should be the same: compared to a search without cache, no extra modulefile or
297 modulerc evaluation will be performed when a cache file is used.
298
299 As cache is recorded with both ``mcookie_check`` and ``mcookie_version_check``
300 options enabled, these two options are not honored (if disabled) when a cache
301 file is used. They are primarily useful to skip I/O tests when walking through
302 the content of a modulepath directory. As these I/O tests are done during the
303 cache build process, the options are useless when using cache files.
304
305 When cache file magic cookie defines a Modules version greater than the
306 current one, the cache file is silently ignored. Raisin error is not useful as
307 different version of Modules may be deployed in the same site environment.
308
309 When cache file is not in sync
310 ------------------------------
311
312 Files or directories are freely available through cache when used even if
313 after cache being built:
314
315 * their access is limited
316 * they are deleted
317 * their content changes and is not anymore valid
318
319 When files or directories have their access limited prior building cache, but
320 afterward these access limitations are lifted. These elements will require
321 an access test to check if they are available. This test will always be
322 successful as element accesses are not anymore limited.
323
324 If files or directories do not exist when cache is built, they will not be
325 found when cache is used.
326
327 If modulefile is recorded in cache as invalid, it will stay invalid if cache
328 is used even the modulefile is fixed. Cache need to be regenerated.
329
330 Read/write performances
331 -----------------------
332
333 :mconfig:`cache_buffer_bytes` configuration option defines size of the buffer
334 when reading or writing cache files.
335
336 With a bigger buffer, fewer read or write system calls are needed to read or
337 write cache file. On busy storage systems it can improve I/O performances.
338
339 Cache evaluation
340 ----------------
341
342 A Tcl sub-interpreter is created to analyze cache files. This sub-interp is
343 setup to evaluate cache file-specific commands, like ``modulefile-content``.
344
345 When evaluated, ``modulefile-content``, ``modulerc-content`` and
346 ``modulefile-invalid`` commands populate the read cache structure of modulerc
347 and modulefiles. This way when the modulefile for instance need to be read,
348 its content is already found in memory cache structure. It corresponds to the
349 following global variables:
350
351 * ``::g_modfileContent``
352 * ``::g_fileMtime`` (only for valid modulefile)
353 * ``::g_modfileValid`` (only for modulefile, valid or not)
354
355 In addition a ``::g_cacheModpath`` array is filled with an entry dedicated for
356 each modulepath. The content of this entry mimics the result list returned by
357 ``findModules`` procedures with information for the whole content of the
358 modulepath.
359
360 Limited access files and directories described in cache by
361 ``limited-access-file`` and ``limited-access-directory`` commands populate
362 specific structures to indicate some entries in modulepath have to be tested
363 (and walked down for directories) to determine if they are available to
364 current user:
365
366 * ``g_cacheFLimitedModpath``
367 * ``g_cacheDLimitedModpath``
368
369 These two structures are arrays with one entry per cached modulepath. Limited
370 access entries are tested if they match search query. Test is done through
371 ``findModulesFromDirsAndFiles`` procedure which corresponds to the walk down
372 code extracted from ``findModules``.
373
374 This specific interpreter is reused between different cache file evaluations.
375 As for modulefile interpreter, a consistency check is performed before each
376 reuse to test that the cache file-specific commands have not be rewritten
377 during previous cache file evaluation.
378
379 Cache file evaluation is tracked to avoid evaluating twice the same cache
380 file.
381
382 Cache evaluation stops if an erroneous command or syntax is encountered. Like
383 for erroneous modulerc, error is not reported during ``avail``, ``spider`` or
384 ``whatis`` commands unless if ran in debug mode. Error is reporting during a
385 ``load`` evaluation. Cache evaluation is considered failed if there is an
386 error in the cache file, thus a non-cache module search will occur instead of
387 relying on cache module listing. However descriptions of modulefile and
388 modulerc evaluated in cache prior the error occurs are retained.
389
390 cachebuild sub-command
391 ----------------------
392
393 :subcmd:`cachebuild` sub-command creates a module cache file in modulepaths.
394 Without arguments, it attempts to create cache in every enabled modulepaths
395 where running user has the right to write. If arguments are provided, cache
396 is build in the directories pointed by these arguments.
397
398 General properties:
399
400 * Shortcut name: none
401 * Accepted option: none
402 * Expected number of argument: 0 to N
403 * Accept boolean variant specification: no
404 * Parse module version specification: no
405 * Fully read modulefile when checking validity: yes
406 * Sub-command only called from top level: yes
407 * Lead to modulefile evaluation: yes (``cachebuild``)
408
409 An error is returned for each specified directories where current user has no
410 write access.
411
412 An error is returned if a modulefile or a modulerc cannot be read. This error
413 ends cache content generation for current modulepath. Build continues with
414 next modulepath after this error.
415
416 Modulepaths where current user has no write rights are skipped and reported
417 with a warning notice.
418
419 Reports a ``Creating <modulepath>`` block header message for each cache file
420 created or updated. This report is made when verbosity is set to ``normal``
421 or higher mode.
422
423 :mconfig:`mcookie_check` and :mconfig:`mcookie_version_check` options are both
424 enabled when recording cache. This is produced with exact same content whether
425 these options are enabled or not.
426
427 cacheclear sub-command
428 ----------------------
429
430 :subcmd:`cacheclear` sub-command deletes all module cache file in enabled
431 modulepaths.
432
433 General properties:
434
435 * Shortcut name: none
436 * Accepted option: none
437 * Expected number of argument: 0
438 * Accept boolean variant specification: no
439 * Parse module version specification: no
440 * Fully read modulefile when checking validity: no
441 * Sub-command only called from top level: yes
442 * Lead to modulefile evaluation: no
443
444 Modulepaths where current user has no write rights on the modulepath directory
445 are skipped and reported with a warning notice.
446
447 Reports a ``Deleting <modulepath>`` block header message for each cache file
448 created or updated. This report is made when verbosity is set to ``normal``
449 or higher mode.
450
451 .. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: