Debian Codegraph @

doc/source/design/module-cache.rst

raw
1 .. _module-cache:
2
3 Module cache
4 ============
5
6 Objective is to consolidate available module information in a few number of
7 files in order to process these files when doing global module queries like
8 done by :subcmd:`avail`, :subcmd:`spider` or :subcmd:`whatis` sub-commands
9 instead of walking through the whole modulepath directories to compute this
10 information.
11
12 With cache files, it is expected to have lesser IO load and more efficiently
13 handle environments with thousands modulefiles stored in a parallel
14 filesystem.
15
16 How to store cache
17 ------------------
18
19 Module cache will be a file aggregating available module information. Module
20 are organized into modulepaths that are enabled in user environment. Goal is
21 to have one module cache file per modulepath. This file will be stored under
22 the modulepath root with the ``.modulecache`` file name.
23
24 Depending on modulepath ownership, modulepath may or may not have a cache file
25 built. It seems more efficient to save cache on a per-modulepath basis as a
26 user may enable diverse modulepaths, some of them not maintained by the admins
27 of the machine they are running on.
28
29 Having a central system cache may miss some user-managed modulepaths.
30
31 Having a per-user cache file does not seem a good idea. If built for the user,
32 a lot more information may be pre-processed: dynamic information from the
33 modulerc files for instance may be processed at cache file creation, not when
34 using the cache file. But it cannot be expected from the user to build its
35 cache and keep it up to date. Having the cache on the user side can lead
36 to additional issue with users complaining to the support that they cannot
37 find a given module, but due to an outdated cache file.
38
39 In definitive, there will be one cache file per modulepath directory.
40
41 Cache content
42 -------------
43
44 As cache files should be used instead of walking through modulepath directory
45 to search for valid modulercs and modulefiles. Cache file should contain
46 everything to be autonomous to proceed ``avail``, ``spider`` or ``whatis``
47 queries.
48
49 Yet module search can be very dynamic as things set for instance in modulerc
50 files may dynamically produce new modulefiles or define properties specific to
51 some users or groups.
52
53 So the produced cache file cannot be the static view generated for the user
54 who has built the cache file. Thus dynamic shaping is preserved by storing
55 scripts and commands in the cache file.
56
57 Modulerc files may be highly dynamic so their full content is purely saved
58 into the cache file. An entry is set for each valid modulerc file. Same goes
59 for modulefiles as some conditional structure in their code can dynamically
60 change dependency, whatis or other definitions.
61
62 It is expected that modulefile will be evaluated on ``avail``, ``spider`` or
63 ``whatis`` commands for complex queries. So for such use case it is important
64 to have modulefile content cached.
65
66 Thus by reading the cache file, in one IO the content of all the modulerc and
67 modulefiles of the modulepath will be fetched.
68
69 Goal is to record in cache file every information obtained as a result of
70 ``findModules`` procedure. Thus cache file contains modulefile modification
71 times, which is reported on ``--long`` output. Modulefile validity check
72 result is also recorded if modulefile is not valid. Modulerc validity check
73 is not recorded as it is not an information produced by ``findModules``.
74
75 Note: modulercs and modulefiles are not tested when generating cache file to
76 see if they are valid, if their code is correctly executed. Their bare content
77 is recorded. Same result is obtained this way whether a cache file is used or
78 not.
79
80 Cache format
81 ------------
82
83 Goal is to make the cache file a Tcl script, like modulefiles. It will not
84 be exactly a modulefile as specific commands will be setup to evaluate cache
85 file and cache file will not make use of modulefile commands. See `Cache
86 evaluation`_ section for details on how a cache file is evaluated.
87
88 Cache file starts with a ``#%Module`` string like modulefiles. A version
89 number is appended next to these first characters (for instance
90 ``%Module5.3``). See `Cache validity`_ section for details on the cache file
91 *prefix* string.
92
93 Modulefile content in modulepath is defined in cache with the
94 ``modulefile-content`` command. Which takes in this order:
95
96 * relative file path
97 * file last modification date (as a Unix epoch time)
98 * file module header
99 * file body content.
100
101 For instance:
102
103 * ``modulefile-content foo/1.0 1234567890 #%Module {body}``
104 * or ``modulefile-content {foo/w s} 1234567890 #%Module5.2 {body}``
105
106 Similar command for modulerc: ``modulerc-content`` which accepts following
107 syntax:
108
109 * relative file path
110 * file module header
111 * file body content.
112
113 For instance: ``modulerc-content foo/.modulerc #%Module {body}``. Modification
114 time is not needed for modulerc as this information is not reported on an
115 ``avail``, ``spider`` sub-commands in long mode.
116
117 Every modulercs and modulefiles file contained in modulepath are recorded
118 in cache with ``modulerc-content`` or ``modulefile-content`` command.
119 Modulefiles whose name start with a dot character are also recorded in cache.
120
121 Invalid modulefiles are recorded in cache file through ``modulefile-invalid``
122 command rather ``modulefile-content``. It accepts following arguments:
123
124 * relative file path
125 * invalidity kind
126 * error message
127
128 For instance: ``modulefile-invalid foo/2.0 invalid {Magic cookie '#%Module'
129 missing}``
130
131 Files or directories that have limited access are recorded with specific
132 commands:
133
134 * ``limited-access-file foo/1.0``
135 * ``limited-access-directory foo``
136
137 Limited access means for a file that it cannot be read by user that builds
138 cache or other users. For a directory it means that it cannot be either read
139 or searched by user that builds cache or other users.
140
141 A modulefile or a modulerc is not recorded with ``modulefile-content`` or
142 ``modulerc-content`` if file or one of its parent directory has limited
143 access. This way only content that can be read by everyone is recorded into
144 cache file. Sensitive information are excluded from cache file.
145
146 Note: a cache file generated by a privileged user (which has access to every
147 thing) and a cache file generated by a less privileged user will be the same
148 as limited-access information will not be included in cache file.
149
150 Files or directories recorded as limited access will need to be tested (and
151 walked down for directories) when cache file will be evaluated. It is
152 important to distinguish files from directories to save some file stat test
153 to determine if an element is a directory when limited access elements will be
154 tested. No need for files to distinguish modulefile from modulerc as this
155 difference is visible with file name.
156
157 Note: Limited access tests are skipped on Windows platform as Unix-style file
158 permission cannot be tested there.
159
160 Recording full modulefile content or subset of elements
161 -------------------------------------------------------
162
163 It was initially drafted that only a subset of element of modulefiles would be
164 recorded in cache file, to reduce size of this file and reduce its evaluation
165 time.
166
167 Recording full modulefile content is in the end preferred as:
168
169 * this solution is simpler to implement
170 * cache file size is not too big in the end (~ hundreds of KB for a thousand
171 of modulefiles)
172 * evaluation time of large cache file is acceptable (time taken to evaluate
173 cannot be noticed by user)
174 * simpler to understand and manage for sites
175
176 Producing a cache entry for a modulefile with only a subset of commands
177 recorded (like variant or requirement) is only feasible for modulefiles not
178 using conditionals or specific evaluation scheme. Recording full content will
179 work in any scenario whereas recording a subset limits cache usage. Moreover
180 it is hard to determine, depending of the modulefile set, where cache can be
181 used or not if only a subset of elements is recorded.
182
183 Cache validity
184 --------------
185
186 From Modules 5.3 to 5.5
187 ^^^^^^^^^^^^^^^^^^^^^^^
188
189 Cache file header indicates a Modules version number. It corresponds to the
190 Modules version:
191
192 * the cache file has been built with
193 * the cache file is compatible with
194
195 It seems reasonable to ask for a cache file update every time Modules is
196 upgraded to a newer minor version (for instance from 5.3 to 5.4):
197
198 * it is simpler to understand for staff and user when the cache is taken into
199 account, when it is ignored
200 * better to ensure cache file is accurate for the Modules version as modulerc
201 and modulefile commands may evolve from one version to another
202
203 Starting Modules 5.6
204 ^^^^^^^^^^^^^^^^^^^^
205
206 Approach to cache validity is changed on Modules 5.6 to cover use cases where
207 different Modules installation uses the same set of modulepaths. This is for
208 instance the case when a service provides modulepaths with cache files in them
209 and just requires a minimal version of Modules. Or on a computing center, when
210 the same modulepaths are used by different supercomputers which may use
211 different version of Modules at some point in time.
212
213 With this change the cache file header still indicates a Modules version
214 number. But this version number now corresponds to the minimal Modules version
215 able to use this cache file.
216
217 With such change, backward compatibility is expected on module cache Tcl
218 commands. Addition of new commands will lead to an increase of the minimal
219 version required. Argument change to existing command is not possible, only
220 the introduction of optional argument will be ok (with increase of minimal
221 version required).
222
223 When generating cache file, minimal version required is set back to ``5.3`` as
224 no change occurred on the cache file format since the introduction of the
225 cache feature.
226
227 As minimal Modules version check code is the same for cache file than for
228 modulefile since the introduction of the cache feature, cache file has always
229 been taken into account as soon as the Modules version matches the minimal
230 requirement.
231
232 Cache usage
233 -----------
234
235 Any time a modulepath directory is opened to get its content, the module cache
236 file will be used instead if available.
237
238 Modulepath content analysis is performed by ``findModules`` procedures. So
239 any sub-command calling it (directly or through ``getModules`` or
240 ``getPathToModule``) will use the cache file. It corresponds to the following
241 sub-commands:
242
243 * lint
244 * paths
245 * search
246 * whatis
247 * aliases
248 * avail
249 * spider
250 * switch
251 * restore
252 * save
253 * display
254 * path
255 * source
256 * load
257 * test
258 * edit
259 * help
260
261 It may also occur during other sub-commands that evaluates modulefiles using
262 the ``is-avail`` command: like unload or refresh.
263
264 Cache files are ignored if :mconfig:`ignore_cache` configuration option is
265 enabled. This option can also be enabled just for one execution with the
266 :option:`--ignore-cache` command line option.
267
268 Cache files are ignored if cache expiry mechanism is enabled through the
269 :mconfig:`cache_expiry_secs` configuration option. When this option is set to
270 0, it means a cache file never expires. This is the default behavior. If set
271 to something else, cache file is expired if its last modification time is
272 older than the number of seconds defined in ``cache_expiry_secs``. Option
273 value is an integer between 1 and 31536000, which is the number of seconds
274 during 1 year.
275
276 Is there an impact at evaluating the full cache file rather making a directory
277 walk-through to find a module? Cache file is fully read, but not all the files
278 described in it are evaluated. Just those corresponding to the search, like it
279 is done when walking modulepath directory and evaluating only the modulerc
280 files corresponding to the query. So results between using cache file or not
281 should be the same: compared to a search without cache, no extra modulefile or
282 modulerc evaluation will be performed when a cache file is used.
283
284 As cache is recorded with both ``mcookie_check`` and ``mcookie_version_check``
285 options enabled, these two options are not honored (if disabled) when a cache
286 file is used. They are primarily useful to skip I/O tests when walking through
287 the content of a modulepath directory. As these I/O tests are done during the
288 cache build process, the options are useless when using cache files.
289
290 When cache file magic cookie defines a Modules version greater than the
291 current one, the cache file is silently ignored. Raisin error is not useful as
292 different version of Modules may be deployed in the same site environment.
293
294 When cache file is not in sync
295 ------------------------------
296
297 Files or directories are freely available through cache when used even if
298 after cache being built:
299
300 * their access is limited
301 * they are deleted
302 * their content changes and is not anymore valid
303
304 When files or directories have their access limited prior building cache, but
305 afterward these access limitations are lifted. These elements will require
306 an access test to check if they are available. This test will always be
307 successful as element accesses are not anymore limited.
308
309 If files or directories do not exist when cache is built, they will not be
310 found when cache is used.
311
312 If modulefile is recorded in cache as invalid, it will stay invalid if cache
313 is used even the modulefile is fixed. Cache need to be regenerated.
314
315 Read/write performances
316 -----------------------
317
318 :mconfig:`cache_buffer_bytes` configuration option defines size of the buffer
319 when reading or writing cache files.
320
321 With a bigger buffer, fewer read or write system calls are needed to read or
322 write cache file. On busy storage systems it can improve I/O performances.
323
324 Cache evaluation
325 ----------------
326
327 A Tcl sub-interpreter is created to analyze cache files. This sub-interp is
328 setup to evaluate cache file-specific commands, like ``modulefile-content``.
329
330 When evaluated, ``modulefile-content``, ``modulerc-content`` and
331 ``modulefile-invalid`` commands populate the read cache structure of modulerc
332 and modulefiles. This way when the modulefile for instance need to be read,
333 its content is already found in memory cache structure. It corresponds to the
334 following global variables:
335
336 * ``::g_modfileContent``
337 * ``::g_fileMtime`` (only for valid modulefile)
338 * ``::g_modfileValid`` (only for modulefile, valid or not)
339
340 In addition a ``::g_cacheModpath`` array is filled with an entry dedicated for
341 each modulepath. The content of this entry mimics the result list returned by
342 ``findModules`` procedures with information for the whole content of the
343 modulepath.
344
345 Limited access files and directories described in cache by
346 ``limited-access-file`` and ``limited-access-directory`` commands populate
347 specific structures to indicate some entries in modulepath have to be tested
348 (and walked down for directories) to determine if they are available to
349 current user:
350
351 * ``g_cacheFLimitedModpath``
352 * ``g_cacheDLimitedModpath``
353
354 These two structures are arrays with one entry per cached modulepath. Limited
355 access entries are tested if they match search query. Test is done through
356 ``findModulesFromDirsAndFiles`` procedure which corresponds to the walk down
357 code extracted from ``findModules``.
358
359 This specific interpreter is reused between different cache file evaluations.
360 As for modulefile interpreter, a consistency check is performed before each
361 reuse to test that the cache file-specific commands have not be rewritten
362 during previous cache file evaluation.
363
364 Cache file evaluation is tracked to avoid evaluating twice the same cache
365 file.
366
367 Cache evaluation stops if an erroneous command or syntax is encountered. Like
368 for erroneous modulerc, error is not reported during ``avail``, ``spider`` or
369 ``whatis`` commands unless if ran in debug mode. Error is reporting during a
370 ``load`` evaluation. Cache evaluation is considered failed if there is an
371 error in the cache file, thus a non-cache module search will occur instead of
372 relying on cache module listing. However descriptions of modulefile and
373 modulerc evaluated in cache prior the error occurs are retained.
374
375 cachebuild sub-command
376 ----------------------
377
378 :subcmd:`cachebuild` sub-command creates a module cache file in modulepaths.
379 Without arguments, it attempts to create cache in every enabled modulepaths
380 where running user has the right to write. If arguments are provided, cache
381 is build in the directories pointed by these arguments.
382
383 General properties:
384
385 * Shortcut name: none
386 * Accepted option: none
387 * Expected number of argument: 0 to N
388 * Accept boolean variant specification: no
389 * Parse module version specification: no
390 * Fully read modulefile when checking validity: yes
391 * Sub-command only called from top level: yes
392 * Lead to modulefile evaluation: yes (``cachebuild``)
393
394 An error is returned for each specified directories where current user has no
395 write access.
396
397 An error is returned if a modulefile or a modulerc cannot be read. This error
398 ends cache content generation for current modulepath. Build continues with
399 next modulepath after this error.
400
401 Modulepaths where current user has no write rights are skipped and reported
402 with a warning notice.
403
404 Reports a ``Creating <modulepath>`` block header message for each cache file
405 created or updated. This report is made when verbosity is set to ``normal``
406 or higher mode.
407
408 :mconfig:`mcookie_check` and :mconfig:`mcookie_version_check` options are both
409 enabled when recording cache. This is produced with exact same content whether
410 these options are enabled or not.
411
412 cacheclear sub-command
413 ----------------------
414
415 :subcmd:`cacheclear` sub-command deletes all module cache file in enabled
416 modulepaths.
417
418 General properties:
419
420 * Shortcut name: none
421 * Accepted option: none
422 * Expected number of argument: 0
423 * Accept boolean variant specification: no
424 * Parse module version specification: no
425 * Fully read modulefile when checking validity: no
426 * Sub-command only called from top level: yes
427 * Lead to modulefile evaluation: no
428
429 Modulepaths where current user has no write rights on the modulepath directory
430 are skipped and reported with a warning notice.
431
432 Reports a ``Deleting <modulepath>`` block header message for each cache file
433 created or updated. This report is made when verbosity is set to ``normal``
434 or higher mode.
435
436 .. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: