Debian Codegraph @

doc/source/design/extra-match-search.rst

raw
1 .. _extra-match-search:
2
3 Extra match search
4 ==================
5
6 Traditional module lookup only performs modulefile search, module version
7 resolution, module alias and symbolic version resolution. This is based on
8 file lookup and evaluation of modulerc files.
9
10 Searching modules with a query exceeding traditional lookup is called *extra
11 match search*. It mainly looks at modulefile content to retain those matching
12 specified query. This content lookup requires a specific modulefile evaluation
13 (see `Modulefile evaluation mode for extra match search`_).
14
15 Requirements
16 ------------
17
18 An extra match search query requires :ref:`advanced-module-version-specifiers`
19 configuration to be enabled. Otherwise query specifiers will be understood as
20 module name and version.
21
22 When extra match search applies
23 -------------------------------
24
25 On :ref:`module_version_specification_to_return_all_matching_modules` context.
26 On such context, variant specification in search query was previously ignored.
27 Extra match search adds here a feature to filter returned results.
28
29 On :ref:`module_version_specification_to_select_one_module` context, extra
30 match search is not enabled at the moment. As it changes obtained result. For
31 instance when loading a module with a given variant value, *extra match
32 search* will remove from available modules those not declaring this variant
33 and its specified value. It is considered to add an option to apply *extra
34 match search* to the *select one module* context. This option will be disabled
35 by default on Modules v5 and enabled by default on v6.
36
37 What triggers an extra match search?
38 ------------------------------------
39
40 * module variants or provided module aliases should be reported
41 * variant is specified in query on :subcmd:`avail`, :subcmd:`paths`,
42 :subcmd:`spider` or :subcmd:`whatis`
43 * an extra specifier is specified in query
44
45 Extra match search process
46 --------------------------
47
48 Extra match search is performed from ``getModules`` procedure. All subsequent
49 procedures relying on ``getModules`` will then benefit from this extra process
50 without a change.
51
52 Modulefile scan for Extra match search is done after *phase 2: early filtering
53 of non-matching elements*. Right after this scan, collected provided
54 aliases will be included into the search result list, if enabled, to apply
55 them phase 4 and phase 5 filtering stages.
56
57 Module filtering for Extra match search is done right after *phase 5:
58 elaborate directory content with default element selection*. At this stage:
59
60 * symbolic versions, aliases and virtual modules are known
61 * dynamically hidden or expired elements have been filtered
62 * automatically defined symbols are set
63 * initial directory content and their default version are set
64
65 This way the subsequent phases that filters result based on name and version
66 search query, adjusts directory content and no-indepth mode does not need to be
67 updated and will apply on the extra match search result.
68
69 In case no-indepth mode is asked and extra match search should be performed,
70 traditional file lookup (by ``findModules``) is made to return all existing
71 files. Not only those matching depth level. As extra match filtering process
72 may eliminate some entries, all of them should be tested to obtain accurate
73 result.
74
75 If provided aliases should be reported, all modulefiles should be scanned to
76 fetch all aliases they defined. Thus ``findModules`` is asked to get all
77 modulefiles and *phase 2* early filtering is skipped.
78
79 Extra match search will only be performed if search query requires it. It will
80 be determined by ``isExtraMatchSearchRequired`` procedure. See `What triggers
81 an extra match search?`_ for applying conditions. Current search module
82 specification is passed as argument. This string helps to find if:
83
84 * variant has been specified in *ModuleVersSpec* structure
85 * extra specifier has been specified in *ModuleVersSpec* structure
86
87 If search query is a traditional one, no extra match search is performed. It
88 saves this way substantial processing time as no evaluation of modulefile is
89 performed.
90
91 ``getModules`` will call dedicated procedures:
92
93 * ``scanExtraMatchSearch`` evaluates all entries in obtained result array that
94 are of modulefile or virtual types. See `Modulefile evaluation mode for
95 extra match search`_ section.
96 * ``insertProvidedAliases`` adds provided aliases found during scan to the
97 result array.
98 * ``filterExtraMatchSearch`` filters entries from result array with the
99 consolidated information gathered from scan evaluations. Entries that do not
100 match the extra match query are withdrawn.
101
102 These 3 procedures obtains the name of result array as argument and will bind
103 to this array variable (to change it without copy). Search module
104 specification is also set as argument to help to know the extra match query to
105 filter modules on.
106
107 Symbolic versions
108 ^^^^^^^^^^^^^^^^^
109
110 From ``filterExtraMatchSearch``, symbolic version elements are retained unless
111 if they target an element that is withdrawn by the filter process.
112
113 They need to be filtered if their target is filtered as it may confuse some
114 sub-commands like ``paths`` if they are part of ``getModules`` result.
115
116 During wild search (no module name specified) with variant specified, auto
117 symbolic versions (*latest* and *default*) are disabled. This is done
118 especially for sub-commands like ``paths`` that take getModules result and try
119 to resolve each entry. In this situation it is important not to have any auto
120 symbol that indirectly targets an element withdrawn from extra match filter.
121
122 Module aliases
123 ^^^^^^^^^^^^^^
124
125 From ``filterExtraMatchSearch``, module alias elements are purely withdrawn by
126 the filter process as they are not resolved on *return all matching modules*
127 context. They are also withdrawn even if searched module name matches alias
128 name.
129
130 They need to be filtered when search query mentions a specific variant and
131 value as alias does not define variant in itself.
132
133 When variants have to be reported but are not part of the search query, then
134 no filtering process occurs, aliases are not withdrawn.
135
136 Directories overridden by aliases take precedence over these aliases when
137 *extra match search* is performed.
138
139 FUTURE: it may be revised to implement extra match search on *select one
140 module* context.
141
142 Module tags
143 ^^^^^^^^^^^
144
145 Modulefiles tagged *forbidden* are excluded from evaluation mechanism. If
146 forbidden tag only applies to some specific variant values, modulefile is
147 evaluated.
148
149 Note that in a *select one module* context, if forbidden tag applies
150 specifically to searched variant, this module is excluded from result.
151
152 If module hidden tag only applies to some specific variant values, modulefile
153 is evaluated in scan mode.
154
155 Error entries
156 ^^^^^^^^^^^^^
157
158 Files in error (e.g., without a module header string, not readable) are
159 ignored by ``filterExtraMatchSearch`` and kept in result array. These error
160 entries are useful to report the message they contain if erroneous file was
161 specifically selected.
162
163 Extra specifier
164 ---------------
165
166 Extra specifier enables to query whose modules defines a given element. With
167 them it is possible to get all the modules defining a given environment
168 variable or requiring a given module.
169
170 Most of the modulefile Tcl commands are associated to an extra specifier. Some
171 aliases are also provided, to gather several command elements into the general
172 same concept they are about. Following table list the available extra
173 specifiers and their associated modulefile command.
174
175 +--------------------+-----------------------------------------+
176 | Modulefile command | Extra specifier(s) |
177 +====================+=========================================+
178 | variant | variant |
179 +--------------------+-----------------------------------------+
180 | setenv | setenv, envvar |
181 +--------------------+-----------------------------------------+
182 | unsetenv | unsetenv, envvar |
183 +--------------------+-----------------------------------------+
184 | pushenv | pushenv, envvar |
185 +--------------------+-----------------------------------------+
186 | append-path | append-path, envvar, |
187 | | use (if envvar is MODULEPATH) |
188 +--------------------+-----------------------------------------+
189 | prepend-path | prepend-path, envvar, |
190 | | use (if envvar is MODULEPATH) |
191 +--------------------+-----------------------------------------+
192 | remove-path | remove-path, envvar |
193 +--------------------+-----------------------------------------+
194 | complete | complete |
195 +--------------------+-----------------------------------------+
196 | uncomplete | uncomplete |
197 +--------------------+-----------------------------------------+
198 | set-alias | set-alias |
199 +--------------------+-----------------------------------------+
200 | unset-alias | unset-alias |
201 +--------------------+-----------------------------------------+
202 | set-function | set-function |
203 +--------------------+-----------------------------------------+
204 | unset-function | unset-function |
205 +--------------------+-----------------------------------------+
206 | chdir | chdir |
207 +--------------------+-----------------------------------------+
208 | family | family, provided-alias |
209 +--------------------+-----------------------------------------+
210 | provide | provide, provided-alias |
211 +--------------------+-----------------------------------------+
212 | prereq | prereq, prereq-any, require |
213 +--------------------+-----------------------------------------+
214 | prereq-any | prereq-any, depends-on-any, prereq, |
215 | | require |
216 +--------------------+-----------------------------------------+
217 | prereq-all | prereq-all, depends-on, require |
218 +--------------------+-----------------------------------------+
219 | depends-on | depends-on, prereq-all, require |
220 +--------------------+-----------------------------------------+
221 | depends-on-any | depends-on-any, prereq-any, prereq, |
222 | | require |
223 +--------------------+-----------------------------------------+
224 | always-load | always-load, require |
225 +--------------------+-----------------------------------------+
226 | conflict | conflict, incompat |
227 +--------------------+-----------------------------------------+
228 | module load | load, require |
229 +--------------------+-----------------------------------------+
230 | module load-any | load-any, require |
231 +--------------------+-----------------------------------------+
232 | module try-load | try-load, require |
233 +--------------------+-----------------------------------------+
234 | module unload | unload, incompat |
235 +--------------------+-----------------------------------------+
236 | module switch | switch, switch-on, require, switch-off, |
237 | | incompat |
238 +--------------------+-----------------------------------------+
239 | module use | use |
240 +--------------------+-----------------------------------------+
241
242 +--------------------+-----------------------------------------+
243 | Modulerc command | Extra specifier(s) |
244 +====================+=========================================+
245 | module-tag | tag |
246 +--------------------+-----------------------------------------+
247 | module-hide | tag |
248 +--------------------+-----------------------------------------+
249 | module-forbid | tag |
250 +--------------------+-----------------------------------------+
251 | module-warn | tag |
252 +--------------------+-----------------------------------------+
253
254 Extra specifier are only valid on *return all matching modules* context. An
255 error is returned when used in query on other contexts.
256
257 Using an unknown extra specifier (not part of the above table) produces an
258 error. Helps to distinguish between a bad specification and no modulefile
259 declaring the associated modulefile command.
260
261 Extra specifier are not made to resolve module alias or symbolic version, as
262 such resolution is part of the regular match search.
263
264 The ``module switch`` command produces content in different extra specifiers.
265 Switched-on module can be queried via ``switch``, ``switch-on`` and
266 ``require`` specifiers. Switched-off module can be queried via ``switch``,
267 ``switch-off`` and ``incompat`` specifiers. On its one-arg form, no
268 switched-off module is added to relative specifiers.
269
270 Change on the ``MODULEPATH`` environment variable either through
271 ``module use``, ``append-path`` or ``prepend-path`` may be queried with
272 ``use`` extra specifier. Modulepath are translated to their absolute path form
273 to be recorded. Variable reference are resolved at record time. Symbolic links
274 are recorded as is (they are not translated into their target)
275
276 Recording extra specifier specification
277 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
278
279 Extra specifiers defined in a search query are processed and saved in
280 *ModuleVersSpec* structure along other query elements. As the search query
281 string is passed along the procedures, it is possible to retrieved the
282 extra specifier properties from this query string (with
283 ``getExtraListFromVersSpec`` procedure).
284
285 Extra specifiers are saved as a list of list. Every extra specifier defined in
286 query is set as a list whose first element is extra specifier name, then the
287 other element corresponds to the list of values provided to this argument.
288
289 For example, search query ``mod/1.0 setenv:FOO variant:bar setenv:BAR``
290 produces internal representation ``{setenv FOO} {variant bar} {setenv BAR}``.
291
292 Another example with multiple values, search query ``mod/1.0 setenv:FOO,BAR``
293 produces internal representation ``{setenv FOO BAR}``.
294
295 Value specified may contain a space character if it is escaped on command line
296 either by quotes (``"prereq:[email protected] foo=bar"``) or backslash
297 (``prereq:[email protected]\ foo=bar``). This is especially useful for complex module
298 specification.
299
300 Filtering extra specifier results
301 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
302
303 Every *-sc* variant of modulefile Tcl command, record their properties in the
304 ``g_scanModuleElt`` global dictionary. It is a nested structure that contains
305 at the top level the list of defined extra specifier names. Each of these
306 names is in turn a structure that contains all defined extra specifier values
307 associated to the module name and version defining this extra specifier name
308 and value pair.
309
310 For example, when module *mod/1.0* defines *setenv FOO value* in its file,
311 then the global dictionary is updated to add *mod/1.0* to the *setenv > FOO*
312 nested key.
313
314 When extra specifier has alias name(s), each of these aliases have their own
315 entry in the global directory. Reusing the previous example, *envvar* is an
316 alias on *setenv* extra specifier. When module *mod/1.0* defines *setenv FOO
317 value*, the global dictionary is also updated to add *mod/1.0* to the *envvar
318 > FOO* nested key (in addition to the *setenv > FOO* nested key.
319
320 Such data structure optimizes filtering work: it is done once for all
321 modulefiles after finishing the scan evaluation of all of them. For each
322 extra specifier criteria, goal is to match the corresponding nested key in
323 ``g_scanModuleElt`` global directory. Modulefiles to keep in result of those
324 listed as value in every matched nested keys.
325
326 When there are several extra specifiers in search query, result are the
327 modules present in the value list of every matched keys (intersection of value
328 list obtained for every extra specifier criterion).
329
330 Module tags, queried with *tag* extra specifier, are not recorded in the
331 same structure than other extra specifier. Regular tag mechanism and recording
332 structure is used.
333
334 Specific filtering work is achieved for extra specifiers accepting a module
335 specification. See next section.
336
337 Filtering module specification
338 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
339
340 Some extra specifiers accept a module specification as value. It corresponds
341 to all extra specifiers relative to requirements and incompatibilities
342 expressed by modulefiles (*prereq*, *conflict*, etc).
343
344 Module specification passed as extra specifier value has to be compared to the
345 module specification set in the modulefile definition for corresponding
346 command.
347
348 When parsing such extra specifier value, when evaluating command line
349 arguments, module specification is parsed with a specific available module
350 resolution: all matching available modules are recorded into module
351 specification (``g_moduleVersSpec``). It enables to fetch:
352
353 * every module name and version matching a version list or range specification
354 * every generic or fully qualified names
355
356 Match against modulefile definitions of such extra specifiers is performed in
357 the same location than for other extra specifiers. It relies on a ``modEq``
358 comparison that has been adapted to also compare alternative names fetched and
359 stored into module specification structure. Relying on ``modEq`` procedure,
360 comparison leverages *icase* and *extended_default* features.
361
362 **LIMITATIONS**: Current module specification match does not support:
363
364 * Comparison of module alias or symbolic names when used in either extra
365 specifier value or in modulefile definition
366 * Version range or list specified in extra specifier value is converted into
367 a list of existing modulefiles, thus a version in such specification that
368 does not correspond to an existing module will not be matched
369
370 Variant value comparison is enabled on ``modEq`` test. ``modVariantCmp``
371 internal test has been adapted to let a missing variant definition be
372 considered as a match.
373
374 Query grammar
375 -------------
376
377 In extra match search query, variants are expressed like in *select one
378 module* context:
379
380 * multiple variants mentioned act as an *AND* operation
381
382 * ``module avail mod/1.0 foo=val1 bar=val2``
383 * means module defines *foo* variant with *val1* as an available value and
384 *bar* variant with *val2* as an available value
385
386 * same variant mentioned multiple times: all mentions retained and act as an
387 *AND* operation
388
389 * ``module avail mod/1.0 foo=val1 foo=val2``
390 * means module defines *foo* variant with *val1* and *val2* as available
391 values
392
393 Behavior for a single variant specified multiple times is adapted on *return
394 all matching modules* context to be the same as a single extra specifier set
395 multiple times.
396
397 In extra match search query, extra specifiers are expressed with *name:value*
398 syntax. Using *:* as separator helps to distinguish from variant
399 specification. As a consequence, such character is not recommended to be part
400 of a module name or version (which was already the case, as *:* character is
401 also a separator used in environment variables like *LOADEDMODULES*). As extra
402 specifiers only applies to *return all matching modules* context, the use of
403 *name:value* specific syntax may help user to distinguish from the *select one
404 module* context.
405
406 NOTE: use of a *--where* or *--with* options was also considered to be able
407 to use a *name=value* syntax like variant specification. It was preferred to
408 use a different syntax (*name:value*) to avoid having to type an extra option.
409
410 NOTE: if *name=value* syntax where used for extra specifier, all extra
411 specifier names would have been forbidden to use as variant names.
412
413 * multiple extra specifiers mentioned act as an *AND* operation
414
415 * ``module avail mod/1.0 setenv:FOO pushenv:BAR``
416 * means module defines *setenv* command to set *FOO* environment variable
417 and defines *pushenv* command to set BAR*
418
419 * same extra specifier mentioned multiple times: all mentions retained and act
420 as an *AND* operation
421
422 * ``module avail mod/1.0 variant:foo variant:bar``
423 * means defines variant *foo* and variant *bar*
424
425 Value of either variant or extra specifier are full name. No wildcard
426 characters are taken into account (they are treated literally).
427
428 An error is raised when an empty extra specifier name or value is specified.
429
430 Expressing an *OR* operation
431
432 * for instance by mentioning multiple values separated by ``,`` character
433 * ``module avail mod/1.0 foo=val1,val2 bar=val2``
434 * means *foo* equals *val1* or *val2* and *bar* equals *val2*
435 * such syntax is equivalent to version list specification
436 * does not seem useful to allow ranges like for version
437 * Such syntax is only allowed for *return all matching modules* context
438
439 Expressing a *NOT* operation
440
441 * by adding a prefix string ``not:`` prior value
442 * ``module avail mod/1.0 not:foo=val1 bar=val2``
443 * means *bar* equals *val2* and either *foo* variant is not defined or it
444 equals any value expect *val1*
445 * such syntax is only allowed for *return all matching modules* context
446 * ``not:`` syntax is chosen over prefix character like ``!``, which is
447 interpreted as an event on shells like Tcsh or Bash, and ``~``, which is
448 resolved a HOME directory path if following string matches an existing user
449 * NOTE: *NOT* operator is not specified currently on Spack's side.
450 * FUTURE: If a *not* keyword appear on Spack later on, it may also be
451 supported here as an alias over the ``not:`` prefix.
452
453 What triggers ``scan`` evaluation?
454 ----------------------------------
455
456 * everything that triggers an extra match search
457 * except *tag* extra specifier, as tags are defined outside of modulefiles
458
459 Modulefile evaluation mode for extra match search
460 -------------------------------------------------
461
462 A specific evaluation mode is created for looking at modulefile content in
463 order to solve an extra match search. This mode is named ``scan``.
464
465 Modulefile Tcl interpreter is configured to run a specific variant of
466 modulefile Tcl commands. These variants will gather content of modulefile in
467 global structures that will be searched after all *scan* evaluations to
468 determine what modules match the extra query. Names of scan variant of
469 modulefile Tcl command scan variant procedures are internally suffixed with
470 *-sc*.
471
472 If modulefile content queries the current evaluation mode, it will obtain the
473 ``scan`` string during a *scan* evaluation. We expect modulefile code to cope
474 with this behavior, especially to still be able to get all *load* information
475 of modulefile, even if mode is not ``load``. A ``load`` string cannot be
476 returned during a *scan* evaluation as modulefile may have some code to
477 produce log message when a *load* attempt is detected.
478
479 Even if extra match search only concern a specific element, like querying
480 module variants, all modulefile commands are setup to gather all content
481 information about modulefile. This way, it is evaluated the same way whatever
482 the query. Scan evaluation code will also be simpler.
483
484 Environment variable definition through modulefile commands like ``setenv``,
485 ``append-path``, etc are handled like in ``whatis`` evaluation mode. Variables
486 are set to an empty value. Some processing time is saved this way. It may have
487 an impact on some complex modulefiles: content scan may not be accurate. If
488 this is an issue on some setup, an option could be added in the future to set
489 variable to their expected value also on ``whatis`` and ``scan`` evaluation
490 modes.
491
492 When an error is raised from a modulefile evaluation. No error message is
493 reported as such message is silenced during the whole ``avail`` process.
494 Rendered result is ok for all kind of errors even for ``break`` or ``exit``.
495 No error result is rendered for these two, unlike for a ``whatis`` evaluation,
496 not to disturb the whole search result for one modulefile not coping with scan
497 evaluation.
498
499 ``puts`` commands in modulefile are rendered like on a ``whatis`` evaluation
500 mode: output is effectively produced toward the designated channel. It is up
501 to modulefile writer to handle the ``scan`` evaluation, like done for
502 ``whatis``.
503
504 ``source-sh`` commands in modulefile are rendered like on a ``load``
505 evaluation. They will produce ``setenv``, ``set-alias``, ``set-function`` and
506 so on commands that will be in turn evaluated in *scan* mode.
507
508 Specific impact
509 ---------------
510
511 * With a ``contains`` search match every modulefiles in modulepath are
512 evaluated in scan mode.
513
514 - FUTURE: there may have room for optimization here
515
516 * As environment variable defined by modulefile commands like ``setenv`` are
517 set to an empty value during scan evaluation, if a modulepath entry contains
518 a reference to such environment variable, it will not match its expecting
519 value.
520
521 .. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: