Debian Codegraph @

doc/source/design/modulepath-ignore.rst

raw
1 .. _modulepath-ignore:
2
3 Ignore files in modulepath
4 ==========================
5
6 When walking down the content of a modulepath directory to search for
7 modulefiles, every file found is checked to determine if it is a modulefile
8 and every sub-directory is walked down. Some modulepath directories contain
9 files that are not modulefiles: helper scripts sourced by modulefiles or
10 software installation files when modulefiles are mixed with the software
11 trees they describe.
12
13 Currently the only way to make the module command skip such files is to
14 give them a name starting with a dot character. This is not always possible
15 or desirable and it does not avoid these files to be tested or read. When
16 the amount of non-modulefiles is very large (millions of files), the
17 :subcmd:`cachebuild` sub-command takes a huge amount of time and memory as
18 an entry is recorded in the cache file for every invalid modulefile found
19 (see issue #562 and the discussion on the modules-interest mailing-list
20 linked there).
21
22 Objective is to let modulepath maintainers describe the content of their
23 modulepath directory that should be ignored when searching for modulefiles.
24 Ignored files and directories are treated as if they do not exist: they are
25 not tested, not walked down, not reported and not recorded in cache files.
26
27 ``modulepath-ignore`` modulerc command
28 --------------------------------------
29
30 A new modulefile command is created: ``modulepath-ignore``. It takes one or
31 more file path patterns as arguments::
32
33 modulepath-ignore pattern1 [pattern2 ...]
34
35 An error is raised if no argument is provided.
36
37 This command is registered as a command alias in the modulerc Tcl
38 interpreter (``g_modrcAliases`` in ``execute-modulerc``). It is also
39 registered in the modulefile interpreter per-mode alias table (like
40 ``module-hide``), as global and user rc files are evaluated as modulefiles
41 through the ``source`` sub-command mechanism. The command itself raises an
42 error when evaluated from a file which is neither a rc file nor evaluated
43 in the global/user rc context (tracked by the ``rc_running`` state), so a
44 regular modulefile using it gets a *not allowed in modulefile* error. It is
45 set no-operation on *refresh* and *scan* evaluation modes.
46
47 The command may be called several times: patterns accumulate in their
48 definition order into a single ordered list.
49
50 The initial specification (issues #561 and #562) also proposed a
51 ``modulepath-only`` command to switch from an *accept all, ignore some*
52 logic to an *ignore all, accept some* logic. This second command is
53 dropped: as pattern syntax follows the gitignore syntax (see below), its
54 negated patterns already cover this need. For instance to only take ``.mod``
55 files into account::
56
57 modulepath-ignore * !*/ !*.mod
58
59 ``*`` ignores everything, ``!*/`` re-includes directories so the walk down
60 continues, ``!*.mod`` re-includes files with the ``.mod`` extension.
61
62 The use case reported on the mailing-list (helper Tcl scripts stored next
63 to modulefiles) is solved with::
64
65 modulepath-ignore *.tcl
66
67 Pattern syntax
68 --------------
69
70 Patterns follow the exact same syntax and behavior as git with
71 ``.gitignore`` files:
72
73 * A pattern without slash (except an optional trailing one) matches files
74 or directories at any depth below the directory of the rc file defining
75 it. For instance ``*.tcl`` matches ``lib.tcl`` and ``mod/1.0/lib.tcl``.
76 * A pattern with a slash at its start or in its middle is anchored to the
77 directory of the defining rc file. For instance ``mod/1*`` matches
78 ``mod/1.0`` but not ``other/mod/1.0``.
79 * A pattern ending with a slash only matches directories.
80 * ``*`` matches anything except a slash. ``?`` matches any single character
81 except a slash. ``[...]`` matches one character from a range or class.
82 * A leading ``**/`` means match in any directory (``**/foo`` equals
83 ``foo``). A trailing ``/**`` matches everything inside the designated
84 directory. ``a/**/b`` matches with zero or more intermediate directories.
85 * A pattern starting with ``!`` negates a previous match: the element is
86 re-included. To match a file name starting with a literal ``!``, escape
87 it with a backslash (``\!``).
88 * When several patterns match an element, the last matching pattern
89 decides.
90 * It is not possible to re-include a file if one of its parent directories
91 is ignored, as ignored directories are not walked down (same behavior as
92 git).
93
94 As patterns are passed as Tcl command arguments and not read from a
95 dedicated file, the ``.gitignore`` file-format rules (comment lines, blank
96 lines, trailing space stripping) do not apply here.
97
98 Pattern anchoring and definition scope
99 --------------------------------------
100
101 ``modulepath-ignore`` is honored in ``.modulerc`` files located within a
102 modulepath directory. Preferred location is the ``.modulerc`` file at the
103 root of the modulepath, as this file determines how to walk through the
104 modulepath content.
105
106 No specific restriction is set when the command is used in a ``.version``
107 file: it proceeds like it does in a ``.modulerc`` file, consistently with
108 the other modulerc commands which are all accepted in ``.version`` files.
109 The purpose of ``.version`` files is however to set the default version of
110 a directory, so module documentation will designate ``.modulerc`` as the
111 place where ``modulepath-ignore`` should be defined.
112
113 Each pattern is recorded along with its anchor directory, which is the
114 directory of the rc file defining it. For instance ``modulepath-ignore 1*``
115 in ``/path/to/modpath/foo/.modulerc`` produces a pattern equivalent to
116 ``/path/to/modpath/foo/1*``. An element is matched against a pattern by
117 first checking the pattern anchor directory is one of its parent
118 directories, then matching its path relative to this anchor against the
119 pattern.
120
121 Recording the anchor directory along the pattern means the modulepath root
122 directory does not need to be known when the command is evaluated, and
123 patterns naturally apply only to the sub-tree of the rc file defining them.
124
125 As rc files of parent directories are evaluated before those of deeper
126 directories (see next section), patterns from deeper rc files are appended
127 after those of parent directories. Combined with the *last matching pattern
128 decides* rule, deeper rc files take precedence over parent ones, like git
129 does with nested ``.gitignore`` files.
130
131 When ``modulepath-ignore`` is evaluated from a rc file not located within a
132 modulepath directory (global rc, user rc), it registers global patterns.
133 Such patterns have no anchor directory: they apply to every modulepath,
134 matched against element paths relative to the root of the walked
135 modulepath, as if they were defined in the ``.modulerc`` file at this root.
136 As global and user rc files are evaluated at startup, before any modulepath
137 walk, global patterns come first in the pattern list: patterns defined
138 within a modulepath take precedence over them through the *last matching
139 pattern decides* rule.
140
141 Evaluation of rc files during modulepath walk
142 ---------------------------------------------
143
144 Currently ``findModules`` collects the ``.modulerc`` and ``.version``
145 files found during the modulepath walk as ``modulerc`` entries in its
146 result list, and ``getModules`` evaluates them afterward, in sorted order,
147 during its consolidation phase.
148
149 This is too late for ``modulepath-ignore``: patterns must be known when the
150 walk occurs. Modulerc evaluation is thus moved to an earlier stage, in the
151 ``findModulesFromDirsAndFiles`` procedure:
152
153 * No ordering requirement is set on the element list returned by
154 ``getFilesInDirectory``: the rc files of a fetched directory are directly
155 looked up by name in the element array built from this list, and the
156 remaining elements are sorted in lexical order when inserted in the
157 walked list.
158 * ``findModulesFromDirsAndFiles`` evaluates the rc files of a directory as
159 soon as its content is fetched, prior trimming the element list for the
160 *stop_mod* optimization: their ignore patterns and hiding rules are in
161 place before any sibling element of this directory is analyzed or
162 processed. Once evaluated and recorded in the result list, these rc
163 files are withdrawn from the element list: only the rc files that are
164 not part of a fetched directory content flow through the walked list and
165 get evaluated and recorded by the walk loop (modulepath root rc file
166 from the initial listing, limited access rc files walked directly from a
167 cache file listing).
168 * The walk becomes depth-first: the content of a directory is inserted
169 right after it in the walked list instead of being appended at its end.
170 Module resolution and symbol definition results depend on the rc file
171 evaluation order: a breadth-first walk would evaluate the rc files of
172 all first-level directories before any deeper rc file, changing the
173 resolution result when a rc file of one sub-tree sets a symbol like
174 ``default`` on modules of another sub-tree. Root elements are walked in
175 lexical order (sorted by ``findModules``) to keep a deterministic
176 evaluation order for the rc files of first-level directories. Deeper
177 sibling directories are walked in lexical order too, as element paths
178 are sorted when inserted in the walked list.
179 * At the start of the walk, the ``.modulerc`` file at the root of the
180 walked directory is evaluated if it exists and is readable, then every rc
181 file already recorded in the result list transmitted to the walk
182 procedure is evaluated in lexical order. This transmitted list is empty
183 on a regular walk, but on cache use it holds the entries of the evaluated
184 cache file: these rc files are not walked through, as they were readable
185 when cache was built, but the patterns they may define apply to the walk
186 down of the limited access elements. Containing this evaluation in
187 ``findModulesFromDirsAndFiles`` keeps ``findModulesInCacheFile`` free of
188 any rc evaluation logic.
189 * On a no-indepth search, ``.version`` files at a different depth level
190 than the search target are currently neither recorded nor evaluated
191 (unlike ``.modulerc`` files, which have no such restriction). This
192 behavior is kept when evaluation moves to walk time, so no extra
193 ``.version`` evaluation occurs. As a consequence, patterns defined in a
194 ``.version`` file lying at a different depth level than a no-indepth
195 search target are not applied on such a search: one more reason to prefer
196 defining patterns in ``.modulerc`` files.
197
198 The ``g_rcfilesSourced`` tracking array still guards against evaluating the
199 same rc file twice. ``modulerc`` entries are still recorded in the result
200 list of ``findModules``: they are needed by the cache build process and to
201 transmit cache file entries to the walk procedure. As every rc file is now
202 evaluated by ``findModulesFromDirsAndFiles`` whether it comes from a
203 directory walk or from a cache file, the consolidation phase of
204 ``getModules`` does not evaluate rc files anymore: it just withdraws the
205 ``modulerc`` entries from the result list.
206
207 Side benefits of this change:
208
209 * rc files are now evaluated in walk order, which guarantees parent
210 directory rc files are evaluated before deeper ones.
211 * ``trimElemListForStopmod`` does not need to source rc files anymore for
212 the *stop_mod* search optimization: the hiding and ignore rules it needs
213 are in place as rc files are evaluated prior its call.
214
215 Impact on module search
216 -----------------------
217
218 Every element found during the walk (file or directory) is matched against
219 the recorded pattern list. When an element is ignored:
220
221 * if it is a file, it is not tested for modulefile validity (no I/O on it)
222 and not recorded in the search results
223 * if it is a directory, it is not walked down and not recorded
224
225 Ignored elements are therefore not reported by :subcmd:`avail`,
226 :subcmd:`spider`, :subcmd:`whatis` or any other listing sub-command. A
227 direct access to an ignored element (e.g., ``module load`` of it) leads to
228 the same *Unable to locate a modulefile* error as a nonexistent file. Same
229 result is obtained through the ``is-avail`` modulefile command.
230
231 As rc files located within an ignored directory are never reached, the
232 aliases, symbolic versions or virtual modules they may define do not exist
233 either.
234
235 The ``.modulerc`` and ``.version`` files of walked-through directories are
236 exempt from ignore matching: they are always evaluated, as they control the
237 walk itself (this also makes the ``modulepath-ignore * ...`` idiom work).
238 The rc files of ignored directories are naturally never evaluated.
239
240 Impact on module cache
241 ----------------------
242
243 The cache build process (``formatModuleCacheContent``) relies on
244 ``findModules`` to collect modulepath content: ignored elements are
245 excluded from its results, so no ``modulefile-content``,
246 ``modulefile-invalid`` or directory-related entry is recorded for them in
247 the generated ``.modulecache`` file. This solves the initial problem of
248 issue #562: non-modulefiles matched by ignore patterns are neither read nor
249 recorded, whatever their number, and cache size stays proportional to the
250 number of actual modulefiles.
251
252 ``getLimitedAccessesInDirectory`` performs its own walk of the modulepath
253 directory: it must also apply the ignore patterns so that ignored files or
254 directories with limited access rights are not recorded as
255 ``limited-access-file`` or ``limited-access-directory`` entries. Patterns
256 are already known at this stage as this procedure is called after
257 ``findModules`` in the cache build process.
258
259 The rc file exemption from pattern matching also applies to this walk: a
260 limited access rc file matching an ignore pattern (which happens with the
261 *ignore all, accept some* idiom, as ``*`` matches ``.modulerc`` itself) is
262 still recorded as a limited access entry. Without this exemption, such a
263 file would be skipped by this walk but still collected by the module
264 search walk, where rc files are exempt: its content would then get plainly
265 recorded in the cache file despite its restricted access.
266
267 No new cache file command is introduced and no existing command signature
268 changes: the minimal Modules version advertised in the cache file header is
269 unchanged.
270
271 When a cache file is used:
272
273 * ignored elements are simply absent from it, so search and direct access
274 behaviors are consistent with the no-cache case
275 * limited access entries recorded in cache are re-walked through
276 ``findModulesFromDirsAndFiles``: ignore patterns are applied on this walk
277 too. The rc files recorded in cache are evaluated at the start of this
278 walk (their content is obtained from the evaluated cache file), so the
279 patterns they may define apply to the walked down limited access
280 elements.
281
282 As ignored elements are excluded from the cache file at build time, the
283 ignore pattern definitions have to be static. If patterns are defined
284 dynamically (e.g., depending on user, group or environment), elements
285 ignored for the user building the cache are definitively absent from the
286 cache file, whatever the dynamic conditions evaluate to for the user of the
287 cache. This limitation must be clearly stated in the module documentation:
288 ``modulepath-ignore`` definitions should not be conditional.
289
290 The same caution applies to global patterns defined in the user rc file:
291 they are honored when this user builds a cache file, so elements they match
292 are excluded from a cache file potentially shared with other users.
293
294 Disabling ignore patterns
295 -------------------------
296
297 A new boolean configuration option controls whether ``modulepath-ignore``
298 definitions are applied: ``modulepath_ignore``. It is enabled by default
299 and can be disabled through the ``MODULES_MODULEPATH_IGNORE`` environment
300 variable, the :subcmd:`config` sub-command or the ``--no-modulepath-ignore``
301 command-line switch for one execution.
302
303 A ``show_ignored`` option name associated with a ``--ignored`` switch (git
304 wording, as in ``git status --ignored``) was first considered then
305 rejected: disabling the mechanism does not *show* the ignored elements, it
306 just stops ignoring them, and they may still not appear anywhere if they
307 are not valid modulefiles in the end. The option is thus named after the
308 mechanism it enables or disables.
309
310 When this option is disabled, ``modulepath-ignore`` commands still evaluate
311 without error but the patterns they register are not applied: every element
312 of the modulepath directories is walked through and tested as if no pattern
313 were defined.
314
315 Disabling this option helps troubleshooting, for instance to determine
316 whether a modulefile is not found because it is ignored. Note that a cache
317 file built with patterns applied does not contain the ignored elements:
318 combining with the :option:`--ignore-cache` option is required to also
319 search elements excluded from cache file. Similarly, running
320 :subcmd:`cachebuild` with ``modulepath_ignore`` disabled records every
321 element in the cache file, as if the feature was not used.
322
323 Interaction with existing mechanisms
324 ------------------------------------
325
326 * :mconfig:`ignored_dirs` configuration option: this option globally lists
327 directory names to ignore (version control directories by default).
328 ``modulepath-ignore`` complements it on a per-modulepath basis with a
329 much finer pattern syntax. The ``ignored_dirs`` check is integrated in
330 the matcher procedure, as an early test made prior applying patterns:
331 the behavior of this option is preserved, as negated patterns cannot
332 re-include a matching directory and directories keep being ignored when
333 the ``modulepath_ignore`` option is disabled.
334 * Backup or version control files (``*~``, ``*,v``, ``#*#``): the walk
335 procedures historically skipped these file names. This check is also
336 integrated in the matcher procedure, with the same properties as the
337 ``ignored_dirs`` one: negated patterns cannot re-include such files and
338 they keep being ignored when the ``modulepath_ignore`` option is
339 disabled.
340 * Dot files: files whose name starts with a dot are hidden but still tested
341 and recorded in cache. Ignore patterns match dot files like any other
342 file and provide a stronger exclusion (no test, no record).
343 * ``module-hide --hard``: hard-hidden modules also behave as nonexistent
344 for users, but their modulefiles are still tested for validity and
345 recorded in cache files, and hiding is applied at query filtering time.
346 ``modulepath-ignore`` acts earlier, at walk time, with no I/O on ignored
347 elements. ``modulepath-ignore`` is meant for content that is not
348 modulefiles, ``module-hide`` for actual modulefiles to withhold.
349 * Virtual modules: a virtual module whose target file matches an ignore
350 pattern is still resolved, as its target is directly read, not found
351 through the modulepath walk.
352
353 Implementation notes
354 --------------------
355
356 * Patterns are compiled once, at definition time, into an ordered list of
357 entries stored in a global structure (e.g.,
358 ``::g_modulepathIgnorePatterns``), each entry holding: anchor directory
359 (empty for global patterns, meaning the root of the walked modulepath),
360 compiled regular expression, negation flag, directory-only flag.
361 * Translating a gitignore pattern into a Tcl regular expression handles the
362 ``*``, ``?``, ``[...]``, ``**`` and anchoring rules. Tcl ``string match``
363 is not sufficient (``*`` would match slashes, no ``**`` support).
364 * A matcher procedure (e.g., ``isPathIgnored modpath fpelt isdir``) applies
365 the *last matching pattern decides* rule and is called from the element
366 loop of ``findModulesFromDirsAndFiles`` and from the walk loop of
367 ``getLimitedAccessesInDirectory``. The walked modulepath is passed to
368 resolve the anchor of global patterns. The rc file exemption is
369 implemented within this procedure and the procedure returns false
370 straight away when the :mconfig:`modulepath_ignore` option is disabled.
371 * Matching is pure string processing: for modulepaths with a large amount
372 of non-modulefiles it replaces one file read per element (magic cookie
373 check) by one in-memory pattern match, which is where the performance
374 gain of this feature comes from.
375 * Code must stay compatible with Tcl 8.5.
376
377 Impacted files
378 --------------
379
380 * ``tcl/mfcmd.tcl``: ``modulepath-ignore`` procedure and pattern
381 compilation helper
382 * ``tcl/interp.tcl.in``: command alias registration in modulerc interpreter
383 * ``tcl/init.tcl.in`` and every file listed in
384 ``doc/source/devel/add-new-config-option.rst`` (shell completion scripts,
385 man pages, config testsuite): ``modulepath_ignore`` configuration option
386 and its ``--no-modulepath-ignore`` command-line switch
387 * ``tcl/modfind.tcl.in``: ``isPathIgnored`` matcher procedure, rc file
388 evaluation moved into ``findModulesFromDirsAndFiles``, ignore matching
389 in walk loop, ``trimElemListForStopmod`` simplification
390 * ``tcl/cache.tcl.in``: ignore matching in ``getLimitedAccessesInDirectory``
391 walk
392 * ``share/nagelfar/syntaxdb_modulerc.tcl``: new command for
393 :subcmd:`lint` sub-command
394 * ``share/vim/syntax/modulefile.vim`` and
395 ``share/emacs/lisp/modulefile-mode.el``: syntax highlighting
396 * ``doc/source/modulefile.rst``: command reference (including the static
397 definition recommendation), ``doc/source/changes.rst`` and ``NEWS.rst``
398 * testsuite: dedicated testfiles for search and direct access behaviors
399 (``modules.20-locate``), listing sub-commands (``modules.90-avail``,
400 ``modules.92-spider``), cache build content and cache use
401 (``modules.30-cache``), plus lint coverage of the new command
402
403 .. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: