| 1 |
.. _modulepath-ignore: |
| 2 |
|
| 3 |
Ignore files in modulepath |
| 4 |
========================== |
| 5 |
|
| 6 |
When walking down the content of a modulepath directory to search for |
| 7 |
modulefiles, every file found is checked to determine if it is a modulefile |
| 8 |
and every sub-directory is walked down. Some modulepath directories contain |
| 9 |
files that are not modulefiles: helper scripts sourced by modulefiles or |
| 10 |
software installation files when modulefiles are mixed with the software |
| 11 |
trees they describe. |
| 12 |
|
| 13 |
Currently the only way to make the module command skip such files is to |
| 14 |
give them a name starting with a dot character. This is not always possible |
| 15 |
or desirable and it does not avoid these files to be tested or read. When |
| 16 |
the amount of non-modulefiles is very large (millions of files), the |
| 17 |
:subcmd:`cachebuild` sub-command takes a huge amount of time and memory as |
| 18 |
an entry is recorded in the cache file for every invalid modulefile found |
| 19 |
(see issue #562 and the discussion on the modules-interest mailing-list |
| 20 |
linked there). |
| 21 |
|
| 22 |
Objective is to let modulepath maintainers describe the content of their |
| 23 |
modulepath directory that should be ignored when searching for modulefiles. |
| 24 |
Ignored files and directories are treated as if they do not exist: they are |
| 25 |
not tested, not walked down, not reported and not recorded in cache files. |
| 26 |
|
| 27 |
``modulepath-ignore`` modulerc command |
| 28 |
-------------------------------------- |
| 29 |
|
| 30 |
A new modulefile command is created: ``modulepath-ignore``. It takes one or |
| 31 |
more file path patterns as arguments:: |
| 32 |
|
| 33 |
modulepath-ignore pattern1 [pattern2 ...] |
| 34 |
|
| 35 |
An error is raised if no argument is provided. |
| 36 |
|
| 37 |
This command is registered as a command alias in the modulerc Tcl |
| 38 |
interpreter (``g_modrcAliases`` in ``execute-modulerc``). It is also |
| 39 |
registered in the modulefile interpreter per-mode alias table (like |
| 40 |
``module-hide``), as global and user rc files are evaluated as modulefiles |
| 41 |
through the ``source`` sub-command mechanism. The command itself raises an |
| 42 |
error when evaluated from a file which is neither a rc file nor evaluated |
| 43 |
in the global/user rc context (tracked by the ``rc_running`` state), so a |
| 44 |
regular modulefile using it gets a *not allowed in modulefile* error. It is |
| 45 |
set no-operation on *refresh* and *scan* evaluation modes. |
| 46 |
|
| 47 |
The command may be called several times: patterns accumulate in their |
| 48 |
definition order into a single ordered list. |
| 49 |
|
| 50 |
The initial specification (issues #561 and #562) also proposed a |
| 51 |
``modulepath-only`` command to switch from an *accept all, ignore some* |
| 52 |
logic to an *ignore all, accept some* logic. This second command is |
| 53 |
dropped: as pattern syntax follows the gitignore syntax (see below), its |
| 54 |
negated patterns already cover this need. For instance to only take ``.mod`` |
| 55 |
files into account:: |
| 56 |
|
| 57 |
modulepath-ignore * !*/ !*.mod |
| 58 |
|
| 59 |
``*`` ignores everything, ``!*/`` re-includes directories so the walk down |
| 60 |
continues, ``!*.mod`` re-includes files with the ``.mod`` extension. |
| 61 |
|
| 62 |
The use case reported on the mailing-list (helper Tcl scripts stored next |
| 63 |
to modulefiles) is solved with:: |
| 64 |
|
| 65 |
modulepath-ignore *.tcl |
| 66 |
|
| 67 |
Pattern syntax |
| 68 |
-------------- |
| 69 |
|
| 70 |
Patterns follow the exact same syntax and behavior as git with |
| 71 |
``.gitignore`` files: |
| 72 |
|
| 73 |
* A pattern without slash (except an optional trailing one) matches files |
| 74 |
or directories at any depth below the directory of the rc file defining |
| 75 |
it. For instance ``*.tcl`` matches ``lib.tcl`` and ``mod/1.0/lib.tcl``. |
| 76 |
* A pattern with a slash at its start or in its middle is anchored to the |
| 77 |
directory of the defining rc file. For instance ``mod/1*`` matches |
| 78 |
``mod/1.0`` but not ``other/mod/1.0``. |
| 79 |
* A pattern ending with a slash only matches directories. |
| 80 |
* ``*`` matches anything except a slash. ``?`` matches any single character |
| 81 |
except a slash. ``[...]`` matches one character from a range or class. |
| 82 |
* A leading ``**/`` means match in any directory (``**/foo`` equals |
| 83 |
``foo``). A trailing ``/**`` matches everything inside the designated |
| 84 |
directory. ``a/**/b`` matches with zero or more intermediate directories. |
| 85 |
* A pattern starting with ``!`` negates a previous match: the element is |
| 86 |
re-included. To match a file name starting with a literal ``!``, escape |
| 87 |
it with a backslash (``\!``). |
| 88 |
* When several patterns match an element, the last matching pattern |
| 89 |
decides. |
| 90 |
* It is not possible to re-include a file if one of its parent directories |
| 91 |
is ignored, as ignored directories are not walked down (same behavior as |
| 92 |
git). |
| 93 |
|
| 94 |
As patterns are passed as Tcl command arguments and not read from a |
| 95 |
dedicated file, the ``.gitignore`` file-format rules (comment lines, blank |
| 96 |
lines, trailing space stripping) do not apply here. |
| 97 |
|
| 98 |
Pattern anchoring and definition scope |
| 99 |
-------------------------------------- |
| 100 |
|
| 101 |
``modulepath-ignore`` is honored in ``.modulerc`` files located within a |
| 102 |
modulepath directory. Preferred location is the ``.modulerc`` file at the |
| 103 |
root of the modulepath, as this file determines how to walk through the |
| 104 |
modulepath content. |
| 105 |
|
| 106 |
No specific restriction is set when the command is used in a ``.version`` |
| 107 |
file: it proceeds like it does in a ``.modulerc`` file, consistently with |
| 108 |
the other modulerc commands which are all accepted in ``.version`` files. |
| 109 |
The purpose of ``.version`` files is however to set the default version of |
| 110 |
a directory, so module documentation will designate ``.modulerc`` as the |
| 111 |
place where ``modulepath-ignore`` should be defined. |
| 112 |
|
| 113 |
Each pattern is recorded along with its anchor directory, which is the |
| 114 |
directory of the rc file defining it. For instance ``modulepath-ignore 1*`` |
| 115 |
in ``/path/to/modpath/foo/.modulerc`` produces a pattern equivalent to |
| 116 |
``/path/to/modpath/foo/1*``. An element is matched against a pattern by |
| 117 |
first checking the pattern anchor directory is one of its parent |
| 118 |
directories, then matching its path relative to this anchor against the |
| 119 |
pattern. |
| 120 |
|
| 121 |
Recording the anchor directory along the pattern means the modulepath root |
| 122 |
directory does not need to be known when the command is evaluated, and |
| 123 |
patterns naturally apply only to the sub-tree of the rc file defining them. |
| 124 |
|
| 125 |
As rc files of parent directories are evaluated before those of deeper |
| 126 |
directories (see next section), patterns from deeper rc files are appended |
| 127 |
after those of parent directories. Combined with the *last matching pattern |
| 128 |
decides* rule, deeper rc files take precedence over parent ones, like git |
| 129 |
does with nested ``.gitignore`` files. |
| 130 |
|
| 131 |
When ``modulepath-ignore`` is evaluated from a rc file not located within a |
| 132 |
modulepath directory (global rc, user rc), it registers global patterns. |
| 133 |
Such patterns have no anchor directory: they apply to every modulepath, |
| 134 |
matched against element paths relative to the root of the walked |
| 135 |
modulepath, as if they were defined in the ``.modulerc`` file at this root. |
| 136 |
As global and user rc files are evaluated at startup, before any modulepath |
| 137 |
walk, global patterns come first in the pattern list: patterns defined |
| 138 |
within a modulepath take precedence over them through the *last matching |
| 139 |
pattern decides* rule. |
| 140 |
|
| 141 |
Evaluation of rc files during modulepath walk |
| 142 |
--------------------------------------------- |
| 143 |
|
| 144 |
Currently ``findModules`` collects the ``.modulerc`` and ``.version`` |
| 145 |
files found during the modulepath walk as ``modulerc`` entries in its |
| 146 |
result list, and ``getModules`` evaluates them afterward, in sorted order, |
| 147 |
during its consolidation phase. |
| 148 |
|
| 149 |
This is too late for ``modulepath-ignore``: patterns must be known when the |
| 150 |
walk occurs. Modulerc evaluation is thus moved to an earlier stage, in the |
| 151 |
``findModulesFromDirsAndFiles`` procedure: |
| 152 |
|
| 153 |
* No ordering requirement is set on the element list returned by |
| 154 |
``getFilesInDirectory``: the rc files of a fetched directory are directly |
| 155 |
looked up by name in the element array built from this list, and the |
| 156 |
remaining elements are sorted in lexical order when inserted in the |
| 157 |
walked list. |
| 158 |
* ``findModulesFromDirsAndFiles`` evaluates the rc files of a directory as |
| 159 |
soon as its content is fetched, prior trimming the element list for the |
| 160 |
*stop_mod* optimization: their ignore patterns and hiding rules are in |
| 161 |
place before any sibling element of this directory is analyzed or |
| 162 |
processed. Once evaluated and recorded in the result list, these rc |
| 163 |
files are withdrawn from the element list: only the rc files that are |
| 164 |
not part of a fetched directory content flow through the walked list and |
| 165 |
get evaluated and recorded by the walk loop (modulepath root rc file |
| 166 |
from the initial listing, limited access rc files walked directly from a |
| 167 |
cache file listing). |
| 168 |
* The walk becomes depth-first: the content of a directory is inserted |
| 169 |
right after it in the walked list instead of being appended at its end. |
| 170 |
Module resolution and symbol definition results depend on the rc file |
| 171 |
evaluation order: a breadth-first walk would evaluate the rc files of |
| 172 |
all first-level directories before any deeper rc file, changing the |
| 173 |
resolution result when a rc file of one sub-tree sets a symbol like |
| 174 |
``default`` on modules of another sub-tree. Root elements are walked in |
| 175 |
lexical order (sorted by ``findModules``) to keep a deterministic |
| 176 |
evaluation order for the rc files of first-level directories. Deeper |
| 177 |
sibling directories are walked in lexical order too, as element paths |
| 178 |
are sorted when inserted in the walked list. |
| 179 |
* At the start of the walk, the ``.modulerc`` file at the root of the |
| 180 |
walked directory is evaluated if it exists and is readable, then every rc |
| 181 |
file already recorded in the result list transmitted to the walk |
| 182 |
procedure is evaluated in lexical order. This transmitted list is empty |
| 183 |
on a regular walk, but on cache use it holds the entries of the evaluated |
| 184 |
cache file: these rc files are not walked through, as they were readable |
| 185 |
when cache was built, but the patterns they may define apply to the walk |
| 186 |
down of the limited access elements. Containing this evaluation in |
| 187 |
``findModulesFromDirsAndFiles`` keeps ``findModulesInCacheFile`` free of |
| 188 |
any rc evaluation logic. |
| 189 |
* On a no-indepth search, ``.version`` files at a different depth level |
| 190 |
than the search target are currently neither recorded nor evaluated |
| 191 |
(unlike ``.modulerc`` files, which have no such restriction). This |
| 192 |
behavior is kept when evaluation moves to walk time, so no extra |
| 193 |
``.version`` evaluation occurs. As a consequence, patterns defined in a |
| 194 |
``.version`` file lying at a different depth level than a no-indepth |
| 195 |
search target are not applied on such a search: one more reason to prefer |
| 196 |
defining patterns in ``.modulerc`` files. |
| 197 |
|
| 198 |
The ``g_rcfilesSourced`` tracking array still guards against evaluating the |
| 199 |
same rc file twice. ``modulerc`` entries are still recorded in the result |
| 200 |
list of ``findModules``: they are needed by the cache build process and to |
| 201 |
transmit cache file entries to the walk procedure. As every rc file is now |
| 202 |
evaluated by ``findModulesFromDirsAndFiles`` whether it comes from a |
| 203 |
directory walk or from a cache file, the consolidation phase of |
| 204 |
``getModules`` does not evaluate rc files anymore: it just withdraws the |
| 205 |
``modulerc`` entries from the result list. |
| 206 |
|
| 207 |
Side benefits of this change: |
| 208 |
|
| 209 |
* rc files are now evaluated in walk order, which guarantees parent |
| 210 |
directory rc files are evaluated before deeper ones. |
| 211 |
* ``trimElemListForStopmod`` does not need to source rc files anymore for |
| 212 |
the *stop_mod* search optimization: the hiding and ignore rules it needs |
| 213 |
are in place as rc files are evaluated prior its call. |
| 214 |
|
| 215 |
Impact on module search |
| 216 |
----------------------- |
| 217 |
|
| 218 |
Every element found during the walk (file or directory) is matched against |
| 219 |
the recorded pattern list. When an element is ignored: |
| 220 |
|
| 221 |
* if it is a file, it is not tested for modulefile validity (no I/O on it) |
| 222 |
and not recorded in the search results |
| 223 |
* if it is a directory, it is not walked down and not recorded |
| 224 |
|
| 225 |
Ignored elements are therefore not reported by :subcmd:`avail`, |
| 226 |
:subcmd:`spider`, :subcmd:`whatis` or any other listing sub-command. A |
| 227 |
direct access to an ignored element (e.g., ``module load`` of it) leads to |
| 228 |
the same *Unable to locate a modulefile* error as a nonexistent file. Same |
| 229 |
result is obtained through the ``is-avail`` modulefile command. |
| 230 |
|
| 231 |
As rc files located within an ignored directory are never reached, the |
| 232 |
aliases, symbolic versions or virtual modules they may define do not exist |
| 233 |
either. |
| 234 |
|
| 235 |
The ``.modulerc`` and ``.version`` files of walked-through directories are |
| 236 |
exempt from ignore matching: they are always evaluated, as they control the |
| 237 |
walk itself (this also makes the ``modulepath-ignore * ...`` idiom work). |
| 238 |
The rc files of ignored directories are naturally never evaluated. |
| 239 |
|
| 240 |
Impact on module cache |
| 241 |
---------------------- |
| 242 |
|
| 243 |
The cache build process (``formatModuleCacheContent``) relies on |
| 244 |
``findModules`` to collect modulepath content: ignored elements are |
| 245 |
excluded from its results, so no ``modulefile-content``, |
| 246 |
``modulefile-invalid`` or directory-related entry is recorded for them in |
| 247 |
the generated ``.modulecache`` file. This solves the initial problem of |
| 248 |
issue #562: non-modulefiles matched by ignore patterns are neither read nor |
| 249 |
recorded, whatever their number, and cache size stays proportional to the |
| 250 |
number of actual modulefiles. |
| 251 |
|
| 252 |
``getLimitedAccessesInDirectory`` performs its own walk of the modulepath |
| 253 |
directory: it must also apply the ignore patterns so that ignored files or |
| 254 |
directories with limited access rights are not recorded as |
| 255 |
``limited-access-file`` or ``limited-access-directory`` entries. Patterns |
| 256 |
are already known at this stage as this procedure is called after |
| 257 |
``findModules`` in the cache build process. |
| 258 |
|
| 259 |
The rc file exemption from pattern matching also applies to this walk: a |
| 260 |
limited access rc file matching an ignore pattern (which happens with the |
| 261 |
*ignore all, accept some* idiom, as ``*`` matches ``.modulerc`` itself) is |
| 262 |
still recorded as a limited access entry. Without this exemption, such a |
| 263 |
file would be skipped by this walk but still collected by the module |
| 264 |
search walk, where rc files are exempt: its content would then get plainly |
| 265 |
recorded in the cache file despite its restricted access. |
| 266 |
|
| 267 |
No new cache file command is introduced and no existing command signature |
| 268 |
changes: the minimal Modules version advertised in the cache file header is |
| 269 |
unchanged. |
| 270 |
|
| 271 |
When a cache file is used: |
| 272 |
|
| 273 |
* ignored elements are simply absent from it, so search and direct access |
| 274 |
behaviors are consistent with the no-cache case |
| 275 |
* limited access entries recorded in cache are re-walked through |
| 276 |
``findModulesFromDirsAndFiles``: ignore patterns are applied on this walk |
| 277 |
too. The rc files recorded in cache are evaluated at the start of this |
| 278 |
walk (their content is obtained from the evaluated cache file), so the |
| 279 |
patterns they may define apply to the walked down limited access |
| 280 |
elements. |
| 281 |
|
| 282 |
As ignored elements are excluded from the cache file at build time, the |
| 283 |
ignore pattern definitions have to be static. If patterns are defined |
| 284 |
dynamically (e.g., depending on user, group or environment), elements |
| 285 |
ignored for the user building the cache are definitively absent from the |
| 286 |
cache file, whatever the dynamic conditions evaluate to for the user of the |
| 287 |
cache. This limitation must be clearly stated in the module documentation: |
| 288 |
``modulepath-ignore`` definitions should not be conditional. |
| 289 |
|
| 290 |
The same caution applies to global patterns defined in the user rc file: |
| 291 |
they are honored when this user builds a cache file, so elements they match |
| 292 |
are excluded from a cache file potentially shared with other users. |
| 293 |
|
| 294 |
Disabling ignore patterns |
| 295 |
------------------------- |
| 296 |
|
| 297 |
A new boolean configuration option controls whether ``modulepath-ignore`` |
| 298 |
definitions are applied: ``modulepath_ignore``. It is enabled by default |
| 299 |
and can be disabled through the ``MODULES_MODULEPATH_IGNORE`` environment |
| 300 |
variable, the :subcmd:`config` sub-command or the ``--no-modulepath-ignore`` |
| 301 |
command-line switch for one execution. |
| 302 |
|
| 303 |
A ``show_ignored`` option name associated with a ``--ignored`` switch (git |
| 304 |
wording, as in ``git status --ignored``) was first considered then |
| 305 |
rejected: disabling the mechanism does not *show* the ignored elements, it |
| 306 |
just stops ignoring them, and they may still not appear anywhere if they |
| 307 |
are not valid modulefiles in the end. The option is thus named after the |
| 308 |
mechanism it enables or disables. |
| 309 |
|
| 310 |
When this option is disabled, ``modulepath-ignore`` commands still evaluate |
| 311 |
without error but the patterns they register are not applied: every element |
| 312 |
of the modulepath directories is walked through and tested as if no pattern |
| 313 |
were defined. |
| 314 |
|
| 315 |
Disabling this option helps troubleshooting, for instance to determine |
| 316 |
whether a modulefile is not found because it is ignored. Note that a cache |
| 317 |
file built with patterns applied does not contain the ignored elements: |
| 318 |
combining with the :option:`--ignore-cache` option is required to also |
| 319 |
search elements excluded from cache file. Similarly, running |
| 320 |
:subcmd:`cachebuild` with ``modulepath_ignore`` disabled records every |
| 321 |
element in the cache file, as if the feature was not used. |
| 322 |
|
| 323 |
Interaction with existing mechanisms |
| 324 |
------------------------------------ |
| 325 |
|
| 326 |
* :mconfig:`ignored_dirs` configuration option: this option globally lists |
| 327 |
directory names to ignore (version control directories by default). |
| 328 |
``modulepath-ignore`` complements it on a per-modulepath basis with a |
| 329 |
much finer pattern syntax. The ``ignored_dirs`` check is integrated in |
| 330 |
the matcher procedure, as an early test made prior applying patterns: |
| 331 |
the behavior of this option is preserved, as negated patterns cannot |
| 332 |
re-include a matching directory and directories keep being ignored when |
| 333 |
the ``modulepath_ignore`` option is disabled. |
| 334 |
* Backup or version control files (``*~``, ``*,v``, ``#*#``): the walk |
| 335 |
procedures historically skipped these file names. This check is also |
| 336 |
integrated in the matcher procedure, with the same properties as the |
| 337 |
``ignored_dirs`` one: negated patterns cannot re-include such files and |
| 338 |
they keep being ignored when the ``modulepath_ignore`` option is |
| 339 |
disabled. |
| 340 |
* Dot files: files whose name starts with a dot are hidden but still tested |
| 341 |
and recorded in cache. Ignore patterns match dot files like any other |
| 342 |
file and provide a stronger exclusion (no test, no record). |
| 343 |
* ``module-hide --hard``: hard-hidden modules also behave as nonexistent |
| 344 |
for users, but their modulefiles are still tested for validity and |
| 345 |
recorded in cache files, and hiding is applied at query filtering time. |
| 346 |
``modulepath-ignore`` acts earlier, at walk time, with no I/O on ignored |
| 347 |
elements. ``modulepath-ignore`` is meant for content that is not |
| 348 |
modulefiles, ``module-hide`` for actual modulefiles to withhold. |
| 349 |
* Virtual modules: a virtual module whose target file matches an ignore |
| 350 |
pattern is still resolved, as its target is directly read, not found |
| 351 |
through the modulepath walk. |
| 352 |
|
| 353 |
Implementation notes |
| 354 |
-------------------- |
| 355 |
|
| 356 |
* Patterns are compiled once, at definition time, into an ordered list of |
| 357 |
entries stored in a global structure (e.g., |
| 358 |
``::g_modulepathIgnorePatterns``), each entry holding: anchor directory |
| 359 |
(empty for global patterns, meaning the root of the walked modulepath), |
| 360 |
compiled regular expression, negation flag, directory-only flag. |
| 361 |
* Translating a gitignore pattern into a Tcl regular expression handles the |
| 362 |
``*``, ``?``, ``[...]``, ``**`` and anchoring rules. Tcl ``string match`` |
| 363 |
is not sufficient (``*`` would match slashes, no ``**`` support). |
| 364 |
* A matcher procedure (e.g., ``isPathIgnored modpath fpelt isdir``) applies |
| 365 |
the *last matching pattern decides* rule and is called from the element |
| 366 |
loop of ``findModulesFromDirsAndFiles`` and from the walk loop of |
| 367 |
``getLimitedAccessesInDirectory``. The walked modulepath is passed to |
| 368 |
resolve the anchor of global patterns. The rc file exemption is |
| 369 |
implemented within this procedure and the procedure returns false |
| 370 |
straight away when the :mconfig:`modulepath_ignore` option is disabled. |
| 371 |
* Matching is pure string processing: for modulepaths with a large amount |
| 372 |
of non-modulefiles it replaces one file read per element (magic cookie |
| 373 |
check) by one in-memory pattern match, which is where the performance |
| 374 |
gain of this feature comes from. |
| 375 |
* Code must stay compatible with Tcl 8.5. |
| 376 |
|
| 377 |
Impacted files |
| 378 |
-------------- |
| 379 |
|
| 380 |
* ``tcl/mfcmd.tcl``: ``modulepath-ignore`` procedure and pattern |
| 381 |
compilation helper |
| 382 |
* ``tcl/interp.tcl.in``: command alias registration in modulerc interpreter |
| 383 |
* ``tcl/init.tcl.in`` and every file listed in |
| 384 |
``doc/source/devel/add-new-config-option.rst`` (shell completion scripts, |
| 385 |
man pages, config testsuite): ``modulepath_ignore`` configuration option |
| 386 |
and its ``--no-modulepath-ignore`` command-line switch |
| 387 |
* ``tcl/modfind.tcl.in``: ``isPathIgnored`` matcher procedure, rc file |
| 388 |
evaluation moved into ``findModulesFromDirsAndFiles``, ignore matching |
| 389 |
in walk loop, ``trimElemListForStopmod`` simplification |
| 390 |
* ``tcl/cache.tcl.in``: ignore matching in ``getLimitedAccessesInDirectory`` |
| 391 |
walk |
| 392 |
* ``share/nagelfar/syntaxdb_modulerc.tcl``: new command for |
| 393 |
:subcmd:`lint` sub-command |
| 394 |
* ``share/vim/syntax/modulefile.vim`` and |
| 395 |
``share/emacs/lisp/modulefile-mode.el``: syntax highlighting |
| 396 |
* ``doc/source/modulefile.rst``: command reference (including the static |
| 397 |
definition recommendation), ``doc/source/changes.rst`` and ``NEWS.rst`` |
| 398 |
* testsuite: dedicated testfiles for search and direct access behaviors |
| 399 |
(``modules.20-locate``), listing sub-commands (``modules.90-avail``, |
| 400 |
``modules.92-spider``), cache build content and cache use |
| 401 |
(``modules.30-cache``), plus lint coverage of the new command |
| 402 |
|
| 403 |
.. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: |