| 1 |
.. _devel-testsuite: |
| 2 |
|
| 3 |
Testsuite structure |
| 4 |
=================== |
| 5 |
|
| 6 |
This document describes the non-regression testsuite of Modules: how it is |
| 7 |
organized, how a test file and a test case are structured, how to run tests, |
| 8 |
how to add new ones, and how to debug a failing test. |
| 9 |
|
| 10 |
The testsuite is built on `DejaGnu <https://www.gnu.org/software/dejagnu/>`_, |
| 11 |
a Tcl/Expect-based framework historically used to test compilers and command |
| 12 |
line tools. DejaGnu groups test files (``.exp``, for *expect script*) under a |
| 13 |
*tool*: a run of ``runtest --tool <name>`` sources every matching ``.exp`` |
| 14 |
file it finds for that tool and reports a ``PASS``/``FAIL``/``XFAIL``/ |
| 15 |
``UNRESOLVED`` line for each individual check performed. Modules defines |
| 16 |
five tools, matched to five kinds of tests (see `Kinds of tests`_ below). |
| 17 |
|
| 18 |
Kinds of tests |
| 19 |
-------------- |
| 20 |
|
| 21 |
The testsuite exercises five different things, run as five separate |
| 22 |
DejaGnu tools: |
| 23 |
|
| 24 |
``modules`` |
| 25 |
The main testsuite. Runs the built :file:`modulecmd.tcl` script (the |
| 26 |
Tcl-based engine behind the :command:`module` command) through every |
| 27 |
supported shell and checks the environment changes and messages it |
| 28 |
produces. This is by far the largest part of the testsuite (around |
| 29 |
25,000 test cases spread over the ``modules.*`` directories described |
| 30 |
below). |
| 31 |
|
| 32 |
``install`` |
| 33 |
Checks a real ``make install``-ed tree: that the :command:`module` |
| 34 |
function/command is correctly defined once shell init scripts are |
| 35 |
sourced, that the ``complete`` modulefile command emits the right |
| 36 |
shell-specific completion registration snippet, that ``modulecmd`` is |
| 37 |
invoked correctly from each shell's wrapper, etc. Driven by the |
| 38 |
:file:`install.00-init` directory. |
| 39 |
|
| 40 |
``lint`` |
| 41 |
Runs static analysis (Nagelfar for Tcl files, ShellCheck for sh/bash/ksh |
| 42 |
scripts) over the repository's own scripts and reports any warning as a |
| 43 |
test failure. Driven by the :file:`lint.00-init` directory. |
| 44 |
|
| 45 |
``completion`` |
| 46 |
Drives a real, interactive shell process (via Expect ``spawn``/``send``/ |
| 47 |
``expect``, not just a captured non-interactive run like the other three |
| 48 |
tools) to press Tab against the built shell completion script and check |
| 49 |
the resulting candidate list -- both that expected module names/option |
| 50 |
flags show up, and that a candidate word built from untrusted text (a |
| 51 |
module name, ``LOADEDMODULES``, ``MODULEPATH``, ...) can never reach a |
| 52 |
shell expansion step. Driven by the :file:`completion.00-init` directory; |
| 53 |
currently covers bash, zsh, fish and tcsh, see `completion.00-init |
| 54 |
layout`_. |
| 55 |
|
| 56 |
``cookbook`` |
| 57 |
Checks the recipes documented under :ref:`cookbook`: for each recipe |
| 58 |
covered, builds a sandboxed fixture from the exact files it ships under |
| 59 |
:file:`doc/example/` and drives it through the commands demonstrated in |
| 60 |
the recipe's ``Usage example`` documentation, parsed straight out of |
| 61 |
the recipe's ``.rst`` file rather than hand-copied (see |
| 62 |
`cookbook.00-init layout`_). Driven by the :file:`cookbook.00-init` |
| 63 |
directory. |
| 64 |
|
| 65 |
Each tool corresponds to one Makefile target (``test``, ``testinstall``, |
| 66 |
``testlint``, ``testcompletion``, ``testcookbook``, see `Running the |
| 67 |
testsuite`_) and to one log file (:file:`modules.log`, :file:`install.log`, |
| 68 |
:file:`lint.log`, :file:`completion.log`, :file:`cookbook.log`) produced in |
| 69 |
the top build directory. |
| 70 |
|
| 71 |
Two additional run modes apply to the ``modules`` tool rather than adding a |
| 72 |
new one: |
| 73 |
|
| 74 |
- *Quick mode* (``QUICKTEST=y``) skips the slower/more exhaustive checks |
| 75 |
guarded by a call to ``skip_if_quick_mode`` (see `Base test procedures`_) |
| 76 |
to get a fast (~1,900 cases, ~1 min) smoke test. |
| 77 |
- *Coverage mode* (``COVERAGE=y``) runs the exact same tests but through a |
| 78 |
Nagelfar-instrumented ``modulecmd.tcl``, then produces marked-up |
| 79 |
:file:`tcl/*.tcl_m` files where lines never hit during the run are flagged |
| 80 |
with ``;# Not covered``. |
| 81 |
|
| 82 |
Directory layout |
| 83 |
---------------- |
| 84 |
|
| 85 |
Everything lives under :file:`testsuite/`. |
| 86 |
|
| 87 |
Test file directories |
| 88 |
~~~~~~~~~~~~~~~~~~~~~ |
| 89 |
|
| 90 |
Test files are grouped in numbered directories named |
| 91 |
``<tool>.<serienum>-<topic>``, e.g. :file:`modules.50-cmds`, |
| 92 |
:file:`install.00-init`, :file:`lint.00-init`, :file:`completion.00-init`, |
| 93 |
:file:`cookbook.00-init`. The ``<tool>`` prefix ties the directory to one of |
| 94 |
the five DejaGnu tools above (DejaGnu only looks at directories whose |
| 95 |
prefix matches the ``--tool`` given to ``runtest``); the two-digit |
| 96 |
``<serienum>`` number fixes run order |
| 97 |
and is what you pass to :file:`script/mt` to select a whole directory (e.g. |
| 98 |
``script/mt 50``); the ``<topic>`` suffix is just a human-readable label. |
| 99 |
|
| 100 |
Current ``modules.*`` series, in run order: |
| 101 |
|
| 102 |
``00-init`` |
| 103 |
Testsuite bootstrap, plus a handful of standalone option/behavior |
| 104 |
checks. Two files in this series matter beyond their own file number |
| 105 |
because every other series relies on what they set up: |
| 106 |
|
| 107 |
- ``005-init_ts.exp`` defines the global variables and procedures |
| 108 |
shared by every test file for the whole run (paths, shell lists, |
| 109 |
common error/message strings, helper procs such as |
| 110 |
``cmpversion``). It runs first, before anything else. |
| 111 |
- ``010-environ.exp`` defines the *initial user environment* every |
| 112 |
test starts from: it clears/resets ``MODULEPATH``, |
| 113 |
``LOADEDMODULES``, and any quarantine/auto-handling/color/... |
| 114 |
configuration coming from the calling shell. The same file (by |
| 115 |
role, not by sharing code) also exists as |
| 116 |
:file:`install.00-init/010-environ.exp` and plays the identical |
| 117 |
part for the ``install`` tool. |
| 118 |
|
| 119 |
The rest of the series defines the ``_test_sub`` procedure that runs |
| 120 |
``modulecmd.tcl`` (``006-procs.exp``, tool-specific -- see `Running |
| 121 |
the command and checking the result`_), builds further shared |
| 122 |
fixtures (module search path, module cache pre-build, |
| 123 |
``save_test_env`` checkpoint), and tests standalone command-line |
| 124 |
switches and configuration behavior (pager, quarantine, siteconfig, |
| 125 |
auto handling, color, access rights, multilib, cwd, logger, Tcl |
| 126 |
extension library). |
| 127 |
``10-use`` |
| 128 |
``module use``/``unuse`` |
| 129 |
``20-locate`` |
| 130 |
Module lookup/resolution (exact version, default, wildcard, |
| 131 |
symbolic version, ...) |
| 132 |
``30-cache`` |
| 133 |
Module cache file handling |
| 134 |
``50-cmds`` |
| 135 |
Modulefile commands (``setenv``, ``prepend-path``, ``conflict``, |
| 136 |
``variant``, ...) -- by far the biggest series |
| 137 |
``51-scan`` |
| 138 |
Modulefile scan / extra-match search |
| 139 |
``60-initx`` |
| 140 |
The ``init*`` sub-commands (``initadd``, ``initprepend``, ``initrm``, |
| 141 |
``initswitch``, ``initlist``, ``initclear``) that edit a user's shell |
| 142 |
startup file |
| 143 |
``61-coll`` |
| 144 |
Module collections |
| 145 |
``70-maint`` |
| 146 |
Maintenance sub-commands (``load``, ``switch``, ``purge``, config, |
| 147 |
...) |
| 148 |
``80-deep`` |
| 149 |
"Deep" modules, i.e. modules whose name has more than one directory |
| 150 |
level (``name/sub/version``): load/unload/switch, listing, ``whatis``, |
| 151 |
alias/symbolic-version resolution, access rights |
| 152 |
``90-avail`` |
| 153 |
``avail`` sub-command |
| 154 |
``91-sort`` |
| 155 |
Module sorting order |
| 156 |
``92-spider`` |
| 157 |
``spider`` sub-command |
| 158 |
``95-version`` |
| 159 |
Version comparison/parsing |
| 160 |
``99-finish`` |
| 161 |
Testsuite teardown (removes cache files created for the run) |
| 162 |
|
| 163 |
``install.00-init``, ``lint.00-init``, ``completion.00-init`` and |
| 164 |
``cookbook.00-init`` are each a single series (those tools are much smaller |
| 165 |
and don't need topic splitting). |
| 166 |
|
| 167 |
Every series directory ends with a ``999-cleanup.exp`` file (see `Test file |
| 168 |
anatomy`_) and, for the ``modules`` tool, most series begin with a |
| 169 |
``0NN-init_ts.exp`` file that sets up whatever fixtures that series' tests |
| 170 |
need (e.g. :file:`modules.90-avail/010-init_ts.exp`). |
| 171 |
|
| 172 |
``completion.00-init`` layout |
| 173 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 174 |
|
| 175 |
The ``completion`` tool follows the common ``005``/``006``/``010``/``011`` |
| 176 |
setup numbering (see `Test file anatomy`_ and `Base test procedures`_ below) |
| 177 |
plus one extra layer, since driving real Tab-key completion is inherently |
| 178 |
shell-specific in a way none of the other three tools are: |
| 179 |
|
| 180 |
- ``005-init_ts.exp`` / ``006-procs.exp`` / ``010-environ.exp`` / |
| 181 |
``011-save_test_env.exp`` set up paths (the ``bash``, ``zsh``, ``fish`` and |
| 182 |
``tcsh`` binaries, the built :file:`init/bash_completion`, |
| 183 |
:file:`init/zsh-functions/_module`, :file:`init/fish_completion` and |
| 184 |
:file:`init/tcsh_completion` scripts, a clean fixture modulepath), the |
| 185 |
shell-agnostic assert procedures (``completion_assert_contains``, |
| 186 |
``completion_assert_not_contains``, ``completion_assert_eq``, |
| 187 |
``completion_assert_no_exec``), and the clean baseline |
| 188 |
environment/``save_test_env`` checkpoint, exactly as for the other tools. |
| 189 |
- ``0NN-<shell>-procs.exp`` defines one ``completion_<shell>_start`` / |
| 190 |
``completion_<shell>_raw`` / ``completion_<shell>_list`` / |
| 191 |
``completion_<shell>_inline`` / ``completion_<shell>_close`` set per |
| 192 |
shell -- e.g. ``020-bash-procs.exp`` spawns a real ``bash`` pty, sources |
| 193 |
:file:`init/bash_completion`, and drives double-Tab listings |
| 194 |
(``completion_bash_list``, for an ambiguous prefix) or single-Tab inline |
| 195 |
completions (``completion_bash_inline``, for a prefix with exactly one |
| 196 |
match -- e.g. checking a directory-style entry completes with a |
| 197 |
trailing ``/`` and no trailing space) through Expect. |
| 198 |
``completion_<shell>_list``/``_inline`` are responsible for recording the |
| 199 |
cmdline they were passed into the shared ``completion_last_cmdline`` |
| 200 |
variable, which the generic assert procedures use to build their test |
| 201 |
label. |
| 202 |
- ``0NN-<shell>.exp`` (e.g. ``021-bash.exp``, ``031-zsh.exp``, |
| 203 |
``041-fish.exp``, ``051-tcsh.exp``) holds the actual test cases for that |
| 204 |
shell, calling only its own ``completion_<shell>_*`` procs plus the shared |
| 205 |
asserts. A shell whose completion script covers less ground than the |
| 206 |
others (e.g. fish's has no ``ml`` support and does not gate its option |
| 207 |
flags by sub-command; the tcsh ``ml`` has no global switches at all, and |
| 208 |
its ``unuse`` only lists modulepaths from a bare, empty word) documents |
| 209 |
each such gap in its own file's header instead of forcing every file to |
| 210 |
test |
| 211 |
the same thing. |
| 212 |
|
| 213 |
Adding a new shell means adding its own ``completion_<shell>_*`` procs file |
| 214 |
and test file; nothing in ``006-procs.exp`` needs to change. |
| 215 |
|
| 216 |
``cookbook.00-init`` layout |
| 217 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 218 |
|
| 219 |
The ``cookbook`` tool follows the common ``005``/``006``/``010``/``011`` |
| 220 |
setup numbering (see `Test file anatomy`_ and `Base test procedures`_ |
| 221 |
below), driving ``modulecmd.tcl`` directly through ``testouterr_cmd``/ |
| 222 |
``testouterr_cmd_re`` exactly like the ``modules`` tool, plus one shared |
| 223 |
helper that is specific to this tool: |
| 224 |
|
| 225 |
- ``005-init_ts.exp`` defines ``cookbookdocdir``/``cookbookexampledir`` |
| 226 |
(pointing at :file:`doc/source/cookbook` and :file:`doc/example`), the |
| 227 |
``cookbooksandbox`` fixture root (recreated fresh for the whole run), |
| 228 |
``cookbook_read_example_file``/``cookbook_write_file`` (read a file |
| 229 |
shipped under :file:`doc/example/<name>/`, applying a ``string map`` |
| 230 |
substitution list, and write it into the sandbox), and |
| 231 |
``cookbook_parse_transcript`` -- parses the ``.. parsed-literal::`` blocks |
| 232 |
of a named section (e.g. ``Usage example``) of a recipe's ``.rst`` file |
| 233 |
and returns the ordered list of ``{command output}`` pairs shown there, |
| 234 |
role markup (``:sgrhi:`text``` and friends) stripped from the output, so |
| 235 |
a recipe test file drives the exact command sequence its own |
| 236 |
documentation demonstrates, in that order, and checks its own real output |
| 237 |
is similar to the one illustrated -- instead of both being a hand-copied |
| 238 |
duplicate that could silently drift from the documentation. |
| 239 |
``cookbook_output_re`` turns one such illustrated output into the regexp |
| 240 |
a sandboxed run's real output is checked against: escaped as a literal, |
| 241 |
except a run of ``-`` padding (widened to ``[-]+``, since its exact width |
| 242 |
depends on terminal-width detection a non-tty test run does not have) and |
| 243 |
whatever literal substrings the recipe test passes it to widen too (e.g. |
| 244 |
a placeholder path mapped to the sandbox's real one). |
| 245 |
- ``006-procs.exp``/``010-environ.exp``/``011-save_test_env.exp`` set up |
| 246 |
``_test_sub``, a clean baseline environment (sandboxed ``$HOME``, no |
| 247 |
color, deterministic ``avail``/``list`` output, ``MODULES_SITECONFIG``/ |
| 248 |
``MODULES_TAG_ABBREV``/``MODULES_NON_EXPORTABLE_TAGS`` left unset so each |
| 249 |
recipe states what it actually relies on), and the ``save_test_env`` |
| 250 |
checkpoint, exactly as for the other tools. |
| 251 |
- ``0NN-<recipe-name>.exp`` (e.g. ``020-sync-remote-appdir.exp``), named |
| 252 |
after the recipe's own ``.rst`` file, holds one recipe's test: it checks |
| 253 |
any external binary the recipe needs (``unsupported`` + ``return`` if |
| 254 |
missing, e.g. ``rsync`` for ``sync-remote-appdir``), builds a |
| 255 |
``$cookbooksandbox/<name>`` fixture from the recipe's real |
| 256 |
:file:`doc/example/<name>/` files via ``cookbook_read_example_file`` (only |
| 257 |
substituting whatever real-system absolute path the recipe hardcodes, |
| 258 |
e.g. ``/remote_apps``, for a sandbox path -- everything else is used byte |
| 259 |
for byte), points the test environment at that sandbox, then loops over |
| 260 |
``cookbook_parse_transcript``'s steps in order, running each with |
| 261 |
``testouterr_cmd_re`` against ``cookbook_output_re`` of its illustrated |
| 262 |
output (stdout, the env-var-assignment syntax a shell evals silently and |
| 263 |
no recipe doc illustrates, is still checked against a hand-built answer |
| 264 |
per step -- ``unresolved`` + ``return`` first if the transcript's step |
| 265 |
count no longer matches what the test knows how to build one for, as a |
| 266 |
signal it needs updating alongside the doc). Simulates each step's |
| 267 |
effect on the environment for the next one exactly as a real shell would |
| 268 |
(``setenv_loaded_module``/``setenv_path_var``), since every |
| 269 |
``modulecmd.tcl`` invocation here is an independent process. |
| 270 |
|
| 271 |
Adding a new recipe means adding its own ``0NN-<recipe-name>.exp`` file; |
| 272 |
nothing in ``005-init_ts.exp`` needs to change unless the new recipe needs a |
| 273 |
kind of substitution or transcript shape the existing helpers do not cover |
| 274 |
yet. |
| 275 |
|
| 276 |
Fixture and support directories |
| 277 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 278 |
|
| 279 |
- :file:`modulefiles/`, :file:`modulefiles.2/`, :file:`modulefiles.3/`, |
| 280 |
:file:`modulefiles.4/`, and a number of single-purpose |
| 281 |
:file:`modulefiles.<name>/` directories (``.deep``, ``.deps``, ``.eb``, |
| 282 |
``.spider1``/``.spider2``/``.spider3``, ``.rc``, ``.path1``/``.path2``/ |
| 283 |
``.path3``, ``.memcache``, ``.indepth``, ``.allin``) hold the modulefiles |
| 284 |
used as fixtures. See `Adding new test fixtures`_ below for which one to |
| 285 |
add to. |
| 286 |
- :file:`config/` holds the DejaGnu/Expect configuration shared by every |
| 287 |
``.exp`` file: :file:`base-config.exp` (all the test procedures, see |
| 288 |
`Base test procedures`_) and :file:`unix.exp` (the low-level plumbing |
| 289 |
that spawns ``modulecmd.tcl`` and captures its stdout/stderr/exit code). |
| 290 |
- :file:`etc/`, :file:`home/`, :file:`home.2/`-:file:`home.4/` provide |
| 291 |
sandboxed ``$MODULESHOME``-like/``$HOME``-like trees (rc files, |
| 292 |
collections, ...) used as fixtures. |
| 293 |
- :file:`bin/` contains small stand-in executables used by tests (fake |
| 294 |
tools invoked by modulefiles or by shell code under test). |
| 295 |
- Miscellaneous single-purpose helper scripts at the top level: |
| 296 |
:file:`is_func_defined` / :file:`is_func_defined.fish` (check whether |
| 297 |
``module``/``mogui`` is defined as a shell function before running the |
| 298 |
``install`` tests), :file:`stdin_to_file`, :file:`stty`, :file:`mode`, |
| 299 |
:file:`not_installed`, :file:`systest`/:file:`systest0-2`, :file:`id`, |
| 300 |
:file:`cmd.exe`, :file:`manpath`, :file:`virttargets/`. |
| 301 |
- :file:`mb/` holds fixtures for the :file:`script/mb` benchmark tool, not |
| 302 |
for DejaGnu tests. |
| 303 |
|
| 304 |
Test file anatomy |
| 305 |
----------------- |
| 306 |
|
| 307 |
Every ``.exp`` file follows the same three-part shape: a header comment |
| 308 |
block, the test content, and (for most files, though not the ``00-init`` |
| 309 |
setup files) a cleanup footer. |
| 310 |
|
| 311 |
Header |
| 312 |
~~~~~~ |
| 313 |
|
| 314 |
A fixed-format comment block, kept for historical (SCCS/CVS-style keyword |
| 315 |
expansion) reasons and consistency rather than for tooling: |
| 316 |
|
| 317 |
.. code-block:: tcl |
| 318 |
|
| 319 |
############################################################################## |
| 320 |
# Modules Revision 3.0 |
| 321 |
# Providing a flexible user environment |
| 322 |
# |
| 323 |
# File: modules.50-cmds/%M% |
| 324 |
# Revision: %I% |
| 325 |
# First Edition: 2021/03/04 |
| 326 |
# Last Mod.: %U%, %G% |
| 327 |
# |
| 328 |
# Authors: Your Name, [email protected] |
| 329 |
# |
| 330 |
# Description: Testuite testsequence |
| 331 |
# Command: load, display, help, test |
| 332 |
# Modulefiles: variant |
| 333 |
# Sub-Command: |
| 334 |
# |
| 335 |
# Comment: %C{ |
| 336 |
# Test 'variant' modulefile command |
| 337 |
# }C% |
| 338 |
# |
| 339 |
############################################################################## |
| 340 |
|
| 341 |
When adding a file, copy this header from a neighboring file in the same |
| 342 |
directory, update ``First Edition`` to the current date, your name/email in |
| 343 |
``Authors`` if you want authorship credit, and fill in ``Command:``/ |
| 344 |
``Modulefiles:``/``Sub-Command:`` and the free-text ``Comment:`` block to |
| 345 |
describe what the file tests. The ``%M%``/``%I%``/``%U%``/``%G%`` tokens are |
| 346 |
left as-is (they are not expanded by anything in this codebase but are kept |
| 347 |
for consistency with the historical format). |
| 348 |
|
| 349 |
Content |
| 350 |
~~~~~~~ |
| 351 |
|
| 352 |
The body of the file is plain Tcl/Expect code, structured as: |
| 353 |
|
| 354 |
1. Optional per-file setup: pick/point to a modulepath (e.g. |
| 355 |
``set mp $modpath.3``), set environment variables relevant to the |
| 356 |
scenario (``setenv_var``/``setenv_path_var``), maybe call |
| 357 |
``skip_if_quick_mode`` or ``skip_if_os_in`` right after the setup that |
| 358 |
only the skipped tests need. |
| 359 |
2. One or more test cases: each is one call to a ``test*_cmd*`` procedure |
| 360 |
from :file:`config/base-config.exp` (see `Base test procedures`_), |
| 361 |
grouped under a short ``#\n# <description>\n#`` comment banner when the |
| 362 |
file covers more than one scenario. |
| 363 |
3. A ``#\n# Cleanup\n#`` banner followed by a call to ``reset_test_env`` |
| 364 |
(see below), for any file that changed environment variables or Tcl |
| 365 |
globals the following files shouldn't see. |
| 366 |
|
| 367 |
Footer |
| 368 |
~~~~~~ |
| 369 |
|
| 370 |
Files that mutate shared state end with: |
| 371 |
|
| 372 |
.. code-block:: tcl |
| 373 |
|
| 374 |
# |
| 375 |
# Cleanup |
| 376 |
# |
| 377 |
|
| 378 |
reset_test_env |
| 379 |
|
| 380 |
``reset_test_env`` restores every Tcl global variable and environment |
| 381 |
variable to the value it had when ``save_test_env`` last ran (this checkpoint |
| 382 |
is taken once per tool, early in the ``00-init`` series, right after the |
| 383 |
common fixtures -- modulepath, cache, etc. -- are set up; see |
| 384 |
:file:`modules.00-init/085-save_test_env.exp`). This is what makes test files |
| 385 |
independent of run order within reasonable bounds: a file can freely |
| 386 |
``setenv_var``, build a collection, flip a config option, etc., and the next |
| 387 |
file starts clean. |
| 388 |
|
| 389 |
A handful of files close with a Vim modeline instead (or in addition): |
| 390 |
|
| 391 |
.. code-block:: tcl |
| 392 |
|
| 393 |
# vim:set tabstop=3 shiftwidth=3 expandtab autoindent: |
| 394 |
|
| 395 |
How a test case is built |
| 396 |
------------------------ |
| 397 |
|
| 398 |
A test case has two ingredients: the environment/fixtures it runs against |
| 399 |
(set up with plain Tcl and the ``setenv_*``/``unsetenv_*`` helpers), and one |
| 400 |
call to a ``test*_cmd*`` procedure that runs a :command:`module` command line |
| 401 |
and checks its outcome. |
| 402 |
|
| 403 |
Setting up environment |
| 404 |
~~~~~~~~~~~~~~~~~~~~~~ |
| 405 |
|
| 406 |
- ``setenv_var var val`` / ``unsetenv_var var`` -- set/unset a plain |
| 407 |
environment variable for the *modulecmd.tcl* child process about to be |
| 408 |
spawned. |
| 409 |
- ``setenv_path_var var elt1 elt2 ...`` / ``unsetenv_path_var var`` -- same, |
| 410 |
for a colon-delimited path-like variable; also transparently maintains the |
| 411 |
matching ``__MODULES_SHARE_<var>`` reference-count variable. |
| 412 |
- ``setenv_loaded_module modlist modfilelist`` / |
| 413 |
``unsetenv_loaded_module`` -- set up ``LOADEDMODULES``/``_LMFILES_`` (and |
| 414 |
``__MODULES_LMTAG`` for auto-loaded modules) to simulate modules already |
| 415 |
loaded prior to the command under test. |
| 416 |
- ``skip_if_quick_mode`` -- bail out of the rest of the *current file* when |
| 417 |
``QUICKTEST=y``. Used to skip expensive/exhaustive variations while |
| 418 |
keeping at least one representative case running in quick mode. |
| 419 |
- ``skip_if_os_in os1 os2 ...`` -- bail out of the rest of the current file |
| 420 |
on the given OS names (``$::os_name``, e.g. ``windows``, ``darwin``). |
| 421 |
- ``change_file_perms``/``restore_file_perms`` -- temporarily lock down a |
| 422 |
file/directory's permissions to test permission-denied paths. |
| 423 |
|
| 424 |
Running the command and checking the result |
| 425 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 426 |
|
| 427 |
Every ``test*_cmd*`` procedure runs ``modulecmd <shell> <cmd>`` through |
| 428 |
``_test_sub`` and compares the captured stdout/stderr/exit code against |
| 429 |
what was passed in, reporting ``pass``/``fail`` to DejaGnu. ``_test_sub`` |
| 430 |
itself is tool-specific: each tool defines its own version in its |
| 431 |
``00-init`` series' ``006-procs.exp`` file (e.g. |
| 432 |
:file:`modules.00-init/006-procs.exp`), which for the ``modules`` tool |
| 433 |
calls down into ``modulecmd_xxx_`` (defined in :file:`config/unix.exp`) to |
| 434 |
actually spawn ``modulecmd.tcl`` and capture its output. |
| 435 |
|
| 436 |
See `Base test procedures`_ for the full family of these procedures and how |
| 437 |
their ``answer``/``anserr`` arguments are interpreted (plain string, a |
| 438 |
"shell out list", or the ``OK``/``ERR``/``ERR2`` shorthands). |
| 439 |
|
| 440 |
Base test procedures |
| 441 |
-------------------- |
| 442 |
|
| 443 |
All of the below live in :file:`testsuite/config/base-config.exp` |
| 444 |
(``shell_*`` internal helpers, used by the procedures below to translate a |
| 445 |
symbolic environment change into the actual syntax of each of the 15+ |
| 446 |
supported shells, are not covered here). |
| 447 |
|
| 448 |
Environment/output description helpers |
| 449 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 450 |
|
| 451 |
``shell_out test_shell out_list`` builds the shell-specific expected output |
| 452 |
for a *list* of symbolic environment changes -- this is what most test cases |
| 453 |
pass as their ``answer``/``anserr`` argument instead of a literal string. |
| 454 |
Each element of ``out_list`` is itself a small list starting with one of: |
| 455 |
|
| 456 |
``set`` |
| 457 |
``{set var val}`` -- variable assignment (pass ``noescval`` as a |
| 458 |
4th element to skip shell escaping) |
| 459 |
``setpath`` |
| 460 |
``{setpath var val}`` -- like ``set``, but also auto-generates the |
| 461 |
matching ``__MODULES_SHARE_<var>`` reference-count assignment |
| 462 |
``unset`` |
| 463 |
``{unset var}`` -- variable unset |
| 464 |
``unsetpath`` |
| 465 |
``{unsetpath var}`` -- like ``unset``, plus reference count |
| 466 |
``alias`` |
| 467 |
``{alias var val}`` -- shell alias definition |
| 468 |
``unalias`` |
| 469 |
``{unalias var}`` -- shell alias removal |
| 470 |
``chdir`` |
| 471 |
``{chdir dir}`` -- directory change |
| 472 |
``xres`` |
| 473 |
``{xres var val}`` -- X11 resource set (``xrdb``) |
| 474 |
``unxres`` |
| 475 |
``{unxres var}`` -- X11 resource removal |
| 476 |
``text`` |
| 477 |
``{text str}`` -- arbitrary literal text (module messages, |
| 478 |
warnings, ...), inserted as-is |
| 479 |
``OK``, ``ERR``, ``ERR2`` |
| 480 |
shell-specific success / 1-error / 2-error status code |
| 481 |
``out``, or anything else |
| 482 |
literal text joined as-is |
| 483 |
|
| 484 |
``is_shell_out_list answer`` tells whether ``answer`` looks like one of |
| 485 |
these lists (used internally by ``_test_out``/``_test_out_re`` to decide |
| 486 |
whether to run it through ``shell_out`` or treat it as a literal string). |
| 487 |
|
| 488 |
Test procedures (the ones test files actually call) |
| 489 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 490 |
|
| 491 |
All of them share the same first two arguments, ``test_shell`` (a shell |
| 492 |
name, or ``ALL`` to run the same check against every shell in |
| 493 |
``$supported_shells``) and ``cmd`` (the :command:`module` command line, |
| 494 |
without the leading ``module`` word), plus a trailing optional |
| 495 |
``failcmd`` (defaults to ``fail``; some tests in the deep-reload/args series |
| 496 |
pass ``untested`` or ``unresolved`` here) and ``launcher`` (to prefix the |
| 497 |
call, e.g. with ``sudo``). |
| 498 |
|
| 499 |
``testouterr_cmd`` is the preferred procedure for a new test case: it |
| 500 |
checks both stdout and stderr, so a message unexpectedly landing on the |
| 501 |
wrong stream does not go unnoticed, and it is by far the most used |
| 502 |
procedure in the existing testsuite. Reach for one of the others only when |
| 503 |
its extra check (exit code, regexp matching, file content, ...) or its |
| 504 |
narrower scope (single-stream only) is actually needed. |
| 505 |
|
| 506 |
``test_cmd test_shell cmd answer {exitval 0} {failcmd fail} {launcher {}}`` |
| 507 |
Checks stdout (exact) and the exit code (default expected: ``0``). |
| 508 |
``test_cmd_re test_shell cmd answer {failcmd fail} {launcher {}}`` |
| 509 |
Checks stdout (regexp). |
| 510 |
``testerr_cmd test_shell cmd answer {failcmd fail} {launcher {}}`` |
| 511 |
Checks stderr (exact). |
| 512 |
``testerr_cmd_re test_shell cmd answer {force_nl 0} {failcmd fail} {launcher {}}`` |
| 513 |
Checks stderr (regexp). ``force_nl`` forces a trailing newline to be |
| 514 |
appended to ``answer`` before it is used as a regexp, for the cases |
| 515 |
where that can't be inferred automatically. |
| 516 |
``testouterr_cmd test_shell cmd answer anserr {failcmd fail} {launcher {}}`` |
| 517 |
Checks stdout (exact, ``answer``) and stderr (exact, ``anserr``). |
| 518 |
**Preferred procedure, see above.** |
| 519 |
``testouterr_cmd_re test_shell cmd answer anserr {force_nl 0} {failcmd fail} {launcher {}}`` |
| 520 |
Checks stdout (regexp, ``answer``) and stderr (regexp, ``anserr``). |
| 521 |
``testouterr_cmd_re_sort test_shell cmd answer anserr {failcmd fail} {launcher {}}`` |
| 522 |
Checks stdout (regexp) and stderr (regexp), both sides line-sorted |
| 523 |
first -- for output whose line order is not guaranteed. |
| 524 |
``testall_cmd test_shell cmd answer anserr exitval {failcmd fail} {launcher {}}`` |
| 525 |
Checks stdout (exact), stderr (exact) and the exit code. |
| 526 |
``testall_cmd_re test_shell cmd answer anserr exitval {force_nl 0} {failcmd fail} {launcher {}}`` |
| 527 |
Checks stdout (regexp), stderr (regexp) and the exit code. |
| 528 |
``testinouterr_cmd test_shell cmd input answer anserr {failcmd fail} {launcher {}}`` |
| 529 |
Like ``testouterr_cmd`` but also feeds ``input`` on stdin. |
| 530 |
``testoutfile_cmd test_shell cmd answer filepath ansfile {failcmd fail} {launcher {}}`` |
| 531 |
Checks stdout (exact, ``answer``) and the content of ``filepath`` on |
| 532 |
disk (exact, ``ansfile``). |
| 533 |
``testouterrfile_cmd test_shell cmd answer anserr filepath ansfile {failcmd fail} {launcher {}}`` |
| 534 |
Checks stdout (exact), stderr (exact) and the content of ``filepath`` |
| 535 |
on disk (exact, ``ansfile``). |
| 536 |
``testouterrfileglob_cmd test_shell cmd answer anserr fileglob ansfile {failcmd fail} {launcher {}}`` |
| 537 |
Like ``testouterrfile_cmd``, but the file to check is resolved from |
| 538 |
the ``fileglob`` glob pattern (last match once sorted). |
| 539 |
``testouterrgloblist_cmd test_shell cmd answer anserr fileglob ansglob {failcmd fail} {launcher {}}`` |
| 540 |
Checks stdout (exact), stderr (exact) and the *sorted list* of files |
| 541 |
matching the ``fileglob`` glob pattern (against ``ansglob``). |
| 542 |
|
| 543 |
For the ``ERR``/``ERR2``/``OK`` shorthands accepted as ``answer``/``anserr``: |
| 544 |
``OK`` means "no output expected", ``ERR``/``ERR2`` expand to the |
| 545 |
shell-specific representation of exit status 1/2 (via ``shell_err``). |
| 546 |
|
| 547 |
Environment save/restore and mode helpers |
| 548 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 549 |
|
| 550 |
- ``save_test_env`` / ``reset_test_env`` -- see `Footer`_ above. |
| 551 |
- ``is_quick_mode`` / ``skip_if_quick_mode`` -- see `Setting up |
| 552 |
environment`_ above. |
| 553 |
- ``skip_if_os_in`` -- see `Setting up environment`_ above. |
| 554 |
|
| 555 |
A worked example |
| 556 |
---------------- |
| 557 |
|
| 558 |
The following two examples put together everything from `How a test case is |
| 559 |
built`_ and `Base test procedures`_ into concrete, runnable test cases: one |
| 560 |
where the load is rejected, one where it succeeds. |
| 561 |
|
| 562 |
Example 1: a rejected load (conflict) |
| 563 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 564 |
|
| 565 |
This is a trimmed-down version of one of the checks in |
| 566 |
:file:`modules.50-cmds/131-conflict-module.exp`: loading a module while a |
| 567 |
conflicting one is already loaded, with automatic conflict-unload disabled |
| 568 |
(the project's default), must fail with an explanatory error. |
| 569 |
|
| 570 |
The scenario relies on two existing fixture modulefiles: ``trace/all_on`` |
| 571 |
and ``conflict/module`` (:file:`testsuite/modulefiles/conflict/module`), |
| 572 |
whose ``.modulerc``-less content simply declares ``conflict trace``. |
| 573 |
|
| 574 |
.. code-block:: tcl |
| 575 |
|
| 576 |
# environment setup |
| 577 |
# |
| 578 |
# MODULEPATH already points to $modpath (the default fixture directory, |
| 579 |
# testsuite/modulefiles) -- that is set up once for the whole run by |
| 580 |
# modules.00-init/050-modpath.exp, so there is nothing to do here unless |
| 581 |
# a test needs a different modulepath. Simulate that 'trace/all_on' is |
| 582 |
# already loaded, without actually running a 'load' command for it: |
| 583 |
setenv_loaded_module trace/all_on $modpath/trace/all_on |
| 584 |
|
| 585 |
# build the expected stdout ($ans) |
| 586 |
# |
| 587 |
# the conflict is detected before any environment change is made, so |
| 588 |
# the load is rejected outright: stdout only carries the shell's |
| 589 |
# failing exit status, i.e. the 'ERR' shorthand (see `Base test |
| 590 |
# procedures`_) |
| 591 |
set ans ERR |
| 592 |
|
| 593 |
# build the expected stderr ($tserr) |
| 594 |
# |
| 595 |
# 'module load' always echoes a "Loading <mod>" banner first; the |
| 596 |
# msg_load and err_conflict helpers (defined in |
| 597 |
# modules.00-init/005-init_ts.exp) build that banner and the |
| 598 |
# conflict/hint message actually printed by modulecmd.tcl |
| 599 |
set tserr [msg_load conflict/module [err_conflict trace/all_on]] |
| 600 |
|
| 601 |
# test launch |
| 602 |
# |
| 603 |
# run 'module load conflict/module' for every supported shell and check |
| 604 |
# its stdout against $ans and its stderr against $tserr |
| 605 |
testouterr_cmd ALL "load conflict/module" $ans $tserr |
| 606 |
|
| 607 |
Running this (e.g. pasted at the end of an existing test file, or as its own |
| 608 |
numbered ``.exp`` file, see `Test file anatomy`_) exercises the real |
| 609 |
:command:`module load` code path for every supported shell and would report |
| 610 |
one ``PASS``/``FAIL`` per shell. For reference, this is what |
| 611 |
``modulecmd.tcl`` itself prints for the ``sh`` shell in this exact scenario:: |
| 612 |
|
| 613 |
$ MODULEPATH=.../modulefiles LOADEDMODULES=trace/all_on \ |
| 614 |
_LMFILES_=.../modulefiles/trace/all_on \ |
| 615 |
modulecmd.tcl sh load conflict/module |
| 616 |
Loading conflict/module |
| 617 |
ERROR: Module cannot be loaded due to a conflict. |
| 618 |
HINT: Might try "module unload trace/all_on" first. |
| 619 |
test 0 = 1; |
| 620 |
|
| 621 |
The last line (``test 0 = 1;``) is the ``sh``-specific encoding of a failing |
| 622 |
exit status -- exactly what ``ERR`` expands to via ``shell_err`` (see |
| 623 |
`Base test procedures`_) -- and everything above it on stderr is what |
| 624 |
``$tserr`` was built to match. |
| 625 |
|
| 626 |
Example 2: a successful load, from ``$modpath.4`` |
| 627 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 628 |
|
| 629 |
This second example contrasts with the first one: a plain, successful load, |
| 630 |
with the environment changes it produces checked on stdout and nothing |
| 631 |
expected on stderr. It also illustrates pointing the modulepath at |
| 632 |
:file:`testsuite/modulefiles.4` (``$modpath.4`` in Tcl, see `Adding new test |
| 633 |
fixtures`_) instead of using the default fixture directory, as a new test |
| 634 |
case exercising a fixture added there would, and running the check against a |
| 635 |
single named shell (``bash``) rather than ``ALL`` -- appropriate when a test |
| 636 |
is not about shell-syntax differences and does not need to be repeated for |
| 637 |
every supported shell. |
| 638 |
|
| 639 |
The fixture used is :file:`testsuite/modulefiles.4/setenv/1.0`, whose |
| 640 |
content is simply:: |
| 641 |
|
| 642 |
#%Module |
| 643 |
setenv TS1 {} |
| 644 |
setenv TS2 $env(TS1) |
| 645 |
|
| 646 |
.. code-block:: tcl |
| 647 |
|
| 648 |
# environment setup |
| 649 |
# |
| 650 |
# point the modulepath at modulefiles.4 instead of the default fixture |
| 651 |
# directory |
| 652 |
setenv_path_var MODULEPATH $modpath.4 |
| 653 |
|
| 654 |
# build the expected stdout ($ans) |
| 655 |
# |
| 656 |
# 'setenv/1.0' sets TS1 to an empty string then TS2 to the (now empty) |
| 657 |
# value of TS1; 'load' also always sets _LMFILES_ and LOADEDMODULES -- |
| 658 |
# each element uses the 'set' tag from the shell_out vocabulary (see |
| 659 |
# `Base test procedures`_) so the answer is rendered in bash syntax |
| 660 |
set ans [list] |
| 661 |
lappend ans [list set TS1 {}] |
| 662 |
lappend ans [list set TS2 {}] |
| 663 |
lappend ans [list set _LMFILES_ $modpath.4/setenv/1.0] |
| 664 |
lappend ans [list set LOADEDMODULES setenv/1.0] |
| 665 |
|
| 666 |
# test launch |
| 667 |
# |
| 668 |
# run against bash only; a plain successful load prints nothing on |
| 669 |
# stderr, hence the empty string passed as the expected anserr |
| 670 |
testouterr_cmd bash "load setenv/1.0" $ans {} |
| 671 |
|
| 672 |
Running the testsuite |
| 673 |
--------------------- |
| 674 |
|
| 675 |
With ``make`` |
| 676 |
~~~~~~~~~~~~~ |
| 677 |
|
| 678 |
.. code-block:: sh |
| 679 |
|
| 680 |
make test # full 'modules' suite via DejaGnu (~25,000 cases, ~12 min) |
| 681 |
make test QUICKTEST=y # quick mode, most essential tests (~1,900 cases, ~1 min) |
| 682 |
make test COVERAGE=y # coverage-instrumented run, produces tcl/*.tcl_m |
| 683 |
make testinstall # 'install' suite against a tree already processed by 'make install' |
| 684 |
make testlint # 'lint' suite (Nagelfar + ShellCheck) |
| 685 |
make testcompletion # 'completion' suite (interactive Tab-completion tests) |
| 686 |
make testcookbook # 'cookbook' suite (doc/source/cookbook recipes) |
| 687 |
|
| 688 |
Each target ends up calling ``runtest --tool <tool> $(RUNTESTFLAGS) |
| 689 |
$(RUNTESTFILES)``: ``RUNTESTFILES``, if set, restricts the run to specific |
| 690 |
``.exp`` file *names* (not paths -- DejaGnu matches by basename across every |
| 691 |
directory of the active tool); ``RUNTESTFLAGS`` controls DejaGnu verbosity |
| 692 |
(``-v`` can be repeated; at ``-v -v`` the test procedures in |
| 693 |
:file:`base-config.exp` additionally dump the raw captured |
| 694 |
``OUT[len]: '...'#>``/``ERR[len]: '...'#>``/``EXIT: '...'#>`` for every |
| 695 |
check, matched vs expected). |
| 696 |
|
| 697 |
With :file:`script/mt` |
| 698 |
~~~~~~~~~~~~~~~~~~~~~~ |
| 699 |
|
| 700 |
:file:`script/mt` is a thin wrapper around the ``make`` targets above that |
| 701 |
adds live progress reporting and a readable diff on failure. Prefer it over |
| 702 |
calling ``make test`` directly when iterating on a specific area. |
| 703 |
|
| 704 |
.. code-block:: sh |
| 705 |
|
| 706 |
script/mt # same as: make test |
| 707 |
script/mt quick # same as: make test QUICKTEST=y |
| 708 |
script/mt cov # same as: make test COVERAGE=y |
| 709 |
script/mt install # same as: make testinstall |
| 710 |
script/mt lint # same as: make testlint |
| 711 |
script/mt comp # same as: make testcompletion |
| 712 |
script/mt cook # same as: make testcookbook |
| 713 |
|
| 714 |
script/mt 50/470 # only testsuite/modules.50-cmds/470-*.exp |
| 715 |
script/mt 50 # every file in testsuite/modules.50-cmds |
| 716 |
script/mt 61 # collection series (always run whole, see below) |
| 717 |
script/mt lint 00/030 # only testsuite/lint.00-init/030-*.exp |
| 718 |
script/mt comp 00/021 # only testsuite/completion.00-init/021-*.exp |
| 719 |
script/mt cook 00/020 # only testsuite/cookbook.00-init/020-*.exp |
| 720 |
script/mt 50/{280,290} 61 # several selections at once |
| 721 |
script/mt --help # full usage |
| 722 |
|
| 723 |
Whichever files are selected, :file:`script/mt` always also runs the |
| 724 |
mandatory setup files for that tool (for ``modules``: |
| 725 |
``00/005 00/006 00/010 00/050 00/060 00/080 00/085``; for ``install``: |
| 726 |
``00/005 00/006 00/010 00/011``; for ``lint``: ``00/005 00/006 00/011``; for |
| 727 |
``completion``: ``00/005 00/006 00/007 00/008 00/010 00/011 00/020 00/030 |
| 728 |
00/040 00/050``; for ``cookbook``: ``00/005 00/006 00/010 00/011``), plus |
| 729 |
the ``999-cleanup.exp`` of every selected series. |
| 730 |
Passing a bare series number always expands to every file in that |
| 731 |
directory, because several of those series are order-sensitive or |
| 732 |
enumerate a whole modulepath (see |
| 733 |
`Adding new test fixtures`_). The collection series (``61``) is one such |
| 734 |
case: its files create real collection files on disk that later files in |
| 735 |
the same series depend on -- e.g. :file:`modules.61-coll/040-restore.exp` |
| 736 |
restores collections that :file:`modules.61-coll/030-save.exp` produced -- |
| 737 |
so running only a subset of that series would fail for reasons unrelated to |
| 738 |
the actual test. |
| 739 |
|
| 740 |
While running, :file:`script/mt` tails the produced log and prints a running |
| 741 |
``pass``/``fail``/``xfail``/``error`` tally, plus a per-testcase breakdown |
| 742 |
whenever a case other than all-pass completes. When done, it invokes |
| 743 |
:file:`script/mtreview` on the log, which pulls out every |
| 744 |
``OUT[..]``/``ERR[..]`` vs ``EXP[..]`` pair recorded by the ``-v -v``-level |
| 745 |
test procedures and renders them as a diff (using ``icdiff`` if available -- |
| 746 |
:file:`script/mt` auto-downloads it once and caches it as ``./icdiff`` -- |
| 747 |
falling back to ``diff -u``). |
| 748 |
|
| 749 |
Coverage |
| 750 |
~~~~~~~~ |
| 751 |
|
| 752 |
.. code-block:: sh |
| 753 |
|
| 754 |
make test COVERAGE=y # instruments modulecmd.tcl with Nagelfar, |
| 755 |
# then runs the full suite and produces |
| 756 |
# tcl/*.tcl_m |
| 757 |
script/mt cov 70/{280,290} # coverage run limited to specific test |
| 758 |
# files |
| 759 |
|
| 760 |
Inspect the resulting :file:`tcl/*.tcl_m` files for lines flagged |
| 761 |
``;# Not covered`` to see what a newly added feature/branch still needs test |
| 762 |
coverage for. |
| 763 |
|
| 764 |
Adding new tests |
| 765 |
---------------- |
| 766 |
|
| 767 |
Adding a new test case to an existing area |
| 768 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 769 |
|
| 770 |
Add a ``test*_cmd*`` call (or a whole new numbered ``.exp`` file, following |
| 771 |
`Test file anatomy`_) to the relevant ``<tool>.<serienum>-<topic>`` directory. |
| 772 |
Pick the file/series whose topic matches, or create a new file with the next |
| 773 |
free number in that series if the scenario doesn't fit an existing file. Run |
| 774 |
the file in isolation first (``script/mt <serienum>/<num>``) before running the |
| 775 |
full suite. |
| 776 |
|
| 777 |
Adding new test fixtures |
| 778 |
~~~~~~~~~~~~~~~~~~~~~~~~ |
| 779 |
|
| 780 |
Add new fixture modulefiles under the *highest-numbered* |
| 781 |
:file:`testsuite/modulefiles.N` directory (check whether a higher one than |
| 782 |
:file:`modulefiles.4` already exists before assuming it's the latest), not |
| 783 |
under the default :file:`modulefiles/` directory. Several existing tests |
| 784 |
(``aliases``, ``avail``, ``spider``, ``scan_eval``, and others) enumerate a |
| 785 |
modulepath's *entire* content and assert on the full listing; adding fixtures |
| 786 |
to a directory shared that broadly changes their expected output. Even a |
| 787 |
numbered directory like :file:`modulefiles.4` is referenced wholesale by |
| 788 |
some tests and by filesystem-glob-order-dependent debug-trace assertions |
| 789 |
elsewhere in the suite, so: |
| 790 |
|
| 791 |
1. Add the new modulefile(s) under the latest :file:`modulefiles.N`. |
| 792 |
2. Run the *full* testsuite (or at least the ``90-avail``/``92-spider`` |
| 793 |
series plus anything else touching that modulepath), not just the new |
| 794 |
test file, to catch any global-enumeration test whose expected output |
| 795 |
now needs updating. |
| 796 |
3. Update the expected output of any such test the new fixture broke. |
| 797 |
|
| 798 |
If a fixture needs a filename ending in a space, it cannot be checked into |
| 799 |
git as a real file (checkout fails on Windows); instead generate it on the |
| 800 |
fly from Tcl, following the pattern in |
| 801 |
``create_endspace_test_modulefiles``/``delete_endspace_test_modulefiles`` in |
| 802 |
:file:`config/base-config.exp`. |
| 803 |
|
| 804 |
Adding a new sub-command or config option |
| 805 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 806 |
|
| 807 |
Don't improvise which testsuite files need touching for these two specific |
| 808 |
tasks -- :doc:`add-new-sub-command` and :doc:`add-new-config-option` each |
| 809 |
enumerate the exact list, in the "Testsuite" section of each document. |
| 810 |
|
| 811 |
Debugging a broken test |
| 812 |
----------------------- |
| 813 |
|
| 814 |
1. **Reproduce narrowly.** Re-run just the failing file with |
| 815 |
:file:`script/mt`, e.g. ``script/mt 50/470``. This also runs the tool's |
| 816 |
mandatory setup files, so the environment matches a full run. |
| 817 |
2. **Read the diff.** :file:`script/mt` runs :file:`script/mtreview` on the |
| 818 |
resulting log automatically and prints an ``EXP``-vs-``OUT``/``ERR`` diff |
| 819 |
for every failing case -- start there instead of reading the raw log. |
| 820 |
3. **Raise verbosity.** For more detail than the diff gives (e.g. to see |
| 821 |
*which* command line in a file produced a given failure, or to see |
| 822 |
``send_user`` progress messages emitted by ``config/base-config.exp`` |
| 823 |
helpers like ``setenv_var``/``change_file_perms``), set |
| 824 |
``RUNTESTFLAGS='-v -v'`` and call ``make test``/``testinstall``/ |
| 825 |
``testlint``/``testcompletion``/``testcookbook`` directly, or invoke |
| 826 |
``runtest`` yourself with the environment variables :file:`script/mt`/the |
| 827 |
Makefile targets set up (``TCLSH``, ``MODULECMD``, ``OBJDIR``, |
| 828 |
``TESTSUITEDIR``) -- see the |
| 829 |
``test``/``testinstall``/``testlint``/``testcompletion``/``testcookbook`` |
| 830 |
targets in :file:`Makefile` for the exact invocation. |
| 831 |
4. **Check for order dependence.** If a test passes alone but fails in a |
| 832 |
full run (or vice-versa), suspect a missing/incomplete ``reset_test_env`` |
| 833 |
footer in an earlier file, or a global-enumeration test (`Adding new test |
| 834 |
fixtures`_) whose expected output a fixture change invalidated. |
| 835 |
5. **Suspect the module cache.** If ``TESTSUITE_ENABLE_MODULECACHE`` is set |
| 836 |
in the environment and a test's behavior seems to depend on whether the |
| 837 |
modulepath is cached, check whether the file calls |
| 838 |
``ignore_modulecache_if_built``/``end_ignore_modulecache_if_built`` |
| 839 |
around the section that shouldn't read a pre-built cache. |
| 840 |
6. **Nagelfar/ShellCheck failures (``lint`` tool)** report the offending |
| 841 |
file and line directly in the diff; fix the flagged code or, if it's a |
| 842 |
deliberate/false-positive pattern, look at how neighboring |
| 843 |
:file:`lint.00-init/0NN-*.exp` files configure linter exclusions (e.g. |
| 844 |
the ``-e SC1090`` ShellCheck exclusion in |
| 845 |
:file:`lint.00-init/020-sh.exp`) before adding a new one. |
| 846 |
7. **``completion`` tool hangs/timeouts.** A run that hangs (rather than |
| 847 |
fails) almost always means an ``expect`` pattern in a |
| 848 |
``completion_<shell>_*`` proc never matched, so it burned the default |
| 849 |
timeout before falling through -- three recurring causes when scripting |
| 850 |
readline-based completion: sending only a double-Tab against a prefix |
| 851 |
that still has an unconsumed common-prefix extension (readline |
| 852 |
auto-inserts it on the first Tab, so the listing needs a *third* Tab -- |
| 853 |
type the full common prefix yourself instead, see the ``module load ba`` |
| 854 |
case in |
| 855 |
:file:`completion.00-init/021-bash.exp`); clearing an input line and then |
| 856 |
matching on the prompt text reappearing (readline redraws a cleared line |
| 857 |
with cursor-movement escapes, not by reprinting the prompt -- submit the |
| 858 |
now-empty line instead, see ``completion_bash_list`` in |
| 859 |
:file:`completion.00-init/020-bash-procs.exp`); and an interactive pager |
| 860 |
(``less``) kicking in on an unexpected warning and blocking for a |
| 861 |
keypress that never comes (set ``MODULES_PAGER=cat``, see |
| 862 |
:file:`completion.00-init/010-environ.exp`). |
| 863 |
8. **Coverage regressions.** If a change is meant to add coverage for a new |
| 864 |
branch, confirm it with ``make test COVERAGE=y`` (or |
| 865 |
``script/mt cov <serienum>/<num>``) and check the relevant :file:`tcl/*.tcl_m` |
| 866 |
file no longer flags that line ``;# Not covered``. |
| 867 |
|
| 868 |
.. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: |