Debian Codegraph @

doc/source/devel/testsuite.rst

raw
1 .. _devel-testsuite:
2
3 Testsuite structure
4 ===================
5
6 This document describes the non-regression testsuite of Modules: how it is
7 organized, how a test file and a test case are structured, how to run tests,
8 how to add new ones, and how to debug a failing test.
9
10 The testsuite is built on `DejaGnu <https://www.gnu.org/software/dejagnu/>`_,
11 a Tcl/Expect-based framework historically used to test compilers and command
12 line tools. DejaGnu groups test files (``.exp``, for *expect script*) under a
13 *tool*: a run of ``runtest --tool <name>`` sources every matching ``.exp``
14 file it finds for that tool and reports a ``PASS``/``FAIL``/``XFAIL``/
15 ``UNRESOLVED`` line for each individual check performed. Modules defines
16 five tools, matched to five kinds of tests (see `Kinds of tests`_ below).
17
18 Kinds of tests
19 --------------
20
21 The testsuite exercises five different things, run as five separate
22 DejaGnu tools:
23
24 ``modules``
25 The main testsuite. Runs the built :file:`modulecmd.tcl` script (the
26 Tcl-based engine behind the :command:`module` command) through every
27 supported shell and checks the environment changes and messages it
28 produces. This is by far the largest part of the testsuite (around
29 25,000 test cases spread over the ``modules.*`` directories described
30 below).
31
32 ``install``
33 Checks a real ``make install``-ed tree: that the :command:`module`
34 function/command is correctly defined once shell init scripts are
35 sourced, that the ``complete`` modulefile command emits the right
36 shell-specific completion registration snippet, that ``modulecmd`` is
37 invoked correctly from each shell's wrapper, etc. Driven by the
38 :file:`install.00-init` directory.
39
40 ``lint``
41 Runs static analysis (Nagelfar for Tcl files, ShellCheck for sh/bash/ksh
42 scripts) over the repository's own scripts and reports any warning as a
43 test failure. Driven by the :file:`lint.00-init` directory.
44
45 ``completion``
46 Drives a real, interactive shell process (via Expect ``spawn``/``send``/
47 ``expect``, not just a captured non-interactive run like the other three
48 tools) to press Tab against the built shell completion script and check
49 the resulting candidate list -- both that expected module names/option
50 flags show up, and that a candidate word built from untrusted text (a
51 module name, ``LOADEDMODULES``, ``MODULEPATH``, ...) can never reach a
52 shell expansion step. Driven by the :file:`completion.00-init` directory;
53 currently covers bash, zsh, fish and tcsh, see `completion.00-init
54 layout`_.
55
56 ``cookbook``
57 Checks the recipes documented under :ref:`cookbook`: for each recipe
58 covered, builds a sandboxed fixture from the exact files it ships under
59 :file:`doc/example/` and drives it through the commands demonstrated in
60 the recipe's ``Usage example`` documentation, parsed straight out of
61 the recipe's ``.rst`` file rather than hand-copied (see
62 `cookbook.00-init layout`_). Driven by the :file:`cookbook.00-init`
63 directory.
64
65 Each tool corresponds to one Makefile target (``test``, ``testinstall``,
66 ``testlint``, ``testcompletion``, ``testcookbook``, see `Running the
67 testsuite`_) and to one log file (:file:`modules.log`, :file:`install.log`,
68 :file:`lint.log`, :file:`completion.log`, :file:`cookbook.log`) produced in
69 the top build directory.
70
71 Two additional run modes apply to the ``modules`` tool rather than adding a
72 new one:
73
74 - *Quick mode* (``QUICKTEST=y``) skips the slower/more exhaustive checks
75 guarded by a call to ``skip_if_quick_mode`` (see `Base test procedures`_)
76 to get a fast (~1,900 cases, ~1 min) smoke test.
77 - *Coverage mode* (``COVERAGE=y``) runs the exact same tests but through a
78 Nagelfar-instrumented ``modulecmd.tcl``, then produces marked-up
79 :file:`tcl/*.tcl_m` files where lines never hit during the run are flagged
80 with ``;# Not covered``.
81
82 Directory layout
83 ----------------
84
85 Everything lives under :file:`testsuite/`.
86
87 Test file directories
88 ~~~~~~~~~~~~~~~~~~~~~
89
90 Test files are grouped in numbered directories named
91 ``<tool>.<serienum>-<topic>``, e.g. :file:`modules.50-cmds`,
92 :file:`install.00-init`, :file:`lint.00-init`, :file:`completion.00-init`,
93 :file:`cookbook.00-init`. The ``<tool>`` prefix ties the directory to one of
94 the five DejaGnu tools above (DejaGnu only looks at directories whose
95 prefix matches the ``--tool`` given to ``runtest``); the two-digit
96 ``<serienum>`` number fixes run order
97 and is what you pass to :file:`script/mt` to select a whole directory (e.g.
98 ``script/mt 50``); the ``<topic>`` suffix is just a human-readable label.
99
100 Current ``modules.*`` series, in run order:
101
102 ``00-init``
103 Testsuite bootstrap, plus a handful of standalone option/behavior
104 checks. Two files in this series matter beyond their own file number
105 because every other series relies on what they set up:
106
107 - ``005-init_ts.exp`` defines the global variables and procedures
108 shared by every test file for the whole run (paths, shell lists,
109 common error/message strings, helper procs such as
110 ``cmpversion``). It runs first, before anything else.
111 - ``010-environ.exp`` defines the *initial user environment* every
112 test starts from: it clears/resets ``MODULEPATH``,
113 ``LOADEDMODULES``, and any quarantine/auto-handling/color/...
114 configuration coming from the calling shell. The same file (by
115 role, not by sharing code) also exists as
116 :file:`install.00-init/010-environ.exp` and plays the identical
117 part for the ``install`` tool.
118
119 The rest of the series defines the ``_test_sub`` procedure that runs
120 ``modulecmd.tcl`` (``006-procs.exp``, tool-specific -- see `Running
121 the command and checking the result`_), builds further shared
122 fixtures (module search path, module cache pre-build,
123 ``save_test_env`` checkpoint), and tests standalone command-line
124 switches and configuration behavior (pager, quarantine, siteconfig,
125 auto handling, color, access rights, multilib, cwd, logger, Tcl
126 extension library).
127 ``10-use``
128 ``module use``/``unuse``
129 ``20-locate``
130 Module lookup/resolution (exact version, default, wildcard,
131 symbolic version, ...)
132 ``30-cache``
133 Module cache file handling
134 ``50-cmds``
135 Modulefile commands (``setenv``, ``prepend-path``, ``conflict``,
136 ``variant``, ...) -- by far the biggest series
137 ``51-scan``
138 Modulefile scan / extra-match search
139 ``60-initx``
140 The ``init*`` sub-commands (``initadd``, ``initprepend``, ``initrm``,
141 ``initswitch``, ``initlist``, ``initclear``) that edit a user's shell
142 startup file
143 ``61-coll``
144 Module collections
145 ``70-maint``
146 Maintenance sub-commands (``load``, ``switch``, ``purge``, config,
147 ...)
148 ``80-deep``
149 "Deep" modules, i.e. modules whose name has more than one directory
150 level (``name/sub/version``): load/unload/switch, listing, ``whatis``,
151 alias/symbolic-version resolution, access rights
152 ``90-avail``
153 ``avail`` sub-command
154 ``91-sort``
155 Module sorting order
156 ``92-spider``
157 ``spider`` sub-command
158 ``95-version``
159 Version comparison/parsing
160 ``99-finish``
161 Testsuite teardown (removes cache files created for the run)
162
163 ``install.00-init``, ``lint.00-init``, ``completion.00-init`` and
164 ``cookbook.00-init`` are each a single series (those tools are much smaller
165 and don't need topic splitting).
166
167 Every series directory ends with a ``999-cleanup.exp`` file (see `Test file
168 anatomy`_) and, for the ``modules`` tool, most series begin with a
169 ``0NN-init_ts.exp`` file that sets up whatever fixtures that series' tests
170 need (e.g. :file:`modules.90-avail/010-init_ts.exp`).
171
172 ``completion.00-init`` layout
173 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
174
175 The ``completion`` tool follows the common ``005``/``006``/``010``/``011``
176 setup numbering (see `Test file anatomy`_ and `Base test procedures`_ below)
177 plus one extra layer, since driving real Tab-key completion is inherently
178 shell-specific in a way none of the other three tools are:
179
180 - ``005-init_ts.exp`` / ``006-procs.exp`` / ``010-environ.exp`` /
181 ``011-save_test_env.exp`` set up paths (the ``bash``, ``zsh``, ``fish`` and
182 ``tcsh`` binaries, the built :file:`init/bash_completion`,
183 :file:`init/zsh-functions/_module`, :file:`init/fish_completion` and
184 :file:`init/tcsh_completion` scripts, a clean fixture modulepath), the
185 shell-agnostic assert procedures (``completion_assert_contains``,
186 ``completion_assert_not_contains``, ``completion_assert_eq``,
187 ``completion_assert_no_exec``), and the clean baseline
188 environment/``save_test_env`` checkpoint, exactly as for the other tools.
189 - ``0NN-<shell>-procs.exp`` defines one ``completion_<shell>_start`` /
190 ``completion_<shell>_raw`` / ``completion_<shell>_list`` /
191 ``completion_<shell>_inline`` / ``completion_<shell>_close`` set per
192 shell -- e.g. ``020-bash-procs.exp`` spawns a real ``bash`` pty, sources
193 :file:`init/bash_completion`, and drives double-Tab listings
194 (``completion_bash_list``, for an ambiguous prefix) or single-Tab inline
195 completions (``completion_bash_inline``, for a prefix with exactly one
196 match -- e.g. checking a directory-style entry completes with a
197 trailing ``/`` and no trailing space) through Expect.
198 ``completion_<shell>_list``/``_inline`` are responsible for recording the
199 cmdline they were passed into the shared ``completion_last_cmdline``
200 variable, which the generic assert procedures use to build their test
201 label.
202 - ``0NN-<shell>.exp`` (e.g. ``021-bash.exp``, ``031-zsh.exp``,
203 ``041-fish.exp``, ``051-tcsh.exp``) holds the actual test cases for that
204 shell, calling only its own ``completion_<shell>_*`` procs plus the shared
205 asserts. A shell whose completion script covers less ground than the
206 others (e.g. fish's has no ``ml`` support and does not gate its option
207 flags by sub-command; the tcsh ``ml`` has no global switches at all, and
208 its ``unuse`` only lists modulepaths from a bare, empty word) documents
209 each such gap in its own file's header instead of forcing every file to
210 test
211 the same thing.
212
213 Adding a new shell means adding its own ``completion_<shell>_*`` procs file
214 and test file; nothing in ``006-procs.exp`` needs to change.
215
216 ``cookbook.00-init`` layout
217 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
218
219 The ``cookbook`` tool follows the common ``005``/``006``/``010``/``011``
220 setup numbering (see `Test file anatomy`_ and `Base test procedures`_
221 below), driving ``modulecmd.tcl`` directly through ``testouterr_cmd``/
222 ``testouterr_cmd_re`` exactly like the ``modules`` tool, plus one shared
223 helper that is specific to this tool:
224
225 - ``005-init_ts.exp`` defines ``cookbookdocdir``/``cookbookexampledir``
226 (pointing at :file:`doc/source/cookbook` and :file:`doc/example`), the
227 ``cookbooksandbox`` fixture root (recreated fresh for the whole run),
228 ``cookbook_read_example_file``/``cookbook_write_file`` (read a file
229 shipped under :file:`doc/example/<name>/`, applying a ``string map``
230 substitution list, and write it into the sandbox), and
231 ``cookbook_parse_transcript`` -- parses the ``.. parsed-literal::`` blocks
232 of a named section (e.g. ``Usage example``) of a recipe's ``.rst`` file
233 and returns the ordered list of ``{command output}`` pairs shown there,
234 role markup (``:sgrhi:`text``` and friends) stripped from the output, so
235 a recipe test file drives the exact command sequence its own
236 documentation demonstrates, in that order, and checks its own real output
237 is similar to the one illustrated -- instead of both being a hand-copied
238 duplicate that could silently drift from the documentation.
239 ``cookbook_output_re`` turns one such illustrated output into the regexp
240 a sandboxed run's real output is checked against: escaped as a literal,
241 except a run of ``-`` padding (widened to ``[-]+``, since its exact width
242 depends on terminal-width detection a non-tty test run does not have) and
243 whatever literal substrings the recipe test passes it to widen too (e.g.
244 a placeholder path mapped to the sandbox's real one).
245 - ``006-procs.exp``/``010-environ.exp``/``011-save_test_env.exp`` set up
246 ``_test_sub``, a clean baseline environment (sandboxed ``$HOME``, no
247 color, deterministic ``avail``/``list`` output, ``MODULES_SITECONFIG``/
248 ``MODULES_TAG_ABBREV``/``MODULES_NON_EXPORTABLE_TAGS`` left unset so each
249 recipe states what it actually relies on), and the ``save_test_env``
250 checkpoint, exactly as for the other tools.
251 - ``0NN-<recipe-name>.exp`` (e.g. ``020-sync-remote-appdir.exp``), named
252 after the recipe's own ``.rst`` file, holds one recipe's test: it checks
253 any external binary the recipe needs (``unsupported`` + ``return`` if
254 missing, e.g. ``rsync`` for ``sync-remote-appdir``), builds a
255 ``$cookbooksandbox/<name>`` fixture from the recipe's real
256 :file:`doc/example/<name>/` files via ``cookbook_read_example_file`` (only
257 substituting whatever real-system absolute path the recipe hardcodes,
258 e.g. ``/remote_apps``, for a sandbox path -- everything else is used byte
259 for byte), points the test environment at that sandbox, then loops over
260 ``cookbook_parse_transcript``'s steps in order, running each with
261 ``testouterr_cmd_re`` against ``cookbook_output_re`` of its illustrated
262 output (stdout, the env-var-assignment syntax a shell evals silently and
263 no recipe doc illustrates, is still checked against a hand-built answer
264 per step -- ``unresolved`` + ``return`` first if the transcript's step
265 count no longer matches what the test knows how to build one for, as a
266 signal it needs updating alongside the doc). Simulates each step's
267 effect on the environment for the next one exactly as a real shell would
268 (``setenv_loaded_module``/``setenv_path_var``), since every
269 ``modulecmd.tcl`` invocation here is an independent process.
270
271 Adding a new recipe means adding its own ``0NN-<recipe-name>.exp`` file;
272 nothing in ``005-init_ts.exp`` needs to change unless the new recipe needs a
273 kind of substitution or transcript shape the existing helpers do not cover
274 yet.
275
276 Fixture and support directories
277 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
278
279 - :file:`modulefiles/`, :file:`modulefiles.2/`, :file:`modulefiles.3/`,
280 :file:`modulefiles.4/`, and a number of single-purpose
281 :file:`modulefiles.<name>/` directories (``.deep``, ``.deps``, ``.eb``,
282 ``.spider1``/``.spider2``/``.spider3``, ``.rc``, ``.path1``/``.path2``/
283 ``.path3``, ``.memcache``, ``.indepth``, ``.allin``) hold the modulefiles
284 used as fixtures. See `Adding new test fixtures`_ below for which one to
285 add to.
286 - :file:`config/` holds the DejaGnu/Expect configuration shared by every
287 ``.exp`` file: :file:`base-config.exp` (all the test procedures, see
288 `Base test procedures`_) and :file:`unix.exp` (the low-level plumbing
289 that spawns ``modulecmd.tcl`` and captures its stdout/stderr/exit code).
290 - :file:`etc/`, :file:`home/`, :file:`home.2/`-:file:`home.4/` provide
291 sandboxed ``$MODULESHOME``-like/``$HOME``-like trees (rc files,
292 collections, ...) used as fixtures.
293 - :file:`bin/` contains small stand-in executables used by tests (fake
294 tools invoked by modulefiles or by shell code under test).
295 - Miscellaneous single-purpose helper scripts at the top level:
296 :file:`is_func_defined` / :file:`is_func_defined.fish` (check whether
297 ``module``/``mogui`` is defined as a shell function before running the
298 ``install`` tests), :file:`stdin_to_file`, :file:`stty`, :file:`mode`,
299 :file:`not_installed`, :file:`systest`/:file:`systest0-2`, :file:`id`,
300 :file:`cmd.exe`, :file:`manpath`, :file:`virttargets/`.
301 - :file:`mb/` holds fixtures for the :file:`script/mb` benchmark tool, not
302 for DejaGnu tests.
303
304 Test file anatomy
305 -----------------
306
307 Every ``.exp`` file follows the same three-part shape: a header comment
308 block, the test content, and (for most files, though not the ``00-init``
309 setup files) a cleanup footer.
310
311 Header
312 ~~~~~~
313
314 A fixed-format comment block, kept for historical (SCCS/CVS-style keyword
315 expansion) reasons and consistency rather than for tooling:
316
317 .. code-block:: tcl
318
319 ##############################################################################
320 # Modules Revision 3.0
321 # Providing a flexible user environment
322 #
323 # File: modules.50-cmds/%M%
324 # Revision: %I%
325 # First Edition: 2021/03/04
326 # Last Mod.: %U%, %G%
327 #
328 # Authors: Your Name, [email protected]
329 #
330 # Description: Testuite testsequence
331 # Command: load, display, help, test
332 # Modulefiles: variant
333 # Sub-Command:
334 #
335 # Comment: %C{
336 # Test 'variant' modulefile command
337 # }C%
338 #
339 ##############################################################################
340
341 When adding a file, copy this header from a neighboring file in the same
342 directory, update ``First Edition`` to the current date, your name/email in
343 ``Authors`` if you want authorship credit, and fill in ``Command:``/
344 ``Modulefiles:``/``Sub-Command:`` and the free-text ``Comment:`` block to
345 describe what the file tests. The ``%M%``/``%I%``/``%U%``/``%G%`` tokens are
346 left as-is (they are not expanded by anything in this codebase but are kept
347 for consistency with the historical format).
348
349 Content
350 ~~~~~~~
351
352 The body of the file is plain Tcl/Expect code, structured as:
353
354 1. Optional per-file setup: pick/point to a modulepath (e.g.
355 ``set mp $modpath.3``), set environment variables relevant to the
356 scenario (``setenv_var``/``setenv_path_var``), maybe call
357 ``skip_if_quick_mode`` or ``skip_if_os_in`` right after the setup that
358 only the skipped tests need.
359 2. One or more test cases: each is one call to a ``test*_cmd*`` procedure
360 from :file:`config/base-config.exp` (see `Base test procedures`_),
361 grouped under a short ``#\n# <description>\n#`` comment banner when the
362 file covers more than one scenario.
363 3. A ``#\n# Cleanup\n#`` banner followed by a call to ``reset_test_env``
364 (see below), for any file that changed environment variables or Tcl
365 globals the following files shouldn't see.
366
367 Footer
368 ~~~~~~
369
370 Files that mutate shared state end with:
371
372 .. code-block:: tcl
373
374 #
375 # Cleanup
376 #
377
378 reset_test_env
379
380 ``reset_test_env`` restores every Tcl global variable and environment
381 variable to the value it had when ``save_test_env`` last ran (this checkpoint
382 is taken once per tool, early in the ``00-init`` series, right after the
383 common fixtures -- modulepath, cache, etc. -- are set up; see
384 :file:`modules.00-init/085-save_test_env.exp`). This is what makes test files
385 independent of run order within reasonable bounds: a file can freely
386 ``setenv_var``, build a collection, flip a config option, etc., and the next
387 file starts clean.
388
389 A handful of files close with a Vim modeline instead (or in addition):
390
391 .. code-block:: tcl
392
393 # vim:set tabstop=3 shiftwidth=3 expandtab autoindent:
394
395 How a test case is built
396 ------------------------
397
398 A test case has two ingredients: the environment/fixtures it runs against
399 (set up with plain Tcl and the ``setenv_*``/``unsetenv_*`` helpers), and one
400 call to a ``test*_cmd*`` procedure that runs a :command:`module` command line
401 and checks its outcome.
402
403 Setting up environment
404 ~~~~~~~~~~~~~~~~~~~~~~
405
406 - ``setenv_var var val`` / ``unsetenv_var var`` -- set/unset a plain
407 environment variable for the *modulecmd.tcl* child process about to be
408 spawned.
409 - ``setenv_path_var var elt1 elt2 ...`` / ``unsetenv_path_var var`` -- same,
410 for a colon-delimited path-like variable; also transparently maintains the
411 matching ``__MODULES_SHARE_<var>`` reference-count variable.
412 - ``setenv_loaded_module modlist modfilelist`` /
413 ``unsetenv_loaded_module`` -- set up ``LOADEDMODULES``/``_LMFILES_`` (and
414 ``__MODULES_LMTAG`` for auto-loaded modules) to simulate modules already
415 loaded prior to the command under test.
416 - ``skip_if_quick_mode`` -- bail out of the rest of the *current file* when
417 ``QUICKTEST=y``. Used to skip expensive/exhaustive variations while
418 keeping at least one representative case running in quick mode.
419 - ``skip_if_os_in os1 os2 ...`` -- bail out of the rest of the current file
420 on the given OS names (``$::os_name``, e.g. ``windows``, ``darwin``).
421 - ``change_file_perms``/``restore_file_perms`` -- temporarily lock down a
422 file/directory's permissions to test permission-denied paths.
423
424 Running the command and checking the result
425 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
426
427 Every ``test*_cmd*`` procedure runs ``modulecmd <shell> <cmd>`` through
428 ``_test_sub`` and compares the captured stdout/stderr/exit code against
429 what was passed in, reporting ``pass``/``fail`` to DejaGnu. ``_test_sub``
430 itself is tool-specific: each tool defines its own version in its
431 ``00-init`` series' ``006-procs.exp`` file (e.g.
432 :file:`modules.00-init/006-procs.exp`), which for the ``modules`` tool
433 calls down into ``modulecmd_xxx_`` (defined in :file:`config/unix.exp`) to
434 actually spawn ``modulecmd.tcl`` and capture its output.
435
436 See `Base test procedures`_ for the full family of these procedures and how
437 their ``answer``/``anserr`` arguments are interpreted (plain string, a
438 "shell out list", or the ``OK``/``ERR``/``ERR2`` shorthands).
439
440 Base test procedures
441 --------------------
442
443 All of the below live in :file:`testsuite/config/base-config.exp`
444 (``shell_*`` internal helpers, used by the procedures below to translate a
445 symbolic environment change into the actual syntax of each of the 15+
446 supported shells, are not covered here).
447
448 Environment/output description helpers
449 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
450
451 ``shell_out test_shell out_list`` builds the shell-specific expected output
452 for a *list* of symbolic environment changes -- this is what most test cases
453 pass as their ``answer``/``anserr`` argument instead of a literal string.
454 Each element of ``out_list`` is itself a small list starting with one of:
455
456 ``set``
457 ``{set var val}`` -- variable assignment (pass ``noescval`` as a
458 4th element to skip shell escaping)
459 ``setpath``
460 ``{setpath var val}`` -- like ``set``, but also auto-generates the
461 matching ``__MODULES_SHARE_<var>`` reference-count assignment
462 ``unset``
463 ``{unset var}`` -- variable unset
464 ``unsetpath``
465 ``{unsetpath var}`` -- like ``unset``, plus reference count
466 ``alias``
467 ``{alias var val}`` -- shell alias definition
468 ``unalias``
469 ``{unalias var}`` -- shell alias removal
470 ``chdir``
471 ``{chdir dir}`` -- directory change
472 ``xres``
473 ``{xres var val}`` -- X11 resource set (``xrdb``)
474 ``unxres``
475 ``{unxres var}`` -- X11 resource removal
476 ``text``
477 ``{text str}`` -- arbitrary literal text (module messages,
478 warnings, ...), inserted as-is
479 ``OK``, ``ERR``, ``ERR2``
480 shell-specific success / 1-error / 2-error status code
481 ``out``, or anything else
482 literal text joined as-is
483
484 ``is_shell_out_list answer`` tells whether ``answer`` looks like one of
485 these lists (used internally by ``_test_out``/``_test_out_re`` to decide
486 whether to run it through ``shell_out`` or treat it as a literal string).
487
488 Test procedures (the ones test files actually call)
489 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
490
491 All of them share the same first two arguments, ``test_shell`` (a shell
492 name, or ``ALL`` to run the same check against every shell in
493 ``$supported_shells``) and ``cmd`` (the :command:`module` command line,
494 without the leading ``module`` word), plus a trailing optional
495 ``failcmd`` (defaults to ``fail``; some tests in the deep-reload/args series
496 pass ``untested`` or ``unresolved`` here) and ``launcher`` (to prefix the
497 call, e.g. with ``sudo``).
498
499 ``testouterr_cmd`` is the preferred procedure for a new test case: it
500 checks both stdout and stderr, so a message unexpectedly landing on the
501 wrong stream does not go unnoticed, and it is by far the most used
502 procedure in the existing testsuite. Reach for one of the others only when
503 its extra check (exit code, regexp matching, file content, ...) or its
504 narrower scope (single-stream only) is actually needed.
505
506 ``test_cmd test_shell cmd answer {exitval 0} {failcmd fail} {launcher {}}``
507 Checks stdout (exact) and the exit code (default expected: ``0``).
508 ``test_cmd_re test_shell cmd answer {failcmd fail} {launcher {}}``
509 Checks stdout (regexp).
510 ``testerr_cmd test_shell cmd answer {failcmd fail} {launcher {}}``
511 Checks stderr (exact).
512 ``testerr_cmd_re test_shell cmd answer {force_nl 0} {failcmd fail} {launcher {}}``
513 Checks stderr (regexp). ``force_nl`` forces a trailing newline to be
514 appended to ``answer`` before it is used as a regexp, for the cases
515 where that can't be inferred automatically.
516 ``testouterr_cmd test_shell cmd answer anserr {failcmd fail} {launcher {}}``
517 Checks stdout (exact, ``answer``) and stderr (exact, ``anserr``).
518 **Preferred procedure, see above.**
519 ``testouterr_cmd_re test_shell cmd answer anserr {force_nl 0} {failcmd fail} {launcher {}}``
520 Checks stdout (regexp, ``answer``) and stderr (regexp, ``anserr``).
521 ``testouterr_cmd_re_sort test_shell cmd answer anserr {failcmd fail} {launcher {}}``
522 Checks stdout (regexp) and stderr (regexp), both sides line-sorted
523 first -- for output whose line order is not guaranteed.
524 ``testall_cmd test_shell cmd answer anserr exitval {failcmd fail} {launcher {}}``
525 Checks stdout (exact), stderr (exact) and the exit code.
526 ``testall_cmd_re test_shell cmd answer anserr exitval {force_nl 0} {failcmd fail} {launcher {}}``
527 Checks stdout (regexp), stderr (regexp) and the exit code.
528 ``testinouterr_cmd test_shell cmd input answer anserr {failcmd fail} {launcher {}}``
529 Like ``testouterr_cmd`` but also feeds ``input`` on stdin.
530 ``testoutfile_cmd test_shell cmd answer filepath ansfile {failcmd fail} {launcher {}}``
531 Checks stdout (exact, ``answer``) and the content of ``filepath`` on
532 disk (exact, ``ansfile``).
533 ``testouterrfile_cmd test_shell cmd answer anserr filepath ansfile {failcmd fail} {launcher {}}``
534 Checks stdout (exact), stderr (exact) and the content of ``filepath``
535 on disk (exact, ``ansfile``).
536 ``testouterrfileglob_cmd test_shell cmd answer anserr fileglob ansfile {failcmd fail} {launcher {}}``
537 Like ``testouterrfile_cmd``, but the file to check is resolved from
538 the ``fileglob`` glob pattern (last match once sorted).
539 ``testouterrgloblist_cmd test_shell cmd answer anserr fileglob ansglob {failcmd fail} {launcher {}}``
540 Checks stdout (exact), stderr (exact) and the *sorted list* of files
541 matching the ``fileglob`` glob pattern (against ``ansglob``).
542
543 For the ``ERR``/``ERR2``/``OK`` shorthands accepted as ``answer``/``anserr``:
544 ``OK`` means "no output expected", ``ERR``/``ERR2`` expand to the
545 shell-specific representation of exit status 1/2 (via ``shell_err``).
546
547 Environment save/restore and mode helpers
548 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
549
550 - ``save_test_env`` / ``reset_test_env`` -- see `Footer`_ above.
551 - ``is_quick_mode`` / ``skip_if_quick_mode`` -- see `Setting up
552 environment`_ above.
553 - ``skip_if_os_in`` -- see `Setting up environment`_ above.
554
555 A worked example
556 ----------------
557
558 The following two examples put together everything from `How a test case is
559 built`_ and `Base test procedures`_ into concrete, runnable test cases: one
560 where the load is rejected, one where it succeeds.
561
562 Example 1: a rejected load (conflict)
563 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
564
565 This is a trimmed-down version of one of the checks in
566 :file:`modules.50-cmds/131-conflict-module.exp`: loading a module while a
567 conflicting one is already loaded, with automatic conflict-unload disabled
568 (the project's default), must fail with an explanatory error.
569
570 The scenario relies on two existing fixture modulefiles: ``trace/all_on``
571 and ``conflict/module`` (:file:`testsuite/modulefiles/conflict/module`),
572 whose ``.modulerc``-less content simply declares ``conflict trace``.
573
574 .. code-block:: tcl
575
576 # environment setup
577 #
578 # MODULEPATH already points to $modpath (the default fixture directory,
579 # testsuite/modulefiles) -- that is set up once for the whole run by
580 # modules.00-init/050-modpath.exp, so there is nothing to do here unless
581 # a test needs a different modulepath. Simulate that 'trace/all_on' is
582 # already loaded, without actually running a 'load' command for it:
583 setenv_loaded_module trace/all_on $modpath/trace/all_on
584
585 # build the expected stdout ($ans)
586 #
587 # the conflict is detected before any environment change is made, so
588 # the load is rejected outright: stdout only carries the shell's
589 # failing exit status, i.e. the 'ERR' shorthand (see `Base test
590 # procedures`_)
591 set ans ERR
592
593 # build the expected stderr ($tserr)
594 #
595 # 'module load' always echoes a "Loading <mod>" banner first; the
596 # msg_load and err_conflict helpers (defined in
597 # modules.00-init/005-init_ts.exp) build that banner and the
598 # conflict/hint message actually printed by modulecmd.tcl
599 set tserr [msg_load conflict/module [err_conflict trace/all_on]]
600
601 # test launch
602 #
603 # run 'module load conflict/module' for every supported shell and check
604 # its stdout against $ans and its stderr against $tserr
605 testouterr_cmd ALL "load conflict/module" $ans $tserr
606
607 Running this (e.g. pasted at the end of an existing test file, or as its own
608 numbered ``.exp`` file, see `Test file anatomy`_) exercises the real
609 :command:`module load` code path for every supported shell and would report
610 one ``PASS``/``FAIL`` per shell. For reference, this is what
611 ``modulecmd.tcl`` itself prints for the ``sh`` shell in this exact scenario::
612
613 $ MODULEPATH=.../modulefiles LOADEDMODULES=trace/all_on \
614 _LMFILES_=.../modulefiles/trace/all_on \
615 modulecmd.tcl sh load conflict/module
616 Loading conflict/module
617 ERROR: Module cannot be loaded due to a conflict.
618 HINT: Might try "module unload trace/all_on" first.
619 test 0 = 1;
620
621 The last line (``test 0 = 1;``) is the ``sh``-specific encoding of a failing
622 exit status -- exactly what ``ERR`` expands to via ``shell_err`` (see
623 `Base test procedures`_) -- and everything above it on stderr is what
624 ``$tserr`` was built to match.
625
626 Example 2: a successful load, from ``$modpath.4``
627 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
628
629 This second example contrasts with the first one: a plain, successful load,
630 with the environment changes it produces checked on stdout and nothing
631 expected on stderr. It also illustrates pointing the modulepath at
632 :file:`testsuite/modulefiles.4` (``$modpath.4`` in Tcl, see `Adding new test
633 fixtures`_) instead of using the default fixture directory, as a new test
634 case exercising a fixture added there would, and running the check against a
635 single named shell (``bash``) rather than ``ALL`` -- appropriate when a test
636 is not about shell-syntax differences and does not need to be repeated for
637 every supported shell.
638
639 The fixture used is :file:`testsuite/modulefiles.4/setenv/1.0`, whose
640 content is simply::
641
642 #%Module
643 setenv TS1 {}
644 setenv TS2 $env(TS1)
645
646 .. code-block:: tcl
647
648 # environment setup
649 #
650 # point the modulepath at modulefiles.4 instead of the default fixture
651 # directory
652 setenv_path_var MODULEPATH $modpath.4
653
654 # build the expected stdout ($ans)
655 #
656 # 'setenv/1.0' sets TS1 to an empty string then TS2 to the (now empty)
657 # value of TS1; 'load' also always sets _LMFILES_ and LOADEDMODULES --
658 # each element uses the 'set' tag from the shell_out vocabulary (see
659 # `Base test procedures`_) so the answer is rendered in bash syntax
660 set ans [list]
661 lappend ans [list set TS1 {}]
662 lappend ans [list set TS2 {}]
663 lappend ans [list set _LMFILES_ $modpath.4/setenv/1.0]
664 lappend ans [list set LOADEDMODULES setenv/1.0]
665
666 # test launch
667 #
668 # run against bash only; a plain successful load prints nothing on
669 # stderr, hence the empty string passed as the expected anserr
670 testouterr_cmd bash "load setenv/1.0" $ans {}
671
672 Running the testsuite
673 ---------------------
674
675 With ``make``
676 ~~~~~~~~~~~~~
677
678 .. code-block:: sh
679
680 make test # full 'modules' suite via DejaGnu (~25,000 cases, ~12 min)
681 make test QUICKTEST=y # quick mode, most essential tests (~1,900 cases, ~1 min)
682 make test COVERAGE=y # coverage-instrumented run, produces tcl/*.tcl_m
683 make testinstall # 'install' suite against a tree already processed by 'make install'
684 make testlint # 'lint' suite (Nagelfar + ShellCheck)
685 make testcompletion # 'completion' suite (interactive Tab-completion tests)
686 make testcookbook # 'cookbook' suite (doc/source/cookbook recipes)
687
688 Each target ends up calling ``runtest --tool <tool> $(RUNTESTFLAGS)
689 $(RUNTESTFILES)``: ``RUNTESTFILES``, if set, restricts the run to specific
690 ``.exp`` file *names* (not paths -- DejaGnu matches by basename across every
691 directory of the active tool); ``RUNTESTFLAGS`` controls DejaGnu verbosity
692 (``-v`` can be repeated; at ``-v -v`` the test procedures in
693 :file:`base-config.exp` additionally dump the raw captured
694 ``OUT[len]: '...'#>``/``ERR[len]: '...'#>``/``EXIT: '...'#>`` for every
695 check, matched vs expected).
696
697 With :file:`script/mt`
698 ~~~~~~~~~~~~~~~~~~~~~~
699
700 :file:`script/mt` is a thin wrapper around the ``make`` targets above that
701 adds live progress reporting and a readable diff on failure. Prefer it over
702 calling ``make test`` directly when iterating on a specific area.
703
704 .. code-block:: sh
705
706 script/mt # same as: make test
707 script/mt quick # same as: make test QUICKTEST=y
708 script/mt cov # same as: make test COVERAGE=y
709 script/mt install # same as: make testinstall
710 script/mt lint # same as: make testlint
711 script/mt comp # same as: make testcompletion
712 script/mt cook # same as: make testcookbook
713
714 script/mt 50/470 # only testsuite/modules.50-cmds/470-*.exp
715 script/mt 50 # every file in testsuite/modules.50-cmds
716 script/mt 61 # collection series (always run whole, see below)
717 script/mt lint 00/030 # only testsuite/lint.00-init/030-*.exp
718 script/mt comp 00/021 # only testsuite/completion.00-init/021-*.exp
719 script/mt cook 00/020 # only testsuite/cookbook.00-init/020-*.exp
720 script/mt 50/{280,290} 61 # several selections at once
721 script/mt --help # full usage
722
723 Whichever files are selected, :file:`script/mt` always also runs the
724 mandatory setup files for that tool (for ``modules``:
725 ``00/005 00/006 00/010 00/050 00/060 00/080 00/085``; for ``install``:
726 ``00/005 00/006 00/010 00/011``; for ``lint``: ``00/005 00/006 00/011``; for
727 ``completion``: ``00/005 00/006 00/007 00/008 00/010 00/011 00/020 00/030
728 00/040 00/050``; for ``cookbook``: ``00/005 00/006 00/010 00/011``), plus
729 the ``999-cleanup.exp`` of every selected series.
730 Passing a bare series number always expands to every file in that
731 directory, because several of those series are order-sensitive or
732 enumerate a whole modulepath (see
733 `Adding new test fixtures`_). The collection series (``61``) is one such
734 case: its files create real collection files on disk that later files in
735 the same series depend on -- e.g. :file:`modules.61-coll/040-restore.exp`
736 restores collections that :file:`modules.61-coll/030-save.exp` produced --
737 so running only a subset of that series would fail for reasons unrelated to
738 the actual test.
739
740 While running, :file:`script/mt` tails the produced log and prints a running
741 ``pass``/``fail``/``xfail``/``error`` tally, plus a per-testcase breakdown
742 whenever a case other than all-pass completes. When done, it invokes
743 :file:`script/mtreview` on the log, which pulls out every
744 ``OUT[..]``/``ERR[..]`` vs ``EXP[..]`` pair recorded by the ``-v -v``-level
745 test procedures and renders them as a diff (using ``icdiff`` if available --
746 :file:`script/mt` auto-downloads it once and caches it as ``./icdiff`` --
747 falling back to ``diff -u``).
748
749 Coverage
750 ~~~~~~~~
751
752 .. code-block:: sh
753
754 make test COVERAGE=y # instruments modulecmd.tcl with Nagelfar,
755 # then runs the full suite and produces
756 # tcl/*.tcl_m
757 script/mt cov 70/{280,290} # coverage run limited to specific test
758 # files
759
760 Inspect the resulting :file:`tcl/*.tcl_m` files for lines flagged
761 ``;# Not covered`` to see what a newly added feature/branch still needs test
762 coverage for.
763
764 Adding new tests
765 ----------------
766
767 Adding a new test case to an existing area
768 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
769
770 Add a ``test*_cmd*`` call (or a whole new numbered ``.exp`` file, following
771 `Test file anatomy`_) to the relevant ``<tool>.<serienum>-<topic>`` directory.
772 Pick the file/series whose topic matches, or create a new file with the next
773 free number in that series if the scenario doesn't fit an existing file. Run
774 the file in isolation first (``script/mt <serienum>/<num>``) before running the
775 full suite.
776
777 Adding new test fixtures
778 ~~~~~~~~~~~~~~~~~~~~~~~~
779
780 Add new fixture modulefiles under the *highest-numbered*
781 :file:`testsuite/modulefiles.N` directory (check whether a higher one than
782 :file:`modulefiles.4` already exists before assuming it's the latest), not
783 under the default :file:`modulefiles/` directory. Several existing tests
784 (``aliases``, ``avail``, ``spider``, ``scan_eval``, and others) enumerate a
785 modulepath's *entire* content and assert on the full listing; adding fixtures
786 to a directory shared that broadly changes their expected output. Even a
787 numbered directory like :file:`modulefiles.4` is referenced wholesale by
788 some tests and by filesystem-glob-order-dependent debug-trace assertions
789 elsewhere in the suite, so:
790
791 1. Add the new modulefile(s) under the latest :file:`modulefiles.N`.
792 2. Run the *full* testsuite (or at least the ``90-avail``/``92-spider``
793 series plus anything else touching that modulepath), not just the new
794 test file, to catch any global-enumeration test whose expected output
795 now needs updating.
796 3. Update the expected output of any such test the new fixture broke.
797
798 If a fixture needs a filename ending in a space, it cannot be checked into
799 git as a real file (checkout fails on Windows); instead generate it on the
800 fly from Tcl, following the pattern in
801 ``create_endspace_test_modulefiles``/``delete_endspace_test_modulefiles`` in
802 :file:`config/base-config.exp`.
803
804 Adding a new sub-command or config option
805 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
806
807 Don't improvise which testsuite files need touching for these two specific
808 tasks -- :doc:`add-new-sub-command` and :doc:`add-new-config-option` each
809 enumerate the exact list, in the "Testsuite" section of each document.
810
811 Debugging a broken test
812 -----------------------
813
814 1. **Reproduce narrowly.** Re-run just the failing file with
815 :file:`script/mt`, e.g. ``script/mt 50/470``. This also runs the tool's
816 mandatory setup files, so the environment matches a full run.
817 2. **Read the diff.** :file:`script/mt` runs :file:`script/mtreview` on the
818 resulting log automatically and prints an ``EXP``-vs-``OUT``/``ERR`` diff
819 for every failing case -- start there instead of reading the raw log.
820 3. **Raise verbosity.** For more detail than the diff gives (e.g. to see
821 *which* command line in a file produced a given failure, or to see
822 ``send_user`` progress messages emitted by ``config/base-config.exp``
823 helpers like ``setenv_var``/``change_file_perms``), set
824 ``RUNTESTFLAGS='-v -v'`` and call ``make test``/``testinstall``/
825 ``testlint``/``testcompletion``/``testcookbook`` directly, or invoke
826 ``runtest`` yourself with the environment variables :file:`script/mt`/the
827 Makefile targets set up (``TCLSH``, ``MODULECMD``, ``OBJDIR``,
828 ``TESTSUITEDIR``) -- see the
829 ``test``/``testinstall``/``testlint``/``testcompletion``/``testcookbook``
830 targets in :file:`Makefile` for the exact invocation.
831 4. **Check for order dependence.** If a test passes alone but fails in a
832 full run (or vice-versa), suspect a missing/incomplete ``reset_test_env``
833 footer in an earlier file, or a global-enumeration test (`Adding new test
834 fixtures`_) whose expected output a fixture change invalidated.
835 5. **Suspect the module cache.** If ``TESTSUITE_ENABLE_MODULECACHE`` is set
836 in the environment and a test's behavior seems to depend on whether the
837 modulepath is cached, check whether the file calls
838 ``ignore_modulecache_if_built``/``end_ignore_modulecache_if_built``
839 around the section that shouldn't read a pre-built cache.
840 6. **Nagelfar/ShellCheck failures (``lint`` tool)** report the offending
841 file and line directly in the diff; fix the flagged code or, if it's a
842 deliberate/false-positive pattern, look at how neighboring
843 :file:`lint.00-init/0NN-*.exp` files configure linter exclusions (e.g.
844 the ``-e SC1090`` ShellCheck exclusion in
845 :file:`lint.00-init/020-sh.exp`) before adding a new one.
846 7. **``completion`` tool hangs/timeouts.** A run that hangs (rather than
847 fails) almost always means an ``expect`` pattern in a
848 ``completion_<shell>_*`` proc never matched, so it burned the default
849 timeout before falling through -- three recurring causes when scripting
850 readline-based completion: sending only a double-Tab against a prefix
851 that still has an unconsumed common-prefix extension (readline
852 auto-inserts it on the first Tab, so the listing needs a *third* Tab --
853 type the full common prefix yourself instead, see the ``module load ba``
854 case in
855 :file:`completion.00-init/021-bash.exp`); clearing an input line and then
856 matching on the prompt text reappearing (readline redraws a cleared line
857 with cursor-movement escapes, not by reprinting the prompt -- submit the
858 now-empty line instead, see ``completion_bash_list`` in
859 :file:`completion.00-init/020-bash-procs.exp`); and an interactive pager
860 (``less``) kicking in on an unexpected warning and blocking for a
861 keypress that never comes (set ``MODULES_PAGER=cat``, see
862 :file:`completion.00-init/010-environ.exp`).
863 8. **Coverage regressions.** If a change is meant to add coverage for a new
864 branch, confirm it with ``make test COVERAGE=y`` (or
865 ``script/mt cov <serienum>/<num>``) and check the relevant :file:`tcl/*.tcl_m`
866 file no longer flags that line ``;# Not covered``.
867
868 .. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: