.. _devel-testsuite: Testsuite structure =================== This document describes the non-regression testsuite of Modules: how it is organized, how a test file and a test case are structured, how to run tests, how to add new ones, and how to debug a failing test. The testsuite is built on `DejaGnu `_, a Tcl/Expect-based framework historically used to test compilers and command line tools. DejaGnu groups test files (``.exp``, for *expect script*) under a *tool*: a run of ``runtest --tool `` sources every matching ``.exp`` file it finds for that tool and reports a ``PASS``/``FAIL``/``XFAIL``/ ``UNRESOLVED`` line for each individual check performed. Modules defines five tools, matched to five kinds of tests (see `Kinds of tests`_ below). Kinds of tests -------------- The testsuite exercises five different things, run as five separate DejaGnu tools: ``modules`` The main testsuite. Runs the built :file:`modulecmd.tcl` script (the Tcl-based engine behind the :command:`module` command) through every supported shell and checks the environment changes and messages it produces. This is by far the largest part of the testsuite (around 25,000 test cases spread over the ``modules.*`` directories described below). ``install`` Checks a real ``make install``-ed tree: that the :command:`module` function/command is correctly defined once shell init scripts are sourced, that the ``complete`` modulefile command emits the right shell-specific completion registration snippet, that ``modulecmd`` is invoked correctly from each shell's wrapper, etc. Driven by the :file:`install.00-init` directory. ``lint`` Runs static analysis (Nagelfar for Tcl files, ShellCheck for sh/bash/ksh scripts) over the repository's own scripts and reports any warning as a test failure. Driven by the :file:`lint.00-init` directory. ``completion`` Drives a real, interactive shell process (via Expect ``spawn``/``send``/ ``expect``, not just a captured non-interactive run like the other three tools) to press Tab against the built shell completion script and check the resulting candidate list -- both that expected module names/option flags show up, and that a candidate word built from untrusted text (a module name, ``LOADEDMODULES``, ``MODULEPATH``, ...) can never reach a shell expansion step. Driven by the :file:`completion.00-init` directory; currently covers bash, zsh, fish and tcsh, see `completion.00-init layout`_. ``cookbook`` Checks the recipes documented under :ref:`cookbook`: for each recipe covered, builds a sandboxed fixture from the exact files it ships under :file:`doc/example/` and drives it through the commands demonstrated in the recipe's ``Usage example`` documentation, parsed straight out of the recipe's ``.rst`` file rather than hand-copied (see `cookbook.00-init layout`_). Driven by the :file:`cookbook.00-init` directory. Each tool corresponds to one Makefile target (``test``, ``testinstall``, ``testlint``, ``testcompletion``, ``testcookbook``, see `Running the testsuite`_) and to one log file (:file:`modules.log`, :file:`install.log`, :file:`lint.log`, :file:`completion.log`, :file:`cookbook.log`) produced in the top build directory. Two additional run modes apply to the ``modules`` tool rather than adding a new one: - *Quick mode* (``QUICKTEST=y``) skips the slower/more exhaustive checks guarded by a call to ``skip_if_quick_mode`` (see `Base test procedures`_) to get a fast (~1,900 cases, ~1 min) smoke test. - *Coverage mode* (``COVERAGE=y``) runs the exact same tests but through a Nagelfar-instrumented ``modulecmd.tcl``, then produces marked-up :file:`tcl/*.tcl_m` files where lines never hit during the run are flagged with ``;# Not covered``. Directory layout ---------------- Everything lives under :file:`testsuite/`. Test file directories ~~~~~~~~~~~~~~~~~~~~~ Test files are grouped in numbered directories named ``.-``, e.g. :file:`modules.50-cmds`, :file:`install.00-init`, :file:`lint.00-init`, :file:`completion.00-init`, :file:`cookbook.00-init`. The ```` prefix ties the directory to one of the five DejaGnu tools above (DejaGnu only looks at directories whose prefix matches the ``--tool`` given to ``runtest``); the two-digit ```` number fixes run order and is what you pass to :file:`script/mt` to select a whole directory (e.g. ``script/mt 50``); the ```` suffix is just a human-readable label. Current ``modules.*`` series, in run order: ``00-init`` Testsuite bootstrap, plus a handful of standalone option/behavior checks. Two files in this series matter beyond their own file number because every other series relies on what they set up: - ``005-init_ts.exp`` defines the global variables and procedures shared by every test file for the whole run (paths, shell lists, common error/message strings, helper procs such as ``cmpversion``). It runs first, before anything else. - ``010-environ.exp`` defines the *initial user environment* every test starts from: it clears/resets ``MODULEPATH``, ``LOADEDMODULES``, and any quarantine/auto-handling/color/... configuration coming from the calling shell. The same file (by role, not by sharing code) also exists as :file:`install.00-init/010-environ.exp` and plays the identical part for the ``install`` tool. The rest of the series defines the ``_test_sub`` procedure that runs ``modulecmd.tcl`` (``006-procs.exp``, tool-specific -- see `Running the command and checking the result`_), builds further shared fixtures (module search path, module cache pre-build, ``save_test_env`` checkpoint), and tests standalone command-line switches and configuration behavior (pager, quarantine, siteconfig, auto handling, color, access rights, multilib, cwd, logger, Tcl extension library). ``10-use`` ``module use``/``unuse`` ``20-locate`` Module lookup/resolution (exact version, default, wildcard, symbolic version, ...) ``30-cache`` Module cache file handling ``50-cmds`` Modulefile commands (``setenv``, ``prepend-path``, ``conflict``, ``variant``, ...) -- by far the biggest series ``51-scan`` Modulefile scan / extra-match search ``60-initx`` The ``init*`` sub-commands (``initadd``, ``initprepend``, ``initrm``, ``initswitch``, ``initlist``, ``initclear``) that edit a user's shell startup file ``61-coll`` Module collections ``70-maint`` Maintenance sub-commands (``load``, ``switch``, ``purge``, config, ...) ``80-deep`` "Deep" modules, i.e. modules whose name has more than one directory level (``name/sub/version``): load/unload/switch, listing, ``whatis``, alias/symbolic-version resolution, access rights ``90-avail`` ``avail`` sub-command ``91-sort`` Module sorting order ``92-spider`` ``spider`` sub-command ``95-version`` Version comparison/parsing ``99-finish`` Testsuite teardown (removes cache files created for the run) ``install.00-init``, ``lint.00-init``, ``completion.00-init`` and ``cookbook.00-init`` are each a single series (those tools are much smaller and don't need topic splitting). Every series directory ends with a ``999-cleanup.exp`` file (see `Test file anatomy`_) and, for the ``modules`` tool, most series begin with a ``0NN-init_ts.exp`` file that sets up whatever fixtures that series' tests need (e.g. :file:`modules.90-avail/010-init_ts.exp`). ``completion.00-init`` layout ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``completion`` tool follows the common ``005``/``006``/``010``/``011`` setup numbering (see `Test file anatomy`_ and `Base test procedures`_ below) plus one extra layer, since driving real Tab-key completion is inherently shell-specific in a way none of the other three tools are: - ``005-init_ts.exp`` / ``006-procs.exp`` / ``010-environ.exp`` / ``011-save_test_env.exp`` set up paths (the ``bash``, ``zsh``, ``fish`` and ``tcsh`` binaries, the built :file:`init/bash_completion`, :file:`init/zsh-functions/_module`, :file:`init/fish_completion` and :file:`init/tcsh_completion` scripts, a clean fixture modulepath), the shell-agnostic assert procedures (``completion_assert_contains``, ``completion_assert_not_contains``, ``completion_assert_eq``, ``completion_assert_no_exec``), and the clean baseline environment/``save_test_env`` checkpoint, exactly as for the other tools. - ``0NN--procs.exp`` defines one ``completion__start`` / ``completion__raw`` / ``completion__list`` / ``completion__inline`` / ``completion__close`` set per shell -- e.g. ``020-bash-procs.exp`` spawns a real ``bash`` pty, sources :file:`init/bash_completion`, and drives double-Tab listings (``completion_bash_list``, for an ambiguous prefix) or single-Tab inline completions (``completion_bash_inline``, for a prefix with exactly one match -- e.g. checking a directory-style entry completes with a trailing ``/`` and no trailing space) through Expect. ``completion__list``/``_inline`` are responsible for recording the cmdline they were passed into the shared ``completion_last_cmdline`` variable, which the generic assert procedures use to build their test label. - ``0NN-.exp`` (e.g. ``021-bash.exp``, ``031-zsh.exp``, ``041-fish.exp``, ``051-tcsh.exp``) holds the actual test cases for that shell, calling only its own ``completion__*`` procs plus the shared asserts. A shell whose completion script covers less ground than the others (e.g. fish's has no ``ml`` support and does not gate its option flags by sub-command; the tcsh ``ml`` has no global switches at all, and its ``unuse`` only lists modulepaths from a bare, empty word) documents each such gap in its own file's header instead of forcing every file to test the same thing. Adding a new shell means adding its own ``completion__*`` procs file and test file; nothing in ``006-procs.exp`` needs to change. ``cookbook.00-init`` layout ~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``cookbook`` tool follows the common ``005``/``006``/``010``/``011`` setup numbering (see `Test file anatomy`_ and `Base test procedures`_ below), driving ``modulecmd.tcl`` directly through ``testouterr_cmd``/ ``testouterr_cmd_re`` exactly like the ``modules`` tool, plus one shared helper that is specific to this tool: - ``005-init_ts.exp`` defines ``cookbookdocdir``/``cookbookexampledir`` (pointing at :file:`doc/source/cookbook` and :file:`doc/example`), the ``cookbooksandbox`` fixture root (recreated fresh for the whole run), ``cookbook_read_example_file``/``cookbook_write_file`` (read a file shipped under :file:`doc/example//`, applying a ``string map`` substitution list, and write it into the sandbox), and ``cookbook_parse_transcript`` -- parses the ``.. parsed-literal::`` blocks of a named section (e.g. ``Usage example``) of a recipe's ``.rst`` file and returns the ordered list of ``{command output}`` pairs shown there, role markup (``:sgrhi:`text``` and friends) stripped from the output, so a recipe test file drives the exact command sequence its own documentation demonstrates, in that order, and checks its own real output is similar to the one illustrated -- instead of both being a hand-copied duplicate that could silently drift from the documentation. ``cookbook_output_re`` turns one such illustrated output into the regexp a sandboxed run's real output is checked against: escaped as a literal, except a run of ``-`` padding (widened to ``[-]+``, since its exact width depends on terminal-width detection a non-tty test run does not have) and whatever literal substrings the recipe test passes it to widen too (e.g. a placeholder path mapped to the sandbox's real one). - ``006-procs.exp``/``010-environ.exp``/``011-save_test_env.exp`` set up ``_test_sub``, a clean baseline environment (sandboxed ``$HOME``, no color, deterministic ``avail``/``list`` output, ``MODULES_SITECONFIG``/ ``MODULES_TAG_ABBREV``/``MODULES_NON_EXPORTABLE_TAGS`` left unset so each recipe states what it actually relies on), and the ``save_test_env`` checkpoint, exactly as for the other tools. - ``0NN-.exp`` (e.g. ``020-sync-remote-appdir.exp``), named after the recipe's own ``.rst`` file, holds one recipe's test: it checks any external binary the recipe needs (``unsupported`` + ``return`` if missing, e.g. ``rsync`` for ``sync-remote-appdir``), builds a ``$cookbooksandbox/`` fixture from the recipe's real :file:`doc/example//` files via ``cookbook_read_example_file`` (only substituting whatever real-system absolute path the recipe hardcodes, e.g. ``/remote_apps``, for a sandbox path -- everything else is used byte for byte), points the test environment at that sandbox, then loops over ``cookbook_parse_transcript``'s steps in order, running each with ``testouterr_cmd_re`` against ``cookbook_output_re`` of its illustrated output (stdout, the env-var-assignment syntax a shell evals silently and no recipe doc illustrates, is still checked against a hand-built answer per step -- ``unresolved`` + ``return`` first if the transcript's step count no longer matches what the test knows how to build one for, as a signal it needs updating alongside the doc). Simulates each step's effect on the environment for the next one exactly as a real shell would (``setenv_loaded_module``/``setenv_path_var``), since every ``modulecmd.tcl`` invocation here is an independent process. Adding a new recipe means adding its own ``0NN-.exp`` file; nothing in ``005-init_ts.exp`` needs to change unless the new recipe needs a kind of substitution or transcript shape the existing helpers do not cover yet. Fixture and support directories ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - :file:`modulefiles/`, :file:`modulefiles.2/`, :file:`modulefiles.3/`, :file:`modulefiles.4/`, and a number of single-purpose :file:`modulefiles./` directories (``.deep``, ``.deps``, ``.eb``, ``.spider1``/``.spider2``/``.spider3``, ``.rc``, ``.path1``/``.path2``/ ``.path3``, ``.memcache``, ``.indepth``, ``.allin``) hold the modulefiles used as fixtures. See `Adding new test fixtures`_ below for which one to add to. - :file:`config/` holds the DejaGnu/Expect configuration shared by every ``.exp`` file: :file:`base-config.exp` (all the test procedures, see `Base test procedures`_) and :file:`unix.exp` (the low-level plumbing that spawns ``modulecmd.tcl`` and captures its stdout/stderr/exit code). - :file:`etc/`, :file:`home/`, :file:`home.2/`-:file:`home.4/` provide sandboxed ``$MODULESHOME``-like/``$HOME``-like trees (rc files, collections, ...) used as fixtures. - :file:`bin/` contains small stand-in executables used by tests (fake tools invoked by modulefiles or by shell code under test). - Miscellaneous single-purpose helper scripts at the top level: :file:`is_func_defined` / :file:`is_func_defined.fish` (check whether ``module``/``mogui`` is defined as a shell function before running the ``install`` tests), :file:`stdin_to_file`, :file:`stty`, :file:`mode`, :file:`not_installed`, :file:`systest`/:file:`systest0-2`, :file:`id`, :file:`cmd.exe`, :file:`manpath`, :file:`virttargets/`. - :file:`mb/` holds fixtures for the :file:`script/mb` benchmark tool, not for DejaGnu tests. Test file anatomy ----------------- Every ``.exp`` file follows the same three-part shape: a header comment block, the test content, and (for most files, though not the ``00-init`` setup files) a cleanup footer. Header ~~~~~~ A fixed-format comment block, kept for historical (SCCS/CVS-style keyword expansion) reasons and consistency rather than for tooling: .. code-block:: tcl ############################################################################## # Modules Revision 3.0 # Providing a flexible user environment # # File: modules.50-cmds/%M% # Revision: %I% # First Edition: 2021/03/04 # Last Mod.: %U%, %G% # # Authors: Your Name, your.email@example.com # # Description: Testuite testsequence # Command: load, display, help, test # Modulefiles: variant # Sub-Command: # # Comment: %C{ # Test 'variant' modulefile command # }C% # ############################################################################## When adding a file, copy this header from a neighboring file in the same directory, update ``First Edition`` to the current date, your name/email in ``Authors`` if you want authorship credit, and fill in ``Command:``/ ``Modulefiles:``/``Sub-Command:`` and the free-text ``Comment:`` block to describe what the file tests. The ``%M%``/``%I%``/``%U%``/``%G%`` tokens are left as-is (they are not expanded by anything in this codebase but are kept for consistency with the historical format). Content ~~~~~~~ The body of the file is plain Tcl/Expect code, structured as: 1. Optional per-file setup: pick/point to a modulepath (e.g. ``set mp $modpath.3``), set environment variables relevant to the scenario (``setenv_var``/``setenv_path_var``), maybe call ``skip_if_quick_mode`` or ``skip_if_os_in`` right after the setup that only the skipped tests need. 2. One or more test cases: each is one call to a ``test*_cmd*`` procedure from :file:`config/base-config.exp` (see `Base test procedures`_), grouped under a short ``#\n# \n#`` comment banner when the file covers more than one scenario. 3. A ``#\n# Cleanup\n#`` banner followed by a call to ``reset_test_env`` (see below), for any file that changed environment variables or Tcl globals the following files shouldn't see. Footer ~~~~~~ Files that mutate shared state end with: .. code-block:: tcl # # Cleanup # reset_test_env ``reset_test_env`` restores every Tcl global variable and environment variable to the value it had when ``save_test_env`` last ran (this checkpoint is taken once per tool, early in the ``00-init`` series, right after the common fixtures -- modulepath, cache, etc. -- are set up; see :file:`modules.00-init/085-save_test_env.exp`). This is what makes test files independent of run order within reasonable bounds: a file can freely ``setenv_var``, build a collection, flip a config option, etc., and the next file starts clean. A handful of files close with a Vim modeline instead (or in addition): .. code-block:: tcl # vim:set tabstop=3 shiftwidth=3 expandtab autoindent: How a test case is built ------------------------ A test case has two ingredients: the environment/fixtures it runs against (set up with plain Tcl and the ``setenv_*``/``unsetenv_*`` helpers), and one call to a ``test*_cmd*`` procedure that runs a :command:`module` command line and checks its outcome. Setting up environment ~~~~~~~~~~~~~~~~~~~~~~ - ``setenv_var var val`` / ``unsetenv_var var`` -- set/unset a plain environment variable for the *modulecmd.tcl* child process about to be spawned. - ``setenv_path_var var elt1 elt2 ...`` / ``unsetenv_path_var var`` -- same, for a colon-delimited path-like variable; also transparently maintains the matching ``__MODULES_SHARE_`` reference-count variable. - ``setenv_loaded_module modlist modfilelist`` / ``unsetenv_loaded_module`` -- set up ``LOADEDMODULES``/``_LMFILES_`` (and ``__MODULES_LMTAG`` for auto-loaded modules) to simulate modules already loaded prior to the command under test. - ``skip_if_quick_mode`` -- bail out of the rest of the *current file* when ``QUICKTEST=y``. Used to skip expensive/exhaustive variations while keeping at least one representative case running in quick mode. - ``skip_if_os_in os1 os2 ...`` -- bail out of the rest of the current file on the given OS names (``$::os_name``, e.g. ``windows``, ``darwin``). - ``change_file_perms``/``restore_file_perms`` -- temporarily lock down a file/directory's permissions to test permission-denied paths. Running the command and checking the result ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Every ``test*_cmd*`` procedure runs ``modulecmd `` through ``_test_sub`` and compares the captured stdout/stderr/exit code against what was passed in, reporting ``pass``/``fail`` to DejaGnu. ``_test_sub`` itself is tool-specific: each tool defines its own version in its ``00-init`` series' ``006-procs.exp`` file (e.g. :file:`modules.00-init/006-procs.exp`), which for the ``modules`` tool calls down into ``modulecmd_xxx_`` (defined in :file:`config/unix.exp`) to actually spawn ``modulecmd.tcl`` and capture its output. See `Base test procedures`_ for the full family of these procedures and how their ``answer``/``anserr`` arguments are interpreted (plain string, a "shell out list", or the ``OK``/``ERR``/``ERR2`` shorthands). Base test procedures -------------------- All of the below live in :file:`testsuite/config/base-config.exp` (``shell_*`` internal helpers, used by the procedures below to translate a symbolic environment change into the actual syntax of each of the 15+ supported shells, are not covered here). Environment/output description helpers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``shell_out test_shell out_list`` builds the shell-specific expected output for a *list* of symbolic environment changes -- this is what most test cases pass as their ``answer``/``anserr`` argument instead of a literal string. Each element of ``out_list`` is itself a small list starting with one of: ``set`` ``{set var val}`` -- variable assignment (pass ``noescval`` as a 4th element to skip shell escaping) ``setpath`` ``{setpath var val}`` -- like ``set``, but also auto-generates the matching ``__MODULES_SHARE_`` reference-count assignment ``unset`` ``{unset var}`` -- variable unset ``unsetpath`` ``{unsetpath var}`` -- like ``unset``, plus reference count ``alias`` ``{alias var val}`` -- shell alias definition ``unalias`` ``{unalias var}`` -- shell alias removal ``chdir`` ``{chdir dir}`` -- directory change ``xres`` ``{xres var val}`` -- X11 resource set (``xrdb``) ``unxres`` ``{unxres var}`` -- X11 resource removal ``text`` ``{text str}`` -- arbitrary literal text (module messages, warnings, ...), inserted as-is ``OK``, ``ERR``, ``ERR2`` shell-specific success / 1-error / 2-error status code ``out``, or anything else literal text joined as-is ``is_shell_out_list answer`` tells whether ``answer`` looks like one of these lists (used internally by ``_test_out``/``_test_out_re`` to decide whether to run it through ``shell_out`` or treat it as a literal string). Test procedures (the ones test files actually call) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ All of them share the same first two arguments, ``test_shell`` (a shell name, or ``ALL`` to run the same check against every shell in ``$supported_shells``) and ``cmd`` (the :command:`module` command line, without the leading ``module`` word), plus a trailing optional ``failcmd`` (defaults to ``fail``; some tests in the deep-reload/args series pass ``untested`` or ``unresolved`` here) and ``launcher`` (to prefix the call, e.g. with ``sudo``). ``testouterr_cmd`` is the preferred procedure for a new test case: it checks both stdout and stderr, so a message unexpectedly landing on the wrong stream does not go unnoticed, and it is by far the most used procedure in the existing testsuite. Reach for one of the others only when its extra check (exit code, regexp matching, file content, ...) or its narrower scope (single-stream only) is actually needed. ``test_cmd test_shell cmd answer {exitval 0} {failcmd fail} {launcher {}}`` Checks stdout (exact) and the exit code (default expected: ``0``). ``test_cmd_re test_shell cmd answer {failcmd fail} {launcher {}}`` Checks stdout (regexp). ``testerr_cmd test_shell cmd answer {failcmd fail} {launcher {}}`` Checks stderr (exact). ``testerr_cmd_re test_shell cmd answer {force_nl 0} {failcmd fail} {launcher {}}`` Checks stderr (regexp). ``force_nl`` forces a trailing newline to be appended to ``answer`` before it is used as a regexp, for the cases where that can't be inferred automatically. ``testouterr_cmd test_shell cmd answer anserr {failcmd fail} {launcher {}}`` Checks stdout (exact, ``answer``) and stderr (exact, ``anserr``). **Preferred procedure, see above.** ``testouterr_cmd_re test_shell cmd answer anserr {force_nl 0} {failcmd fail} {launcher {}}`` Checks stdout (regexp, ``answer``) and stderr (regexp, ``anserr``). ``testouterr_cmd_re_sort test_shell cmd answer anserr {failcmd fail} {launcher {}}`` Checks stdout (regexp) and stderr (regexp), both sides line-sorted first -- for output whose line order is not guaranteed. ``testall_cmd test_shell cmd answer anserr exitval {failcmd fail} {launcher {}}`` Checks stdout (exact), stderr (exact) and the exit code. ``testall_cmd_re test_shell cmd answer anserr exitval {force_nl 0} {failcmd fail} {launcher {}}`` Checks stdout (regexp), stderr (regexp) and the exit code. ``testinouterr_cmd test_shell cmd input answer anserr {failcmd fail} {launcher {}}`` Like ``testouterr_cmd`` but also feeds ``input`` on stdin. ``testoutfile_cmd test_shell cmd answer filepath ansfile {failcmd fail} {launcher {}}`` Checks stdout (exact, ``answer``) and the content of ``filepath`` on disk (exact, ``ansfile``). ``testouterrfile_cmd test_shell cmd answer anserr filepath ansfile {failcmd fail} {launcher {}}`` Checks stdout (exact), stderr (exact) and the content of ``filepath`` on disk (exact, ``ansfile``). ``testouterrfileglob_cmd test_shell cmd answer anserr fileglob ansfile {failcmd fail} {launcher {}}`` Like ``testouterrfile_cmd``, but the file to check is resolved from the ``fileglob`` glob pattern (last match once sorted). ``testouterrgloblist_cmd test_shell cmd answer anserr fileglob ansglob {failcmd fail} {launcher {}}`` Checks stdout (exact), stderr (exact) and the *sorted list* of files matching the ``fileglob`` glob pattern (against ``ansglob``). For the ``ERR``/``ERR2``/``OK`` shorthands accepted as ``answer``/``anserr``: ``OK`` means "no output expected", ``ERR``/``ERR2`` expand to the shell-specific representation of exit status 1/2 (via ``shell_err``). Environment save/restore and mode helpers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - ``save_test_env`` / ``reset_test_env`` -- see `Footer`_ above. - ``is_quick_mode`` / ``skip_if_quick_mode`` -- see `Setting up environment`_ above. - ``skip_if_os_in`` -- see `Setting up environment`_ above. A worked example ---------------- The following two examples put together everything from `How a test case is built`_ and `Base test procedures`_ into concrete, runnable test cases: one where the load is rejected, one where it succeeds. Example 1: a rejected load (conflict) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is a trimmed-down version of one of the checks in :file:`modules.50-cmds/131-conflict-module.exp`: loading a module while a conflicting one is already loaded, with automatic conflict-unload disabled (the project's default), must fail with an explanatory error. The scenario relies on two existing fixture modulefiles: ``trace/all_on`` and ``conflict/module`` (:file:`testsuite/modulefiles/conflict/module`), whose ``.modulerc``-less content simply declares ``conflict trace``. .. code-block:: tcl # environment setup # # MODULEPATH already points to $modpath (the default fixture directory, # testsuite/modulefiles) -- that is set up once for the whole run by # modules.00-init/050-modpath.exp, so there is nothing to do here unless # a test needs a different modulepath. Simulate that 'trace/all_on' is # already loaded, without actually running a 'load' command for it: setenv_loaded_module trace/all_on $modpath/trace/all_on # build the expected stdout ($ans) # # the conflict is detected before any environment change is made, so # the load is rejected outright: stdout only carries the shell's # failing exit status, i.e. the 'ERR' shorthand (see `Base test # procedures`_) set ans ERR # build the expected stderr ($tserr) # # 'module load' always echoes a "Loading " banner first; the # msg_load and err_conflict helpers (defined in # modules.00-init/005-init_ts.exp) build that banner and the # conflict/hint message actually printed by modulecmd.tcl set tserr [msg_load conflict/module [err_conflict trace/all_on]] # test launch # # run 'module load conflict/module' for every supported shell and check # its stdout against $ans and its stderr against $tserr testouterr_cmd ALL "load conflict/module" $ans $tserr Running this (e.g. pasted at the end of an existing test file, or as its own numbered ``.exp`` file, see `Test file anatomy`_) exercises the real :command:`module load` code path for every supported shell and would report one ``PASS``/``FAIL`` per shell. For reference, this is what ``modulecmd.tcl`` itself prints for the ``sh`` shell in this exact scenario:: $ MODULEPATH=.../modulefiles LOADEDMODULES=trace/all_on \ _LMFILES_=.../modulefiles/trace/all_on \ modulecmd.tcl sh load conflict/module Loading conflict/module ERROR: Module cannot be loaded due to a conflict. HINT: Might try "module unload trace/all_on" first. test 0 = 1; The last line (``test 0 = 1;``) is the ``sh``-specific encoding of a failing exit status -- exactly what ``ERR`` expands to via ``shell_err`` (see `Base test procedures`_) -- and everything above it on stderr is what ``$tserr`` was built to match. Example 2: a successful load, from ``$modpath.4`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This second example contrasts with the first one: a plain, successful load, with the environment changes it produces checked on stdout and nothing expected on stderr. It also illustrates pointing the modulepath at :file:`testsuite/modulefiles.4` (``$modpath.4`` in Tcl, see `Adding new test fixtures`_) instead of using the default fixture directory, as a new test case exercising a fixture added there would, and running the check against a single named shell (``bash``) rather than ``ALL`` -- appropriate when a test is not about shell-syntax differences and does not need to be repeated for every supported shell. The fixture used is :file:`testsuite/modulefiles.4/setenv/1.0`, whose content is simply:: #%Module setenv TS1 {} setenv TS2 $env(TS1) .. code-block:: tcl # environment setup # # point the modulepath at modulefiles.4 instead of the default fixture # directory setenv_path_var MODULEPATH $modpath.4 # build the expected stdout ($ans) # # 'setenv/1.0' sets TS1 to an empty string then TS2 to the (now empty) # value of TS1; 'load' also always sets _LMFILES_ and LOADEDMODULES -- # each element uses the 'set' tag from the shell_out vocabulary (see # `Base test procedures`_) so the answer is rendered in bash syntax set ans [list] lappend ans [list set TS1 {}] lappend ans [list set TS2 {}] lappend ans [list set _LMFILES_ $modpath.4/setenv/1.0] lappend ans [list set LOADEDMODULES setenv/1.0] # test launch # # run against bash only; a plain successful load prints nothing on # stderr, hence the empty string passed as the expected anserr testouterr_cmd bash "load setenv/1.0" $ans {} Running the testsuite --------------------- With ``make`` ~~~~~~~~~~~~~ .. code-block:: sh make test # full 'modules' suite via DejaGnu (~25,000 cases, ~12 min) make test QUICKTEST=y # quick mode, most essential tests (~1,900 cases, ~1 min) make test COVERAGE=y # coverage-instrumented run, produces tcl/*.tcl_m make testinstall # 'install' suite against a tree already processed by 'make install' make testlint # 'lint' suite (Nagelfar + ShellCheck) make testcompletion # 'completion' suite (interactive Tab-completion tests) make testcookbook # 'cookbook' suite (doc/source/cookbook recipes) Each target ends up calling ``runtest --tool $(RUNTESTFLAGS) $(RUNTESTFILES)``: ``RUNTESTFILES``, if set, restricts the run to specific ``.exp`` file *names* (not paths -- DejaGnu matches by basename across every directory of the active tool); ``RUNTESTFLAGS`` controls DejaGnu verbosity (``-v`` can be repeated; at ``-v -v`` the test procedures in :file:`base-config.exp` additionally dump the raw captured ``OUT[len]: '...'#>``/``ERR[len]: '...'#>``/``EXIT: '...'#>`` for every check, matched vs expected). With :file:`script/mt` ~~~~~~~~~~~~~~~~~~~~~~ :file:`script/mt` is a thin wrapper around the ``make`` targets above that adds live progress reporting and a readable diff on failure. Prefer it over calling ``make test`` directly when iterating on a specific area. .. code-block:: sh script/mt # same as: make test script/mt quick # same as: make test QUICKTEST=y script/mt cov # same as: make test COVERAGE=y script/mt install # same as: make testinstall script/mt lint # same as: make testlint script/mt comp # same as: make testcompletion script/mt cook # same as: make testcookbook script/mt 50/470 # only testsuite/modules.50-cmds/470-*.exp script/mt 50 # every file in testsuite/modules.50-cmds script/mt 61 # collection series (always run whole, see below) script/mt lint 00/030 # only testsuite/lint.00-init/030-*.exp script/mt comp 00/021 # only testsuite/completion.00-init/021-*.exp script/mt cook 00/020 # only testsuite/cookbook.00-init/020-*.exp script/mt 50/{280,290} 61 # several selections at once script/mt --help # full usage Whichever files are selected, :file:`script/mt` always also runs the mandatory setup files for that tool (for ``modules``: ``00/005 00/006 00/010 00/050 00/060 00/080 00/085``; for ``install``: ``00/005 00/006 00/010 00/011``; for ``lint``: ``00/005 00/006 00/011``; for ``completion``: ``00/005 00/006 00/007 00/008 00/010 00/011 00/020 00/030 00/040 00/050``; for ``cookbook``: ``00/005 00/006 00/010 00/011``), plus the ``999-cleanup.exp`` of every selected series. Passing a bare series number always expands to every file in that directory, because several of those series are order-sensitive or enumerate a whole modulepath (see `Adding new test fixtures`_). The collection series (``61``) is one such case: its files create real collection files on disk that later files in the same series depend on -- e.g. :file:`modules.61-coll/040-restore.exp` restores collections that :file:`modules.61-coll/030-save.exp` produced -- so running only a subset of that series would fail for reasons unrelated to the actual test. While running, :file:`script/mt` tails the produced log and prints a running ``pass``/``fail``/``xfail``/``error`` tally, plus a per-testcase breakdown whenever a case other than all-pass completes. When done, it invokes :file:`script/mtreview` on the log, which pulls out every ``OUT[..]``/``ERR[..]`` vs ``EXP[..]`` pair recorded by the ``-v -v``-level test procedures and renders them as a diff (using ``icdiff`` if available -- :file:`script/mt` auto-downloads it once and caches it as ``./icdiff`` -- falling back to ``diff -u``). Coverage ~~~~~~~~ .. code-block:: sh make test COVERAGE=y # instruments modulecmd.tcl with Nagelfar, # then runs the full suite and produces # tcl/*.tcl_m script/mt cov 70/{280,290} # coverage run limited to specific test # files Inspect the resulting :file:`tcl/*.tcl_m` files for lines flagged ``;# Not covered`` to see what a newly added feature/branch still needs test coverage for. Adding new tests ---------------- Adding a new test case to an existing area ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Add a ``test*_cmd*`` call (or a whole new numbered ``.exp`` file, following `Test file anatomy`_) to the relevant ``.-`` directory. Pick the file/series whose topic matches, or create a new file with the next free number in that series if the scenario doesn't fit an existing file. Run the file in isolation first (``script/mt /``) before running the full suite. Adding new test fixtures ~~~~~~~~~~~~~~~~~~~~~~~~ Add new fixture modulefiles under the *highest-numbered* :file:`testsuite/modulefiles.N` directory (check whether a higher one than :file:`modulefiles.4` already exists before assuming it's the latest), not under the default :file:`modulefiles/` directory. Several existing tests (``aliases``, ``avail``, ``spider``, ``scan_eval``, and others) enumerate a modulepath's *entire* content and assert on the full listing; adding fixtures to a directory shared that broadly changes their expected output. Even a numbered directory like :file:`modulefiles.4` is referenced wholesale by some tests and by filesystem-glob-order-dependent debug-trace assertions elsewhere in the suite, so: 1. Add the new modulefile(s) under the latest :file:`modulefiles.N`. 2. Run the *full* testsuite (or at least the ``90-avail``/``92-spider`` series plus anything else touching that modulepath), not just the new test file, to catch any global-enumeration test whose expected output now needs updating. 3. Update the expected output of any such test the new fixture broke. If a fixture needs a filename ending in a space, it cannot be checked into git as a real file (checkout fails on Windows); instead generate it on the fly from Tcl, following the pattern in ``create_endspace_test_modulefiles``/``delete_endspace_test_modulefiles`` in :file:`config/base-config.exp`. Adding a new sub-command or config option ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Don't improvise which testsuite files need touching for these two specific tasks -- :doc:`add-new-sub-command` and :doc:`add-new-config-option` each enumerate the exact list, in the "Testsuite" section of each document. Debugging a broken test ----------------------- 1. **Reproduce narrowly.** Re-run just the failing file with :file:`script/mt`, e.g. ``script/mt 50/470``. This also runs the tool's mandatory setup files, so the environment matches a full run. 2. **Read the diff.** :file:`script/mt` runs :file:`script/mtreview` on the resulting log automatically and prints an ``EXP``-vs-``OUT``/``ERR`` diff for every failing case -- start there instead of reading the raw log. 3. **Raise verbosity.** For more detail than the diff gives (e.g. to see *which* command line in a file produced a given failure, or to see ``send_user`` progress messages emitted by ``config/base-config.exp`` helpers like ``setenv_var``/``change_file_perms``), set ``RUNTESTFLAGS='-v -v'`` and call ``make test``/``testinstall``/ ``testlint``/``testcompletion``/``testcookbook`` directly, or invoke ``runtest`` yourself with the environment variables :file:`script/mt`/the Makefile targets set up (``TCLSH``, ``MODULECMD``, ``OBJDIR``, ``TESTSUITEDIR``) -- see the ``test``/``testinstall``/``testlint``/``testcompletion``/``testcookbook`` targets in :file:`Makefile` for the exact invocation. 4. **Check for order dependence.** If a test passes alone but fails in a full run (or vice-versa), suspect a missing/incomplete ``reset_test_env`` footer in an earlier file, or a global-enumeration test (`Adding new test fixtures`_) whose expected output a fixture change invalidated. 5. **Suspect the module cache.** If ``TESTSUITE_ENABLE_MODULECACHE`` is set in the environment and a test's behavior seems to depend on whether the modulepath is cached, check whether the file calls ``ignore_modulecache_if_built``/``end_ignore_modulecache_if_built`` around the section that shouldn't read a pre-built cache. 6. **Nagelfar/ShellCheck failures (``lint`` tool)** report the offending file and line directly in the diff; fix the flagged code or, if it's a deliberate/false-positive pattern, look at how neighboring :file:`lint.00-init/0NN-*.exp` files configure linter exclusions (e.g. the ``-e SC1090`` ShellCheck exclusion in :file:`lint.00-init/020-sh.exp`) before adding a new one. 7. **``completion`` tool hangs/timeouts.** A run that hangs (rather than fails) almost always means an ``expect`` pattern in a ``completion__*`` proc never matched, so it burned the default timeout before falling through -- three recurring causes when scripting readline-based completion: sending only a double-Tab against a prefix that still has an unconsumed common-prefix extension (readline auto-inserts it on the first Tab, so the listing needs a *third* Tab -- type the full common prefix yourself instead, see the ``module load ba`` case in :file:`completion.00-init/021-bash.exp`); clearing an input line and then matching on the prompt text reappearing (readline redraws a cleared line with cursor-movement escapes, not by reprinting the prompt -- submit the now-empty line instead, see ``completion_bash_list`` in :file:`completion.00-init/020-bash-procs.exp`); and an interactive pager (``less``) kicking in on an unexpected warning and blocking for a keypress that never comes (set ``MODULES_PAGER=cat``, see :file:`completion.00-init/010-environ.exp`). 8. **Coverage regressions.** If a change is meant to add coverage for a new branch, confirm it with ``make test COVERAGE=y`` (or ``script/mt cov /``) and check the relevant :file:`tcl/*.tcl_m` file no longer flags that line ``;# Not covered``. .. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: