New features ************ This document describes the major changes occurring between versions of Modules. It provides an overview of the new features and changed behaviors that will be encountered when upgrading. v5.7 ==== This new version is backward-compatible with previous version 5 release. It fixes bugs but also introduces new functionalities that are described in this section. See the 5.7 release notes for a complete list of the changes between Modules v5.6 and v5.7. Improving evaluation performances --------------------------------- We observed that Modules could feel slow when evaluating more than one hundred modulefiles within a single command. To address this, several improvements have been implemented: * In-memory caching of expensive operations: the results of costly operations, such as module specification comparisons, are now cached in memory. * Reworked environment synchronization between Tcl interpreters: environment synchronization during Tcl modulefile evaluation has been redesigned to rely on Tcl’s internal mechanisms rather than a custom reimplementation. As part of this change, calls to the Tcl "append" command have been replaced with "set", which reliably triggers synchronization when environment variables are modified. * More efficient dependency checks: when verifying dependencies against loaded modules, comparisons are now limited to the subset of loaded modules that match the module root name specification, instead of comparing against all loaded modules. * Lighter-weight resolution of an exact "name/version" specification: sites with a modulepath directory holding hundreds or thousands of sibling modulefiles reported slow "module load" commands even when the requested version already exists. Correctly computing the implicit "default" and "latest" symbolic versions of such a directory used to require validating every sibling modulefile in it. The lookup now walks candidate versions from the dictionary-highest to the dictionary-lowest and stops validating as soon as this can be answered: either a higher valid version is found (the requested one is then known not to be "default" or "latest"), or the requested version itself is reached, in which case nothing higher exists. This bypasses the exhaustive scan without changing the resolved result. For this work, the **mb** utility was extended with new benchmark tests that loads 137 modulefiles ("load2"), lists the loaded modules ("list2"), and then purges them. Running this benchmark highlights the performance improvements, as shown in the table below: +-----------+---------------+----------------------+ | | v5.6.1 | v5.7.0 | |===========|===============|======================| | load2 | 1038 ms | 384 ms (-63%) | +-----------+---------------+----------------------+ | list2 | 222 ms | 29 ms (-87%) | +-----------+---------------+----------------------+ | purge | 568 ms | 303 ms (-46%) | +-----------+---------------+----------------------+ A "load3" benchmark test was also added, loading a low, non-default version of a module out of 500 sibling versions in the same directory. It specifically exercises the lighter-weight resolution described above: +-----------+---------------+----------------------+ | | v5.6.1 | v5.7.0 | |===========|===============|======================| | load3 | 30 ms | 19 ms (-36%) | +-----------+---------------+----------------------+ These measurements were obtained on a system where all modulefiles are stored on local flash storage. If your modulefiles reside on a shared network filesystem, using a Module cache is strongly recommended. A module cache file aggregates the content of all the modulercs and modulefiles found in a modulepath and this whole file is evaluated when a module search occurs, even if the search targets specific modules. On setups with a very large number of modulefiles, resulting in cache files of several megabytes, evaluating the whole cache file may take longer than walking through the modulepath directory to find the few requested modules. The new "ignore_cache_subcmds" configuration option defines a list of module sub-commands on which module cache files are ignored. For instance to keep cache benefit on listing sub-commands but avoid its penalty on targeted ones: $ module config ignore_cache_subcmds load:try-load:load-any When the configuration option is changed from its default value using the "config" sub-command, the "MODULES_IGNORE_CACHE_SUBCMDS" environment variable is defined accordingly. See "MODULES_IGNORE_CACHE_SUBCMDS" for details. Reorder existing entries in path-like variables ----------------------------------------------- A new "path_entry_reorder" configuration option is introduced to control how path-like environment variables are modified when "prepend-path", "append-path" or "use" target a path entry that is already defined in the environment variable. By default, this configuration option is set to "0", meaning that when a path entry already exists, the environment variable is left unchanged. When set to "1", an existing entry is reordered to the beginning or the end of the variable, depending on the command used, unless duplicates are explicitly allowed. $ module append-path PATHVAR /foo $ module append-path PATHVAR /bar $ module append-path PATHVAR /foo $ echo $PATHVAR /foo:/bar $ module config path_entry_reorder 1 $ module append-path PATHVAR /foo $ echo $PATHVAR /bar:/foo $ module append-path --duplicates PATHVAR /bar $ echo $PATHVAR /bar:/foo:/bar The default value of "path_entry_reorder" can be defined at installation time with the "--enable-path-entry-reorder" option. When the configuration option is changed from its default value using the "config" sub-command, the "MODULES_PATH_ENTRY_REORDER" environment variable is defined accordingly. See "MODULES_PATH_ENTRY_REORDER" for details. Linked environment variables ---------------------------- A new configuration option, "linked_envvars", can be used to automatically mirror environment variable changes onto other environment variables. When a modulefile modifies an environment variable using commands such as "setenv", "unsetenv", "prepend-path", "append-path", "remove-path" or "pushenv", the same operation is also applied to all variables linked through the "linked_envvars" configuration option. For example, with the following configuration: $ module config linked_envvars "CPATH&INCLUDE_PATH" loading a module that prepends a directory to "CPATH": prepend-path CPATH /opt/pkg/include will also prepend the same directory to "INCLUDE_PATH". Links are directional. In the previous example, changes made to "CPATH" are also applied to "INCLUDE_PATH", but changes made directly to "INCLUDE_PATH" are not applied back to "CPATH" unless an explicit reverse link is configured. When enabling this feature, note that existing values are not synchronized when the link is established. Only subsequent modifications performed through modulefile environment variable commands are propagated to linked variables. The output of "module display" also includes the environment variable changes applied to linked environment variables, making these propagated updates visible before a module is loaded. Initialize environment variables on first modification ------------------------------------------------------ A new configuration option, "init_envvars", has been added to define initial values for environment variables that are modified through path-like modulefile commands. This feature is particularly useful for variables such as "MANPATH", which require a leading or trailing colon character to preserve access to the system default search paths. Previously, modifying an undefined "MANPATH" could prevent system manual pages from being found. See the Handle man pages search path cookbook for details. The "init_envvars" option defines a colon-separated list of "VARIABLE_NAME=initial_value" entries. When a path-like operation is applied to an environment variable that is not currently set, the configured initial value is assigned before the modification is performed. For example, with the following configuration: $ module config init_envvars MANPATH= loading a module that prepends a directory to "MANPATH": prepend-path MANPATH /opt/pkg/man will keep a colon character at the end of "MANPATH" $ echo $MANPATH /opt/pkg/man: This mechanism applies to all path-like environment variable management commands, including "append-path", "prepend-path" and "remove-path". When values are later removed from an environment variable, it is automatically unset if its resulting value matches the configured initial value and no explicit reference counter is associated with it. Hook API -------- Site code that needs to run right before or after a modulefile or modulerc evaluation could so far only be attached through the "trace" Tcl command or by renaming an internal "modulecmd.tcl" procedure, both of which bind to implementation details that may change from one Modules version to the next. A new "add-hook" siteconfig command is introduced to register a procedure on one of four stable events without relying on such internal details: "before-modulefile-eval", "after-modulefile-eval", "before-modulerc-eval" and "after-modulerc-eval". Several procedures can be registered on the same event; they are then called in their registration order. proc auditRequestedLoad {modfile modname modnamevr modspec mode requested} { if {$requested && $mode eq {load}} { set fd [open /var/log/modules-audit.log a] puts $fd "[clock format [clock seconds]] $modnamevr" close $fd } } add-hook before-modulefile-eval auditRequestedLoad An error raised by a hook procedure is reported but does not abort the running "module" command nor prevent other procedures registered on the same event from running, since these hooks fire on nearly every modulefile or modulerc evaluation. See the *Hooks* section of *module(1)* man page for the full argument contract of each event. The "trace"/rename techniques remain available for anything not (yet) covered by a hook event. These four events are only the first ones introduced through "add- hook"; additional hook events will be added in the future as real site needs for them come up. See Add new hook event for the guide contributors can follow to propose one. Ignore files in modulepath -------------------------- Modulepath directories sometimes hold files that are not modulefiles, like helper scripts sourced by modulefiles or software installation files when modulefiles are mixed with the software trees they describe. The new "modulepath-ignore" modulefile command lets modulepath maintainers describe from rc files such content to ignore when searching for modulefiles. Ignored files and directories are handled as if they do not exist: they are not checked, not walked down, not reported and not recorded in modulepath cache file. Cache build time and size are thus kept proportional to the number of actual modulefiles, whatever the amount of other files stored in modulepath. Patterns follow the same syntax and behavior than git with ".gitignore" files. For instance to ignore every file with the ".tcl" extension and the whole content of the "soft" directory, add to the ".modulerc" file at the root of the modulepath: modulepath-ignore *.tcl soft/ Negated patterns switch to an *ignore all, accept some* logic. For instance to only take files with the ".mod" extension into account: modulepath-ignore * !*/ !*.mod When defined in a ".modulerc" file located within a modulepath, patterns are relative to the directory of this rc file. When defined in a global or user rc file, patterns apply to every modulepath. As ignored elements are excluded from cache files when they are built, pattern definitions should not be conditional. The "modulepath_ignore" configuration option, enabled by default, controls whether pattern definitions are applied. Patterns can be skipped for one command execution with the "--no-modulepath-ignore" command line switch, which helps finding out whether a module is missing because it is ignored. v5.6 ==== This new version is backward-compatible with previous version 5 release. It fixes bugs but also introduces new functionalities that are described in this section. See the 5.6 release notes for a complete list of the changes between Modules v5.5 and v5.6. spider sub-command ------------------ Support has been added for the "spider" sub-command introduced by the Lmod project. This sub-command lists available modules found in enabled modulepaths and recursively found in modulepaths enabled by available modules. $ module avail --------------------- /path/to/modulefiles --------------------- foo/1 foo/2 $ module show foo/1 ------------------------------------------------------------------- /path/to/modulefiles/foo/1: conflict foo module use /path/to/modulefiles.2 ------------------------------------------------------------------- $ module show foo/2 ------------------------------------------------------------------- /path/to/modulefiles/foo/2: conflict foo append-path MODULEPATH /path/to/modulefiles.3 ------------------------------------------------------------------- $ module spider --------------------- /path/to/modulefiles --------------------- foo/1 foo/2 -------------- /path/to/modulefiles.2 (via foo/1) -------------- bar/1 bar/2 bar/3 -------------- /path/to/modulefiles.3 (via foo/2) -------------- bar/3 bar/4 -------------- /path/to/modulefiles.3 (via bar/1) -------------- qux/1 qux/2 The "spider" sub-command relies on the Extra match search mechanism to scan modulefiles and identify the modulepaths they enable. Since all modulefiles are evaluated during this scan, it is recommended to build and use a Module cache to improve search performance. The output of "spider" is similar to that of the "avail" sub-command and supports the same set of options and queries. $ module spider -t bar@2: /path/to/modulefiles.2: bar/2 bar/3 /path/to/modulefiles.3: bar/3 bar/4 The "spider" sub-command has its own configuration options to control the content of its output: "spider_indepth", "spider_output", and "spider_terse_output". These settings make it possible to define output for "spider" that differs from the "avail" sub-command. $ module config spider_output -modulepath $ module config spider_indepth 0 $ module spider bar/ foo/ qux/ When a modulepath is enabled by a specific module, that modulepath is considered available *via* that module. When the module is loaded, this *via* information is stored in the "__MODULES_LMUSE" environment variable to track which modules enable which modulepaths. By default, the *via* information is included in the standard output of the "spider" sub-command. Additionally, the JSON output for both the "avail" and "spider" sub-commands has been updated to include *via* details for each modulefile. $ module spider -j bar@2: {"/path/to/modulefiles.2": { "bar/2": { "name": "bar/2", "type": "modulefile", "symbols": [], "tags": [], "pathname": "/path/to/modulefiles.2/bar/2", "via": "foo/1"} "bar/2": { "name": "bar/3", "type": "modulefile", "symbols": [], "tags": [], "pathname": "/path/to/modulefiles.2/bar/3", "via": "foo/1"} }, "/path/to/modulefiles.3": { "bar/3": { "name": "bar/3", "type": "modulefile", "symbols": [], "tags": [], "pathname": "/path/to/modulefiles.3/bar/3", "via": "foo/2"}, "bar/4": { "name": "bar/4", "type": "modulefile", "symbols": [], "tags": [], "pathname": "/path/to/modulefiles.3/bar/4", "via": "foo/2"} }} Requiring *via* module ---------------------- A loaded module that enables a modulepath is considered the *via* module of other loaded modules whose modulefiles are stored in this modulepath. The "require_via" configuration option is introduced to consider a *via* module a requirement for the loaded module stored in the modulepath it enables. A *module hierarchy* mechanism, as introduced by the Lmod project, is now supported with this feature. It allows organizing modulefiles through a primary modulepath, where loading certain modules enables additional modulepaths. "require_via" maintains a link between modulefiles in these additional paths and the *via* module that activated them. The "require_via" setting is disabled by default to maintain compatibility with previous Modules 5 releases. However, users are encouraged to enable this feature if they want to use the *module hierarchy* mechanism. $ module config require_via 1 When unloading a *via* module, all the modules stored in its enabled modulepath are automatically unloaded thanks to the *Dependent Unload* mechanism of the Automated module handling mode. $ module load foo/1 bar/2 $ module avail -------------- /path/to/modulefiles.2 (via foo/1) -------------- bar/1 bar/2 bar/3 --------------------- /path/to/modulefiles --------------------- foo/1 foo/2 Key: loaded modulepath $ module unload foo Unloading foo/1 Unloading dependent: bar/2 When a *via* module is replaced by another, any modulepaths it enabled are updated accordingly. If the newly enabled modulepath provides alternatives to modules previously loaded from the original path, those modules are reloaded from the new location. If no alternatives are found, the dependent modules are unloaded. $ module config conflict_unload 1 $ module load foo/1 bar/3 $ module load foo/2 Loading foo/2 Unloading dependent: bar/3 Unloading conflict: foo/1 Reloading dependent: bar/3 $ module avail -------------- /path/to/modulefiles.3 (via foo/2) -------------- bar/3 bar/4 --------------------- /path/to/modulefiles --------------------- foo/1 foo/2 Key: loaded modulepath Always see hidden modules ------------------------- The "hidden" element is a new allowed value for the "avail_output", "avail_terse_output", "list_output", "list_terse_output", "spider_output" and "spider_terse_output" configuration options. When set into the value list of these options hidden modules are shown on the output of corresponding commands ("avail", "list" and "spider"). --------------- /path/to/modulefiles --------------- bar/1.0 $ module config avail_output +hidden $ ml av --------------- /path/to/modulefiles --------------- bar/1.0 foo/1.0 When set, this is equivalent to always adding the "--all" command-line option. Issue warning upon module evaluation ------------------------------------ The mfcmd:*module-warn* modulefile Tcl command is introduced to define warning message to print when loading specified modulefiles. $ cat /path/to/modulefiles/foo/1.0 #%Module module-warn --message {Please consider using "bar" module instead} foo $ module load foo Loading foo/1.0 WARNING: Please consider using "bar" module instead The warning message is printed when targeted modulefile is evaluated in *load*, *display*, *help* or *test* mode. "module-warn" supports the same kind of options than "module-tag" or "module-hide": it is possible to specify a list of affected or unaffected users or groups. Warning may also be only effective before or after a particular date. The "warning" tag is set on modules targeted by a mfcmd:*module-warn* command. Simplified way to define module help message -------------------------------------------- Until now module help message was defined through a specific Tcl procedure that outputs text to *stderr* channel: #%Module proc ModulesHelp {} { puts stderr {Module Name: foo, Version: 1.0} puts stderr {This modulefile does this and that} } A new modulefile Tcl command is introduced, "module-help", to simplify a bit the definition of help message: #%Module module-help {Module Name: foo, Version: 1.0} module-help {This modulefile does this and that} With this modulefile command there is no need to define the "ModulesHelp" procedure anymore. As "module-help" is reported on module *display* evaluation, help message will also be visible there. Support for newline character in environment variable value ----------------------------------------------------------- It is now possible on *sh*, *bash*, *ksh*, *zsh* and *fish* shells to set value for environment variable that contains newline character: $ cat /path/to/modulefiles/bar/1.0 #%Module setenv BAR "multinlinenvalue" $ module load bar $ echo "$BAR" multi line value Generated shell code on these shells has been changed to enclose value within single quotes rather escaping each special characters: $ module mod-to-sh bash bar/1.0 BAR='multi line value'; export BAR; Declare additional elements provided by a module ------------------------------------------------ New "provide" modulefile command is added to declare additional elements included in a modulefile. When the modulefile is loaded, each argument specified in the "provide" command is registered as an alias for the module. $ module show foo ------------------------------------------------------------------- /path/to/modulefiles/foo/1: provide liba/1.2 libb/2.1 conflict foo conflict liba libb ------------------------------------------------------------------- $ module load foo It is advised to define "conflict" over the name of these additional elements to ensure only one provider is found loaded in user environment. $ module show bar/1 ------------------------------------------------------------------- /path/to/modulefiles/bar/1: provide liba/1.2 libb/2.1 conflict bar conflict liba libb ------------------------------------------------------------------- $ module load bar Loading bar/1 ERROR: Module cannot be loaded due to a conflict. HINT: Might try "module unload foo/1" first. Once a module with a provided alias is loaded, it fulfills any requirements declared by other modules for that name and version, just like any other type of module alias. $ module show qux ------------------------------------------------------------------- /path/to/modulefiles/qux/1: prereq liba/1.2 ------------------------------------------------------------------- $ module load qux $ module list Currently Loaded Modulefiles: 1) foo/1 2) qux/1 Module aliases defined via the "provide" command are not available during module resolution, as modulefiles are not evaluated in that context. Therefore, it is recommended to define a global module alias in a "modulerc" file using the "module-alias" command to specify a default target for the alias. $ cat /path/to/modulefiles/.modulerc #%Module module-alias liba/1.2 bar/1 $ module purge $ module load qux Loading qux/1 Loading requirement: bar/1 The "provide" modulefile command also serves as an alias for the "extensions" command, ensuring compatibility with modulefiles written for Lmod. By default an "avail" command includes module aliases defined in "modulerc" files. To also get aliases defined within modulefiles with "provide" or "family", output element "provided-alias" has to be included. It is supported on "avail_output", "avail_terse_output", "spider_output" and "spider_terse_output" configuration options. Having "provided-alias" included in output triggers Extra match search mechanism that scans all modulefiles to find these aliases. It is thus advised to have Module cache built and used to improve search speed. $ module avail --------------------- /path/to/modulefiles --------------------- bar/1 foo/1 liba/1.2 qux/1 Key: modulepath module-alias $ module avail -o +provided-alias --------------------- /path/to/modulefiles --------------------- bar/1 foo/1 liba/1.2 libb/2.1 qux/1 Key: modulepath module-alias The "provided-alias" Extra specifier is also introduced to find modules that define such alias with "provide" or "family" modulefile commands. $ module avail provided-alias:liba/1.2 --------------------- /path/to/modulefiles --------------------- bar/1 foo/1 Key: modulepath v5.5 ==== This new version is backward-compatible with previous version 5 release. It fixes bugs but also introduces new functionalities that are described in this section. See the 5.5 release notes for a complete list of the changes between Modules v5.4 and v5.5. Logging activity ---------------- The ability to log module command activity is now available out of the box. It could be enabled on previous versions by using a specific "siteconfig.tcl" configuration script as described in the Log module command cookbook recipe. Integrated logging feature relies on two configuration options: * "logger", the command run to transmit messages to the log system * "logged_events", list of module event to log "logger" option relies on the **logger** command by default, which is usually available and already installed on most systems. The "logged_events" option is empty by default, so no content is sent to logs by default. It recognizes the following events: * "requested_cmd": record module commands typed by users * "requested_eval": record modulefile evaluations requested by users * "auto_eval": record modulefile evaluations automatically triggered $ module config logged_events +requested_cmd:requested_eval $ ml av --------------- /path/to/modulefiles --------------- bar/1.0 foo/1.0 qux/1.0 $ module load bar Loading bar/1.0 Loading requirement: qux/1.0 $ module purge In the above example the module command and modulefile evaluations directly requested by users are sent to the system log: $ journalctl -q -t modules Apr 29 07:47:42 hostname modules[3777797]: user="username" command="avail" arguments="" Apr 29 07:48:10 hostname modules[3777876]: user="username" command="load" arguments="bar" Apr 29 07:48:10 hostname modules[3777876]: user="username" mode="load" module="bar/1.0" specified="bar" modulefile="/path/to/modulefiles/bar/1.0" requested="1" Apr 29 07:48:17 hostname modules[3777914]: user="username" command="purge" arguments="" Apr 29 07:48:17 hostname modules[3777914]: user="username" mode="unload" module="bar/1.0" specified="bar/1.0" modulefile="/path/to/modulefiles/bar/1.0" requested="1" Some messages can also be sent during modulefile evaluation by using the "log" channel of the "puts" command: $ cat /path/to/modulefiles/foo/1.0 #%Module puts log {some message sent to log} $ ml foo/1.0 $ journalctl -q -t modules Apr 29 07:47:51 hostname modules[3777835]: user="username" command="ml" arguments="foo/1.0" Apr 29 07:47:51 hostname modules[3777835]: some message sent to log Apr 29 07:47:51 hostname modules[3777835]: user="username" mode="load" module="foo/1.0" specified="foo/1.0" modulefile="/path/to/modulefiles/foo/1.0" requested="1" Negating extra match search criteria ------------------------------------ The "not:" prefix string is introduced for extra specifier and variant criteria. When this prefix is used, the search criteria where it is applied is negated. $ module config avail_output +variant $ module config variant_shortcut toolchain=% $ module avail --------------------- /path/to/modulefiles --------------------- bar/1.0{%x86_64} foo/2.0{%x86_64,arm64} bar/2.0{%x86_64,arm64} qux/1.0{%x86_64} foo/1.0{%x86_64} qux/2.0{%x86_64,arm64} $ module avail not:envvar:FOO,BAR --------------------- /path/to/modulefiles --------------------- qux/1.0{%x86_64} qux/2.0{%x86_64,arm64} The above search query returns all modules not defining "FOO" and "BAR" environment variables. $ module avail not:envvar:BAR not:%arm64 --------------------- /path/to/modulefiles --------------------- foo/1.0{%x86_64} qux/1.0{%x86_64} The above example returns all modules not defining "BAR" environment variable and not defining "toolchain" variant or defining this variant without "arm64" among the possible values. This prefix is recognized on module search context (i.e., "avail", "whatis" and "paths" sub-commands). PowerShell support ------------------ Support for PowerShell ("pwsh") is added. This shell may be used with Modules on either Unix or Windows platforms. All main kind of environment changes are supported. A "envmodule" shell function is configured on PowerShell, as "module" is a reserved keyword on this shell. The Windows-specific distribution zipball of Modules has been updated to also contain the required files to setup PowerShell support. Conflict unload --------------- Automated module handling is enhanced with the addition of a new mechanism: *Conflict Unload*. When loading a module, this new mechanism automatically unloads conflicting modules and their dependent. $ module load foo/1 Loading foo/1{-debug} Loading requirement: qux/1 bar/1 $ module load foo/2 Loading foo/2 ERROR: Module cannot be loaded due to a conflict. HINT: Might try "module unload foo/1" first. $ module config conflict_unload 1 $ module load foo/2 Loading foo/2{-debug} Unloading conflict: foo/1{-debug} As seen in the above example, *Conflict Unload* is controlled by the "conflict_unload" configuration option. It is disabled by default as it changes behaviors of the "automated module handling mode". But everyone is encouraged to enable this new option to benefit from an highly automated experience. This option can be changed at installation time with "--enable-conflict-unload" configure script option. *Conflict Unload* handles all kind of conflicts: conflicts defined by already loaded modules, conflict declared by loading module through "conflict", "family" or "module unload" commands. It also supports the unload of a module that is loaded again but with different variant values. $ module load foo/2 +debug Loading foo/2{+debug} Unloading conflict: foo/2{-debug} Along with *Conflict Unload*, the *Useless Requirement Unload* mechanism is added to load evaluation. With it, auto loaded requirements of modules unloaded by *Conflict Unload* mechanism are automatically unloaded. $ module load foo/3 Loading foo/3{-debug} Unloading conflict: foo/2{+debug} bar/1 Loading requirement: bar/2 Unloading useless requirement: qux/1 Dependent modules of *Conflict Unload* modules are considered *Dependent Reload* modules. They are unloaded prior unloading the *Conflict Unload* module that depend on them. Reload occurs after loading main module and only if the loaded environment satisfies the dependencies of the module to reload. $ module load bar/3 Loading bar/3 Loading requirement: qux/2 $ module load qux/1 Loading qux/1 Unloading dependent: bar/3 Unloading conflict: qux/2 Reloading dependent: bar/3 Improved automated module handling mechanisms --------------------------------------------- Along with the introduction of *Conflict Unload* mechanism, some of the existing automated module handling mechanisms have been adapted and enhanced. *Useless Requirement Unload* (UReqUn) mechanism is moved during a "switch" sub-command from the end of the unload phase to the end of the load phase. Both *UReqUn* modules coming from unload and load phases are this way treated jointly. If *UReqUn* modules resulting from the switch unload phase are detected as conflict to the switched- on module, it is advised to enable the "conflict_unload" mechanism to automatically handle them. $ module switch foo/3 foo/4 Loading foo/4 ERROR: Module cannot be loaded due to a conflict. HINT: Might try "module unload bar/2" first. Switching from foo/3{-debug} to foo/4 ERROR: Load of switched-on foo/4 failed $ module config conflict_unload 1 $ module switch foo/3 foo/4 Switching from foo/3{-debug} to foo/4 Unloading conflict: bar/2 *Dependent Reload* (DepRe) modules that also are *UReqUn* modules are now unloaded during the *DepRe* unload phase instead of during the *UReqUn* process. As a consequence these modules are now unloaded prior main module action, as they are a dependent of this main module. This kind of *UReqUn* modules may not be a dependency of a module unloaded in the current processing. Modules from *Dependent Unload* (DepUn) and *Dependent Reload* (DepRe) mechanisms are now mixed together to proceed to their unload. They are this way unloaded in their reverse loading order. So if a module is part of the *DepUn* process and one of its requirement is part of the *DepRe* process, the requirement is now unloaded after its dependent. Reload of *Dependent Reload* (DepRe) module is now skipped if this module is not found loadable in environment. Either because its requirements are not loaded or a conflict is spotted. Such module is considered *Dependent Unload* rather *Dependent Reload*. If *DepRe* module is sticky, reload is attempted anyway. Reload of all *Dependent Reload* (DepRe) modules now occurs after main module evaluation rather after each sub module evaluation that triggered the *Dependent Reload* unload phase. When a loading module requires a *Dependent Unload* (DepUn) module, the unload of this module was forced when it was triggered by a Conflict Unload or a switched-off module. The unload of such *DepUn* module is not forced anymore by default and an error is raised if a loading module requiring the *DepUn* module is detected. When unloading *Dependent Reload* modules, do not force by default this unload unless asked with "--force" option. As a result an error is obtained when trying to reload Dependent Reload modules that conflict with other loaded modules. This error may be by-passed with " --force" command-line option. Specific modulepath for requirements ------------------------------------ A "--modulepath" option is introduced on the "always-load", "depends- on", "prereq", "prereq-all" and "prereq-any" modulefile commands. This option indicates that requirement should be specifically searched in the provided list of modulepaths. Such mechanism helps to ensure loaded requirements are those expected and they are not coming from an unexpected module tree. In the following example, "foo" module requires "bar". It expects the "bar" module from the same modulepath than "foo". User's environment has enabled another modulepath that also provides a "bar" module. Use of "--modulepath" option guides the requirement load mechanism to the expected module. $ module avail --------------------- /path/to/othermfiles --------------------- bar/1 --------------------- /path/to/modulefiles --------------------- bar/1 foo/1 foo/2 $ module show foo/1 ------------------------------------------------------------------- /path/to/modulefiles/foo/1: prereq --modulepath .. bar/1 ------------------------------------------------------------------- $ module load foo/1 Loading foo/1 Loading requirement: bar/1 $ echo $_LMFILES_ /path/to/modulefiles/bar/1:/path/to/modulefiles/foo/1 Specific modulepath set may guide to a directory not necessarily enabled (i.e., not defined in "MODULEPATH" environment variable): $ module show foo/2 ------------------------------------------------------------------- /path/to/modulefiles/foo/2: prereq --modulepath /path/to/alternatefiles bar/2 ------------------------------------------------------------------- $ module load foo/2 Loading foo/2 Loading requirement: bar/2 $ module list Currently Loaded Modulefiles: 1) bar/2 2) foo/2 $ echo $_LMFILES_ /path/to/alternatefiles/bar/2:/path/to/modulefiles/foo/2 If a required module is already loaded from a modulepath that does not correspond to the requirement definition, an error is obtained. However if the *Conflict Unload* mechanism described above is enabled, the conflicting module will automatically be unloaded. $ module purge $ module load bar/1 $ echo $_LMFILES_ /path/to/othermfiles/bar/1 $ module load foo/1 Loading bar/1 ERROR: Module already loaded from a different modulepath Loading foo/1 ERROR: Load of requirement bar/1 (specific path) failed $ module config conflict_unload 1 $ module load foo/1 Loading foo/1 Unloading conflict: bar/1 Loading requirement: bar/1 $ echo $_LMFILES_ /path/to/modulefiles/bar/1:/path/to/modulefiles/foo/1 v5.4 ==== This new version is backward-compatible with previous version 5 release. It fixes bugs but also introduces new functionalities that are described in this section. See the 5.4 release notes for a complete list of the changes between Modules v5.3 and v5.4. Enhancing extra specifiers -------------------------- Extra specifier "tag" is introduced to search modules based on tags applied. It applies to "avail", "whatis" and "paths" sub-commands. Tag abbreviation may also be used as extra specifier name. $ module avail tag:sticky --------------------- /path/to/modulefiles --------------------- bar/1 Multiple names can now be specified on one extra specifier criterion to select modules matching any of these names. $ module avail tag:S,F --------------------- /path/to/modulefiles --------------------- bar/1 qux/1 Multiple values can now be specified on one variant criterion to select modules providing any of these variant values. $ module config variant_shortcut toolchain=% $ module avail %a,c --------------------- /path/to/modulefiles --------------------- baz/1{%c} foo/1{%a,b,c} $ module avail %a,c %b --------------------- /path/to/modulefiles --------------------- foo/1{%a,b,c} Purging sticky modules ---------------------- The "sticky_purge" configuration option is added to define the behavior of "purge" sub-command when unloading a sticky or super- sticky module. By default an "error" is raised. "sticky_purge" can be changed to emit a "warning" message instead or to be "silent". $ module purge Unloading foo/1.0 ERROR: Unload of sticky module skipped $ module config sticky_purge warning $ module purge Unloading foo/1.0 WARNING: Unload of sticky module skipped $ module config sticky_purge silent $ module purge $ module list Currently Loaded Modulefiles: 1) foo/1.0 Key: sticky Specific modulepath labels -------------------------- "modulepath-label" command is introduced to define a label to use to designate modulepath in module "avail" output. This new command should be used in global or modulepath-specific rc files. $ cat /path/to/modulefiles/.modulerc #%Module modulepath-label . Tools $ module avail foo ---------------------- Tools ---------------------- foo/1.0 foo/2.0 Unique module name loaded ------------------------- The configuration option "unique_name_loaded" is introduced to allow only one module loaded per module name. It is similar to the *One name rule* feature introduced by the Lmod project. When enabled, "unique_name_loaded" produces a conflict definition at the start of modulefile evaluation for each actual and alternative module root name. For instance when loading a module named "foo/sub/1.0" with a "qux/1.0" alias, it produces a conflict against "foo" and "qux". When loading a module that shares a name with an already loaded module, an error is raised due to the conflict definition. This error aborts the module load evaluation. $ cat /path/to/modulefiles/bar/1.0 #%Module $ cat /path/to/modulefiles/bar/2.0 #%Module $ module config unique_name_loaded 1 $ module load bar/1.0 $ module load bar/2.0 Loading bar/2.0 ERROR: Module cannot be loaded due to a conflict. HINT: Might try "module unload bar" first. "unique_name_loaded" is disabled by default. It can be changed with module "config" sub-command or at installation time with "--enable- unique-name-loaded" configure script option. Cache sourced files ------------------- You may have gathered common Tcl code files used by all your modulefiles in Tcl files that are sourced with *source(n)* Tcl command. When loading multiple modulefiles at once, these sourced files may be read for every loaded modules, which may induce some I/O load. The configuration option "source_cache" is introduced to cache files evaluated through "source" Tcl command in order to only read them once during a "modulecmd.tcl" execution. When this option is enabled, sourced file read also benefits from optimized I/O calls of Modules Tcl extension library. In the following example, 20 modules are loaded, each of them sources the same Tcl file: $ syscall_list=close,fcntl,ioctl,newfstatat,openat,read,readlink $ strace -f -e $syscall_list -c $MODULES_CMD bash load foo/20 % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 31.48 0.000781 1 466 read 28.58 0.000709 5 131 16 openat 14.19 0.000352 2 142 3 newfstatat 9.71 0.000241 1 124 close 8.02 0.000199 0 212 211 readlink 4.55 0.000113 1 67 47 ioctl 3.47 0.000086 1 60 fcntl ------ ----------- ----------- --------- --------- ---------------- 100.00 0.002481 2 1202 277 total When sourced file is cached, it is only read once which saves a substantial amount of I/O operations: $ module config source_cache 1 $ strace -f -e $syscall_list -c $MODULES_CMD bash load foo/20 % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 34.41 0.000523 4 112 16 openat 27.37 0.000416 0 428 read 11.84 0.000180 1 122 3 newfstatat 11.51 0.000175 1 105 close 7.11 0.000108 2 47 27 ioctl 5.99 0.000091 0 92 91 readlink 1.78 0.000027 0 40 fcntl ------ ----------- ----------- --------- --------- ---------------- 100.00 0.001520 1 946 137 total Abort on error -------------- When an error occurs during the evaluation of a modulefile, several module sub-commands, like "load" or "unload", continue their processing. It means these sub-commands do not stop if they have multiple modulefiles to evaluate. This is named the *continue on error* behavior. The "abort_on_error" configuration option is introduced to abort, for the sub-commands listed in its value, the evaluation of multiple modulefiles if one fails to evaluate. The *abort on error* behavior is applied by default on **ml** command, "reload" sub-command and the unload phase of "switch" sub-command. This behavior may also be turned on for "load", "mod-to-sh", "purge", "switch", "try-load" and "unload" sub-commands. If a command is removed from "abort_on_error"'s value, it applies the *continue on error* behavior. In the following example, *abort on error* behavior is applied to "load" sub-command to stop evaluation instead of loading the remaining modulefile in the sequence if first modulefile fails to evaluate. $ module load foo bar Loading foo/1 ERROR: Module evaluation aborted $ module list Currently Loaded Modulefiles: 1) bar/1 $ module purge $ module config abort_on_error +load $ module load foo bar Loading foo/1 ERROR: Module evaluation aborted $ module list No Modulefiles Currently Loaded. When enabled for "switch" sub-command, whole action aborts if switched-on modulefile fails to load. By default, "switch" only aborts if switched-off modulefile fails to unload. $ module load bar $ module switch bar foo Loading foo/1 ERROR: Module evaluation aborted Switching from bar/1 to foo/1 WARNING: Load of switched-on foo/1 failed $ module list No Modulefiles Currently Loaded. $ module load bar $ module config abort_on_error +switch $ module switch bar foo Loading foo/1 ERROR: Module evaluation aborted Switching from bar/1 to foo/1 WARNING: Load of switched-on foo/1 failed $ module list Currently Loaded Modulefiles: 1) bar/1 "abort_on_error" configure option has no impact on "module" commands defined in modulefiles. When "--force" option is set, sub-commands apply the *continue on error* behavior. When "--enable-new-features" installation option is set, the *abort on error* behavior is also enabled on "load" and "switch" sub-commands. Improve error reporting ----------------------- Module evaluation error reports have been tweaked to appear where these errors occur rather reporting them on the message block of the main action. Such change helps to better understand the module evaluation flow. $ module load baz Loading baz/1 Loading requirement: foo/1 $ module switch foo bar Unloading baz/1 Module ERROR: invalid command name "bad_command" while executing "bad_command" (file "/path/to/modulefiles/baz/1" line 4) Please contact Unloading foo/1 ERROR: Unload of dependent baz/1 failed Switching from foo/1 to bar ERROR: Unload of switched-off foo/1 failed "switch" sub-command specific error messages are not reported when "module switch" command is run from a modulefile (like *unload of switched-off module failed*, *load of switched-on module failed* messages or *Switching* block message). On "switch" sub-command, load failure of switched-on module is now reported as an error. $ module load foo $ module switch foo qux Loading qux/1 Module ERROR: invalid command name "bad_command" while executing "bad_command" (file "/path/to/modulefiles/qux/1" line 2) Please contact Switching from foo/1 to qux/1 ERROR: Load of switched-on qux/1 failed Conflict error messages now describe the loaded module name and version that conflicts rather the generic conflict module specification. $ cat /path/to/modulefiles/bar/1 #%Module conflict foo $ module load foo $ module load bar Loading bar/1 ERROR: Module cannot be loaded due to a conflict. HINT: Might try "module unload foo/1" first. New options for source-sh modulefile command -------------------------------------------- Support for new *shell* mode named "bash-eval" is added to "source-sh" modulefile command. With this new mode, the generated output of the bash shell script is evaluated to get the environment changes instead of sourcing this script. $ cat /path/to/modulefiles/foo/foo-output.sh #!/bin/bash cat <